Filling form to get meteorological data on a website using RCurl in R
I´m trying to get rainfall data from INMET website.
First it needs to post a form with username and password, and then, fill a form with some parameters.
I´m trying first to set the login parameters with "postform" function from package Rcurl, and then generate the url to access data.
params<-list('E-mail do Usuário' = "username", 'Senha' = "xxxxx") #grabbed from html text
html<-postForm('http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php', params = params, curl = getCurlHandle(), style="POST")
station<-"83630"
startdate<-"01/01/1981"
enddate<-"01/01/2018"
url.data<-sprintf("http://www.inmet.gov.br/projetos/rede/pesquisa/gera_serie_txt.php?&mRelEstacao=%s&btnProcesso=serie&mRelDtInicio=%s&mRelDtFim=%s&mAtributos=,,,,,,,,,,1,,,,,,", station, startdate, enddate)
url.data is the final URL which I can access the data via html
I´m a bit stuck on this, but I guess I'm almost there.
r forms url rcurl
asked Nov 23 '18 at 12:11
BindiniBindini
346
add a comment |
I´m trying to get rainfall data from INMET website.
First it needs to post a form with username and password, and then, fill a form with some parameters.
I´m trying first to set the login parameters with "postform" function from package Rcurl, and then generate the url to access data.
params<-list('E-mail do Usuário' = "username", 'Senha' = "xxxxx") #grabbed from html text
html<-postForm('http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php', params = params, curl = getCurlHandle(), style="POST")
station<-"83630"
startdate<-"01/01/1981"
enddate<-"01/01/2018"
url.data<-sprintf("http://www.inmet.gov.br/projetos/rede/pesquisa/gera_serie_txt.php?&mRelEstacao=%s&btnProcesso=serie&mRelDtInicio=%s&mRelDtFim=%s&mAtributos=,,,,,,,,,,1,,,,,,", station, startdate, enddate)
url.data is the final URL which I can access the data via html
I´m a bit stuck on this, but I guess I'm almost there.
r forms url rcurl
asked Nov 23 '18 at 12:11
BindiniBindini
346
add a comment |
I´m trying to get rainfall data from INMET website.
First it needs to post a form with username and password, and then, fill a form with some parameters.
I´m trying first to set the login parameters with "postform" function from package Rcurl, and then generate the url to access data.
params<-list('E-mail do Usuário' = "username", 'Senha' = "xxxxx") #grabbed from html text
html<-postForm('http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php', params = params, curl = getCurlHandle(), style="POST")
station<-"83630"
startdate<-"01/01/1981"
enddate<-"01/01/2018"
url.data<-sprintf("http://www.inmet.gov.br/projetos/rede/pesquisa/gera_serie_txt.php?&mRelEstacao=%s&btnProcesso=serie&mRelDtInicio=%s&mRelDtFim=%s&mAtributos=,,,,,,,,,,1,,,,,,", station, startdate, enddate)
url.data is the final URL which I can access the data via html
I´m a bit stuck on this, but I guess I'm almost there.
r forms url rcurl
asked Nov 23 '18 at 12:11
BindiniBindini
346
I´m trying to get rainfall data from INMET website.
First it needs to post a form with username and password, and then, fill a form with some parameters.
I´m trying first to set the login parameters with "postform" function from package Rcurl, and then generate the url to access data.
params<-list('E-mail do Usuário' = "username", 'Senha' = "xxxxx") #grabbed from html text
html<-postForm('http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php', params = params, curl = getCurlHandle(), style="POST")
station<-"83630"
startdate<-"01/01/1981"
enddate<-"01/01/2018"
url.data<-sprintf("http://www.inmet.gov.br/projetos/rede/pesquisa/gera_serie_txt.php?&mRelEstacao=%s&btnProcesso=serie&mRelDtInicio=%s&mRelDtFim=%s&mAtributos=,,,,,,,,,,1,,,,,,", station, startdate, enddate)
url.data is the final URL which I can access the data via html
I´m a bit stuck on this, but I guess I'm almost there.
r forms url rcurl
r forms url rcurl
asked Nov 23 '18 at 12:11
BindiniBindini
346
asked Nov 23 '18 at 12:11
BindiniBindini
346
asked Nov 23 '18 at 12:11
BindiniBindini
346
asked Nov 23 '18 at 12:11
BindiniBindini
346
asked Nov 23 '18 at 12:11
BindiniBindini
346
346
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I can't get beyond the login since I don't have an account. For the login, that site makes a POST request that ultimately looks like this:
POST /projetos/rede/pesquisa/inicio.php HTTP/1.1
Host: www.inmet.gov.br
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,fr-BE;q=0.3
Referer: http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 78
Connection: close
Cookie: PHPSESSID=0j7eg8u12gka0c774aalahm9s0
Upgrade-Insecure-Requests: 1
mUsuario=USERNAME&mGerModulo=PES&mCod=a&mSenha=PASSWORD&mGerModulo=PES&btnProcesso=+Acessar+
So, it's passing in a few other query parameters. As a list they look like this:
list(
mUsuario = "USERNAME",
mGerModulo = "PES",
mCod = "a",
mSenha = "PASSWORD",
mGerModulo = "PES",
btnProcesso = "+Acessar+"
)
NOTE that you need to use the input form name vs the text of the form. I have no idea if you need to keep passing the user/pass for every request not can I help with any other requests since I cannot go past the login page.
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
I removed the login information
– Bindini
Nov 23 '18 at 13:38
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53446494%2ffilling-form-to-get-meteorological-data-on-a-website-using-rcurl-in-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I can't get beyond the login since I don't have an account. For the login, that site makes a POST request that ultimately looks like this:
POST /projetos/rede/pesquisa/inicio.php HTTP/1.1
Host: www.inmet.gov.br
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,fr-BE;q=0.3
Referer: http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 78
Connection: close
Cookie: PHPSESSID=0j7eg8u12gka0c774aalahm9s0
Upgrade-Insecure-Requests: 1
mUsuario=USERNAME&mGerModulo=PES&mCod=a&mSenha=PASSWORD&mGerModulo=PES&btnProcesso=+Acessar+
So, it's passing in a few other query parameters. As a list they look like this:
list(
mUsuario = "USERNAME",
mGerModulo = "PES",
mCod = "a",
mSenha = "PASSWORD",
mGerModulo = "PES",
btnProcesso = "+Acessar+"
)
NOTE that you need to use the input form name vs the text of the form. I have no idea if you need to keep passing the user/pass for every request not can I help with any other requests since I cannot go past the login page.
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
I removed the login information
– Bindini
Nov 23 '18 at 13:38
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
add a comment |
I can't get beyond the login since I don't have an account. For the login, that site makes a POST request that ultimately looks like this:
POST /projetos/rede/pesquisa/inicio.php HTTP/1.1
Host: www.inmet.gov.br
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,fr-BE;q=0.3
Referer: http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 78
Connection: close
Cookie: PHPSESSID=0j7eg8u12gka0c774aalahm9s0
Upgrade-Insecure-Requests: 1
mUsuario=USERNAME&mGerModulo=PES&mCod=a&mSenha=PASSWORD&mGerModulo=PES&btnProcesso=+Acessar+
So, it's passing in a few other query parameters. As a list they look like this:
list(
mUsuario = "USERNAME",
mGerModulo = "PES",
mCod = "a",
mSenha = "PASSWORD",
mGerModulo = "PES",
btnProcesso = "+Acessar+"
)
NOTE that you need to use the input form name vs the text of the form. I have no idea if you need to keep passing the user/pass for every request not can I help with any other requests since I cannot go past the login page.
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
I removed the login information
– Bindini
Nov 23 '18 at 13:38
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
add a comment |
I can't get beyond the login since I don't have an account. For the login, that site makes a POST request that ultimately looks like this:
POST /projetos/rede/pesquisa/inicio.php HTTP/1.1
Host: www.inmet.gov.br
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,fr-BE;q=0.3
Referer: http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 78
Connection: close
Cookie: PHPSESSID=0j7eg8u12gka0c774aalahm9s0
Upgrade-Insecure-Requests: 1
mUsuario=USERNAME&mGerModulo=PES&mCod=a&mSenha=PASSWORD&mGerModulo=PES&btnProcesso=+Acessar+
So, it's passing in a few other query parameters. As a list they look like this:
list(
mUsuario = "USERNAME",
mGerModulo = "PES",
mCod = "a",
mSenha = "PASSWORD",
mGerModulo = "PES",
btnProcesso = "+Acessar+"
)
NOTE that you need to use the input form name vs the text of the form. I have no idea if you need to keep passing the user/pass for every request not can I help with any other requests since I cannot go past the login page.
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
I can't get beyond the login since I don't have an account. For the login, that site makes a POST request that ultimately looks like this:
POST /projetos/rede/pesquisa/inicio.php HTTP/1.1
Host: www.inmet.gov.br
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.7,fr-BE;q=0.3
Referer: http://www.inmet.gov.br/projetos/rede/pesquisa/inicio.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 78
Connection: close
Cookie: PHPSESSID=0j7eg8u12gka0c774aalahm9s0
Upgrade-Insecure-Requests: 1
mUsuario=USERNAME&mGerModulo=PES&mCod=a&mSenha=PASSWORD&mGerModulo=PES&btnProcesso=+Acessar+
So, it's passing in a few other query parameters. As a list they look like this:
list(
mUsuario = "USERNAME",
mGerModulo = "PES",
mCod = "a",
mSenha = "PASSWORD",
mGerModulo = "PES",
btnProcesso = "+Acessar+"
)
NOTE that you need to use the input form name vs the text of the form. I have no idea if you need to keep passing the user/pass for every request not can I help with any other requests since I cannot go past the login page.
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
answered Nov 23 '18 at 13:18
hrbrmstrhrbrmstr
60.3k687148
60.3k687148
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
I removed the login information
– Bindini
Nov 23 '18 at 13:38
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
add a comment |
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
I removed the login information
– Bindini
Nov 23 '18 at 13:38
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
I see. This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…,,,,,, But I need to be logged to get the data after the request. I still didn´t understand how can I make a request and pass my login info together.
– Bindini
Nov 23 '18 at 13:31
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
You should delete that comment and then change your passwod b/c you just provided the world with your credentials. I am not authorized to use anyone else's credentials but my own and the site won't let me register.
– hrbrmstr
Nov 23 '18 at 13:33
I removed the login information
– Bindini
Nov 23 '18 at 13:38
I removed the login information
– Bindini
Nov 23 '18 at 13:38
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
This URL bellow is an example of a common request: inmet.gov.br/projetos/rede/pesquisa/…, But I need to be logged to get the data after the request. Once I´m logged, I can make another request. Pls, how can I make a request passing my login info together to get the data via html?
– Bindini
Nov 23 '18 at 13:44
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53446494%2ffilling-form-to-get-meteorological-data-on-a-website-using-rcurl-in-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
