HTTP.jl managing cookies similar to Python Requests

New to Julia
,
1

I have to make authenticated get requests, and I am having issues using HTTP.jl. I am getting an authentication error. In the Python Requests package I would programmatically login and then pass my requests like:

response = requests.get(ags_url, params, cookies=session.cookies)

Below are the headers I received after I authenticated:

13-element Array{Pair{SubString{String},SubString{String}},1}:
“Date” => “Mon, 29 Apr 2019 18:27:36 GMT”
“X-Powered-By” => “Servlet/3.0”
“P3P” => “CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"”
“Pragma” => “no-cache”
“Cache-Control” => “no-store, no-cache, must-revalidate, private, proxy-revalidate”
“X-XSS-Protection” => “1; mode=block”
“X-Content-Type-Options” => “nosniff”
“SSO_STATUS” => “ParseOutput”
“Content-Length” => “148”
“Set-Cookie” => “JSESSIONID=0000MPoGXgudavv3ci0bUklNt-I:-1; Path=/; HttpOnly”
“Expires” => “Thu, 01 Dec 1994 16:00:00 GMT”
“Content-Type” => “text/xml;charset=UTF-8”
“Content-Language” => “en-US”

Do I need to pass the Set-Cookies similar to:

url = "http://fakeurl/app?"
cookies = Dict(headers[10])
query = (_f1 = "C", _FC = "Change", _EVT = "CHG", _RTN = "MSG", _LFN = "ALL", _PDL = "DEV", _TKN = "IC16.1", _f3 = "Shares")
response = HTTP.get(url; query = query, cookies=cookies)

I am getting a bad requests error. I guess I am am having trouble understanding the HTTP package after using Requests for so long. Can anyone help?

2

Just FYI, you can use requests via PyCall.

3

If you don’t need to modify the cookies you should be able to just use the default cookie jar without needing to pass them back on every request.

HTTP.request("GET", "http://httpbin.org/cookies/set?test=1"; cookies = true)

After the request the cookies are stored in HTTP.CookieRequest.default_cookiejar

Dict{String,Set{HTTP.Cookies.Cookie}} with 1 entry:
  "httpbin.org" => Set(HTTP.Cookies.Cookie[Cookie("test", "1", "/", "httpbin.org", 0001-01-01T00:00:00, 0, false, false, true, String[])])

You can use your own jar by passing a Dict{String, Set{Cookie}} on the cookiejar argument.

jar = Dict{String, Set{HTTP.Cookie}}()
HTTP.request("GET", "http://httpbin.org/cookies/set?test=1"; cookies = true, cookiejar = jar)

Note: There’s a bug in v0.8.0 which causes an error when just passing cookies = true and you might need to pass a non-empty Dict{String,String}.

HTTP.request("GET", "http://httpbin.org/cookies/set?test=1"; cookies = Dict{String,String}("type" => "quadruplechocchip"))

It’s fixed on the master branch.

1 Like
4

benelsen,

Thank you for the detailed response! I do have http.jl version 0.8.0 and am receiving an error when I try and use the cookies = true argument. From a high-level perspective, once I log in I receive a sessionID (token) for my session that must be passed on each subsequent transaction. So, can i just use the default cookie jar? Or said another way, will this library keep track of these cookies for me and pass them back on each HTTP.request, or will I need to manually set the cookies via the following command?

Dict{String,Set{HTTP.Cookies.Cookie}}

thanks!

1 Like
5

If the server returns a Set-Cookieheader, like in your original post, you should just be able to just use the default cookie jar, but you do need to enable cookie support on each request with cookies = true (or the workaround above in v0.8.0).

1 Like
6

I really appreciate your help. Sorry, i am still a newb :grimacing: