HTTP.jl managing cookies similar to Python Requests

I have to make authenticated get requests, and I am having issues using HTTP.jl. I am getting an authentication error. In the Python Requests package I would programmatically login and then pass my requests like:

response = requests.get(ags_url, params, cookies=session.cookies)

Below are the headers I received after I authenticated:

13-element Array{Pair{SubString{String},SubString{String}},1}:
“Date” => “Mon, 29 Apr 2019 18:27:36 GMT”
“X-Powered-By” => “Servlet/3.0”
“P3P” => “CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"”
“Pragma” => “no-cache”
“Cache-Control” => “no-store, no-cache, must-revalidate, private, proxy-revalidate”
“X-XSS-Protection” => “1; mode=block”
“X-Content-Type-Options” => “nosniff”
“SSO_STATUS” => “ParseOutput”
“Content-Length” => “148”
“Set-Cookie” => “JSESSIONID=0000MPoGXgudavv3ci0bUklNt-I:-1; Path=/; HttpOnly”
“Expires” => “Thu, 01 Dec 1994 16:00:00 GMT”
“Content-Type” => “text/xml;charset=UTF-8”
“Content-Language” => “en-US”

Do I need to pass the Set-Cookies similar to:

url = "http://fakeurl/app?"
cookies = Dict(headers[10])
query = (_f1 = "C", _FC = "Change", _EVT = "CHG", _RTN = "MSG", _LFN = "ALL", _PDL = "DEV", _TKN = "IC16.1", _f3 = "Shares")
response = HTTP.get(url; query = query, cookies=cookies)

I am getting a bad requests error. I guess I am am having trouble understanding the HTTP package after using Requests for so long. Can anyone help?

Just FYI, you can use requests via PyCall.

If you don’t need to modify the cookies you should be able to just use the default cookie jar without needing to pass them back on every request.

HTTP.request("GET", "http://httpbin.org/cookies/set?test=1"; cookies = true)

After the request the cookies are stored in HTTP.CookieRequest.default_cookiejar

Dict{String,Set{HTTP.Cookies.Cookie}} with 1 entry:
  "httpbin.org" => Set(HTTP.Cookies.Cookie[Cookie("test", "1", "/", "httpbin.org", 0001-01-01T00:00:00, 0, false, false, true, String[])])

You can use your own jar by passing a Dict{String, Set{Cookie}} on the cookiejar argument.

jar = Dict{String, Set{HTTP.Cookie}}()
HTTP.request("GET", "http://httpbin.org/cookies/set?test=1"; cookies = true, cookiejar = jar)

Note: There’s a bug in v0.8.0 which causes an error when just passing cookies = true and you might need to pass a non-empty Dict{String,String}.

HTTP.request("GET", "http://httpbin.org/cookies/set?test=1"; cookies = Dict{String,String}("type" => "quadruplechocchip"))

It’s fixed on the master branch.

1 Like

benelsen,

Thank you for the detailed response! I do have http.jl version 0.8.0 and am receiving an error when I try and use the cookies = true argument. From a high-level perspective, once I log in I receive a sessionID (token) for my session that must be passed on each subsequent transaction. So, can i just use the default cookie jar? Or said another way, will this library keep track of these cookies for me and pass them back on each HTTP.request, or will I need to manually set the cookies via the following command?

Dict{String,Set{HTTP.Cookies.Cookie}}

thanks!

1 Like

If the server returns a Set-Cookieheader, like in your original post, you should just be able to just use the default cookie jar, but you do need to enable cookie support on each request with cookies = true (or the workaround above in v0.8.0).

1 Like

I really appreciate your help. Sorry, i am still a newb :grimacing: