How to use cookies with HTTP.jl

New to Julia
1

I can’t for the life of me figure out how to successfully scrape a website where I need to login in order to access.

In my browser there’s 4 request cookies (by inspecting an arbitrary GET request)

{
“Request Cookies”: {
“xf_csrf”: “”,
“xf_logged_in”: “”,
“xf_session”: “”,
“xf_user”: “”
}
}

Which I assume I need to pass to HTTP.jl in some way.
I’ve tried various formats like passing a Dict() with the cookies, but always gets redirected to the login page.

Anyeone have any tips? I presume this is on the stupid side of questions asked in this forum, but I’ve spent like 2 hours now to no avail.

1 Like
2

There is a cookies keyword argument that you can either set to cookies = true to let the library manage cookies for you, or you can pass a dictionary with cookies as cookies = Dict("xf_csrf" => "foo", "xf_logged_in" => "bar", ...).

1 Like
3

On a tangent, I would recommend using Burpsuite as a debugging tool when trying to work out the secrets of websites. You use it as a proxy and MITM yourself. You can intercept any and all requests, edit the values etc.

This is the download page, which is a bit blank

this has a video to show you some of the features

1 Like