# Scraping Site
**URL:** https://discourse.julialang.org/t/scraping-site/14817
**Category:** New to Julia
**Created:** [September 11, 2018, 3:32pm UTC](https://discourse.julialang.org/t/scraping-site/14817 "2018-09-11T15:32:41Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@sdmcallister](https://discourse.julialang.org/u/sdmcallister)
#### Post date: [September 11, 2018, 3:32pm UTC](https://discourse.julialang.org/t/scraping-site/14817/1 "2018-09-11T15:32:41Z")
I have the following simple python script that works:
```julia
import requests
url = "https://ABCDEFG.com/user/login"
url2= "https://ABCDEFG.com/en/admin/events/page-download"
payload = {'name':'FOO',
'user_pin':'1234',
'destination':'https://ABC.com/en',
'commit':'Login'}
with requests.Session() as session:
post = session.post(url,data=payload)
r = session.get(url2)
print(r.text) #or do something else..
```
Is it possible to do something similar in Julia using HTTP.jl or another package?
EDIT:
Using HTTP.jl I’ve tried the following without success:
```julia
using HTTP, JSON
url = "https://ABCDEFG.com/user/login"
url2= "https://ABCDEFG.com/en/admin/events/page-download"
payload = json(Dict("name"=> "FOO",
"user_pin" => "1234",
"destination" => "https://ABC.com/en",
"commit" => "Login"))
r = HTTP.post(url,body=payload, cookies=true)
julia>r.status
200
# so far things look promising
julia>HTTP.get(url2)
ERROR: HTTP.ExceptionRequest.StatusError(403, HTTP.Messages.Response:
HTTP/1.1 403 Forbidden...
```
---
### Author:  [@avik](https://discourse.julialang.org/u/avik)
#### Post date: [September 11, 2018, 3:37pm UTC](https://discourse.julialang.org/t/scraping-site/14817/2 "2018-09-11T15:37:40Z")
[ ](https://www.youtube.com/watch?v=lFLMXaaMXwI)
---
### Author:  [@sdmcallister](https://discourse.julialang.org/u/sdmcallister)
#### Post date: [September 11, 2018, 7:01pm UTC](https://discourse.julialang.org/t/scraping-site/14817/3 "2018-09-11T19:01:35Z")
@avik Is it possible to open a session in HTTP.jl similar to request.py?
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [September 18, 2018, 5:59pm UTC](https://discourse.julialang.org/t/scraping-site/14817/4 "2018-09-18T17:59:08Z")
what do you mean exactly?
---
### Author:  [@sdmcallister](https://discourse.julialang.org/u/sdmcallister)
#### Post date: [September 18, 2018, 7:50pm UTC](https://discourse.julialang.org/t/scraping-site/14817/5 "2018-09-18T19:50:00Z")
I’m not sure what is actually unclear in the initial question.
I’m unable to provide a perfect MWE in order to protect the credentials and site.
What do I need to clarify?
Edit: I may be using the wrong terms given that I’m not very familiar with networking and HTTP. ‘Open a session’ is likely the wrong use of terms. I’m referring to the python code where a session is created…
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [September 18, 2018, 9:16pm UTC](https://discourse.julialang.org/t/scraping-site/14817/7 "2018-09-18T21:16:59Z")
I think the problem is that, you already got the result stored in varibale `r`, in the line
```nohighlight
HTTP.get(url2)
```
You’re not actually pasing the `payload`, rght?
---
### Author:  [@sdmcallister](https://discourse.julialang.org/u/sdmcallister)
#### Post date: [October 1, 2018, 3:41pm UTC](https://discourse.julialang.org/t/scraping-site/14817/8 "2018-10-01T15:41:16Z")
Providing the payload in the get request (these are just the credentials) still results in the same error.
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [October 5, 2018, 3:29am UTC](https://discourse.julialang.org/t/scraping-site/14817/9 "2018-10-05T03:29:22Z")
> [@sdmcallister](#):
>
> requests.Sessio
Sorry for the slow response here; looking things over a bit, it seems like doing `cookies=true` should work for you (in terms of replicating the requests.Session functionality). There might be some kind of bug in the code here: [https://github.com/JuliaWeb/HTTP.jl/blob/master/src/CookieRequest.jl](https://github.com/JuliaWeb/HTTP.jl/blob/master/src/CookieRequest.jl). If you could provide more details, we could perhaps figure out exactly what’s going wrong here. My personal approach would be to hopefully find a way to do the equivalent of HTTP.jl `verbose=2` with the requests to see exactly the request/response headers that are getting sent/coming back. If you could see which cookies/headers the `requests` library is storing/sending, then it should be straightforward to see which ones HTTP.jl isn’t sending. We could then figure out why not, or at least as a work-around you could pass the right header/cookies yourself.
---
### Author:  [@mantzaris](https://discourse.julialang.org/u/mantzaris)
#### Post date: [July 9, 2020, 3:54am UTC](https://discourse.julialang.org/t/scraping-site/14817/10 "2020-07-09T03:54:53Z")
I wish the video also had the code visible to accompany the description
---
### Author:  [@rapasite](https://discourse.julialang.org/u/rapasite)
#### Post date: [November 25, 2020, 4:41pm UTC](https://discourse.julialang.org/t/scraping-site/14817/11 "2020-11-25T16:41:05Z")
Some code for [avik](https://discourse.julialang.org/u/avik) video
From [https://github.com/Algocircle/Cascadia.jl](https://github.com/Algocircle/Cascadia.jl)
```julia
using Cascadia, Gumbo, HTTP
r = HTTP.get("http://stackoverflow.com/questions/tagged/julia-lang")
h = parsehtml(String(r.body))
qs = eachmatch(Selector(".question-summary"),h.root)
println("StackOverflow Julia Questions (votes answered? url)")
for q in qs
votes = nodeText(eachmatch(Selector(".votes .vote-count-post "), q)[1])
answered = length(eachmatch(Selector(".status.answered"), q)) > 0
href = eachmatch(Selector(".question-hyperlink"), q)[1].attributes["href"]
println("$votes $answered http://stackoverflow.com$href")
end
```