I’m trying to learn programming in Julia. i’m using this package HTTP.jl
can anybody explain why this first request won’t work for me, however the second one straight out of the documentation for HTTP.jl works?
This is the second request below: (I couldn’t post a second screen shot because i’m new to discourse)
julia> HTTP.request(“GET”, “http://httpbin.org/ip”)
I’m not sure if it is because I don’t understand how GET works, or if it’s some other reason.
Thanks
Welcome @Bucephalus !
It seems that your Internet is broken 
Or in other words: the error message implies that you can’t connect to the URLs you are using. Maybe you are behind a proxy?
Another thing:
Don’t post screenshots, they are difficult to spot and read.
Just do like I do here:
julia> using HTTP
julia> HTTP.request("GET", "http://httpbin.org/ip")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Date: Sat, 15 Aug 2020 09:43:28 GMT
Content-Type: application/json
Content-Length: 33
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
{
"origin": "176.199.208.97"
}
"""
Copy&Paste into discourse makes it easy for us to copy&paste from discourse.
See Please read: make it easier to help you for more infos.
Thanks @oheil
I will remember the screenshot thing for next time. You might be right about that proxy thing, i’m just working from my home computer. I’m suspecting this code would work alright from an amazon instance, etc. I will keep going through this book.
thanks for your help.
There is some old discussion about HTTP.jl and proxy:
Not sure if this still fits.
I think the general way to solve this is using the environment variables:
http_proxy , https_proxy
There is also a closed issue which shows some information on how to do it with linux/bash:
Thanks, I will investigate more. I can ping google.com from my windows computer but not from my ubuntu computer. I want to try to understand that first.
Your Linux machine might have DNS configuration issues.
I have a feeling it’s because my ISP doesn’t provide IPv6, or related to IPv6. ping -4 works for both computers, but ping -6 does not work for either. Thanks Oheil and Stefan.
HTTP is IP version agnostic.
For me it is not clear, if you are trying on Windows, Linux or from some AWS instance.
You may try from a Windows CMD:
curl --get http://httpbin.org/ip
and see what happens:
{
"origin": "176.199.208.97"
}
Or from powershell:
PS C:\> Invoke-WebRequest -Uri http://httpbin.org/ip
StatusCode : 200
StatusDescription : OK
Content : {
"origin": "176.199.208.97"
}
RawContent : HTTP/1.1 200 OK
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 33
Content-Type: application/json
Date: Sun, 16 Aug 2020 08:55:20 GMT...
Forms : {}
Headers : {[Connection, keep-alive], [Access-Control-Allow-Origin, *], [Access-Control-Allow-Credentials,
true], [Content-Length, 33]...}
Images : {}
InputFields : {}
Links : {}
ParsedHtml : System.__ComObject
RawContentLength : 33
Hi oheil
I’m using an Ubuntu machine, I was just cross-checking behaviour on the windows machine. So when i ping -6 my ISP’s DNS server it fails with feedback to the effect of “This address family not supported…” So I’m pretty sure, at least in my area, at this time, that my ISP is not supporting ipv6. I got some help over at ask ubuntu, and I got some assistance on configuring my Ubuntu machine to using ipv4 by default. Now HTTP.jl works.
I really do not understand how a web browser (http) doesn’t have this problem, but when I use these HTTP commands in terminal I have this problem until I configure Ubuntu to use ipv4 by default. If someone can explain this, that would be good…
For future reference if anybody has similar problem, I had to go to /etc/gai.conf and put this down the bottom of the file:
precedence ::ffff:0:0/96 100
Please reference original advice here
It seems that this is an issue.
HTTP.jl is using stdlib Sockets.jl
https://docs.julialang.org/en/v1/stdlib/Sockets/
where we have
https://docs.julialang.org/en/v1/stdlib/Sockets/#Sockets.getalladdrinfo
Gets all of the IP addresses of the
host. Uses the operating system’s underlyinggetaddrinfoimplementation, which may do a DNS lookup.
julia> getalladdrinfo("google.com")
2-element Array{IPAddr,1}:
ip"172.217.6.174"
ip"2607:f8b0:4000:804::200e"
which is used in HTTP.jl:
https://github.com/JuliaWeb/HTTP.jl/blob/master/src/ConnectionPool.jl#L616
as
Sockets.getalladdrinfo(host)[1]
So if operating system’s underlying getaddrinfo returns IP4 and IP6 adresses, the implementation on HTTP.jl uses the IP4 adress, which than may fail, if only IP6 is allowed.
Not sure if this is analysis is right, I can’t check so easily.
But you may open an issue at
describing your case and refering to this discussion.
To be clear: a can’t check if this is just a misconfiguration of the client or if not, something unusual of the client which has been overseen in HTTP.jl
Well this is awkward - so I deleted that line from the gai.conf file to replicate the behaviour to open an issue at the HTTP github page.
The ping (defaulting to ip -6) and the explicit ping -6 issue came back as expected. However when I went into REPL and did another HTTP.request , the original issue at the top of this post, i could not replicate and it worked. So now it works, even with the gai.conf file back to its original state.
Finally, i turned off my machine and restarted it again and again tried the HTTP.request from REPL and it still worked.
So now that I cannot replicate the behaviour I will not bother opening an issue and see what happens as I progress.
Thanks for your help oheil and Stefan
I’m actually getting confusing results and testing still. I’m not at the stage that I want to raise an issue yet because I’m not sure if it’s my computer or the package. I’m suspecting it’s my computer now because i’m getting the following results: (i changed the gai.conf file back to original state and rebooted before I did these tests.)
david@Ptolemy:~$ ping www.google.com
PING www.google.com(syd09s23-in-x04.1e100.net (2404:6800:4006:80f::2004)) 56 data bytes
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=1 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=2 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=3 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=4 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=5 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=6 Destination unreachable: Address unreachable
^X^C
--- www.google.com ping statistics ---
8 packets transmitted, 0 received, +6 errors, 100% packet loss, time 7090ms
david@Ptolemy:~$ ping www.amazon.com
PING d3ag4hukkh62yn.cloudfront.net (13.224.176.180) 56(84) bytes of data.
64 bytes from server-13-224-176-180.syd1.r.cloudfront.net (13.224.176.180): icmp_seq=1 ttl=244 time=22.1 ms
64 bytes from server-13-224-176-180.syd1.r.cloudfront.net (13.224.176.180): icmp_seq=2 ttl=244 time=21.9 ms
64 bytes from server-13-224-176-180.syd1.r.cloudfront.net (13.224.176.180): icmp_seq=3 ttl=244 time=21.8 ms
^X64 bytes from server-13-224-176-180.syd1.r.cloudfront.net (13.224.176.180): icmp_seq=4 ttl=244 time=21.7 ms
^C
--- d3ag4hukkh62yn.cloudfront.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 21.661/21.860/22.090/0.160 ms
david@Ptolemy:~$ ping -6 www.google.com
PING www.google.com(syd09s23-in-x04.1e100.net (2404:6800:4006:80f::2004)) 56 data bytes
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=1 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=2 Destination unreachable: Address unreachable
From 2001:8003:7405:1200:faab:5ff:feca:8050 (2001:8003:7405:1200:faab:5ff:feca:8050) icmp_seq=3 Destination unreachable: Address unreachable
^C
--- www.google.com ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3026ms
david@Ptolemy:~$ ping -6 www.amazon.com
ping: www.amazon.com: Name or service not known
david@Ptolemy:~$ ping -4 www.google.com
PING www.google.com (142.250.66.196) 56(84) bytes of data.
64 bytes from syd09s23-in-f4.1e100.net (142.250.66.196): icmp_seq=1 ttl=115 time=22.3 ms
64 bytes from syd09s23-in-f4.1e100.net (142.250.66.196): icmp_seq=2 ttl=115 time=27.1 ms
64 bytes from syd09s23-in-f4.1e100.net (142.250.66.196): icmp_seq=3 ttl=115 time=22.3 ms
64 bytes from syd09s23-in-f4.1e100.net (142.250.66.196): icmp_seq=4 ttl=115 time=22.3 ms
64 bytes from syd09s23-in-f4.1e100.net (142.250.66.196): icmp_seq=5 ttl=115 time=21.9 ms
^X64 bytes from syd09s23-in-f4.1e100.net (142.250.66.196): icmp_seq=6 ttl=115 time=22.2 ms
^C
--- www.google.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 21.931/23.008/27.081/1.825 ms
david@Ptolemy:~$ ping -4 www.amazon.com
PING e15316.e22.akamaiedge.net (23.217.176.20) 56(84) bytes of data.
64 bytes from a23-217-176-20.deploy.static.akamaitechnologies.com (23.217.176.20): icmp_seq=1 ttl=60 time=9.58 ms
64 bytes from a23-217-176-20.deploy.static.akamaitechnologies.com (23.217.176.20): icmp_seq=2 ttl=60 time=9.31 ms
64 bytes from a23-217-176-20.deploy.static.akamaitechnologies.com (23.217.176.20): icmp_seq=3 ttl=60 time=9.36 ms
^C
--- e15316.e22.akamaiedge.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.305/9.414/9.578/0.117 ms
If you see that, then see below how HTTP.request for google does not work, but HTTP.request for amazon does work. For this reason I feel my computer is causing the issue - check below:
julia> r = HTTP.request("GET", "https://www.google.com/")
ERROR: IOError(Base.IOError("connect: host is unreachable (EHOSTUNREACH)", -113) during request(https://www.google.com/))
Stacktrace:
[1] wait_connected(::Sockets.TCPSocket) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/Sockets.jl:525
[2] connect at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/Sockets.jl:560 [inlined]
[3] connect(::Sockets.IPv6, ::UInt64) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Sockets/src/Sockets.jl:546
[4] getconnection(::Type{Sockets.TCPSocket}, ::SubString{String}, ::String; keepalive::Bool, connect_timeout::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol,Symbol},NamedTuple{(:require_ssl_verification, :iofunction, :reached_redirect_limit),Tuple{Bool,Nothing,Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/ConnectionPool.jl:616
[5] #getconnection#29 at /home/david/.julia/packages/HTTP/atT5q/src/ConnectionPool.jl:668 [inlined]
[6] newconnection(::HTTP.ConnectionPool.Pod, ::Type{T} where T, ::SubString{String}, ::SubString{String}, ::Int64, ::Bool, ::Int64; kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit),Tuple{Nothing,Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/ConnectionPool.jl:583
[7] getconnection(::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}; connection_limit::Int64, pipeline_limit::Int64, idle_timeout::Int64, reuse_limit::Int64, require_ssl_verification::Bool, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit),Tuple{Nothing,Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/ConnectionPool.jl:527
[8] request(::Type{ConnectionPoolLayer{StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}; proxy::Nothing, socket_type::Type{T} where T, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit),Tuple{Nothing,Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/ConnectionRequest.jl:73
[9] request(::Type{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N; kw::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit),Tuple{Nothing,Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/ExceptionRequest.jl:19
[10] (::Base.var"#56#58"{Base.var"#56#57#59"{ExponentialBackOff,HTTP.RetryRequest.var"#2#3"{Bool,HTTP.Messages.Request},typeof(HTTP.request)}})(::Type{T} where T, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Symbol,Union{Nothing, Bool},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :reached_redirect_limit),Tuple{Nothing,Bool}}}) at ./error.jl:301
[11] #request#1 at /home/david/.julia/packages/HTTP/atT5q/src/RetryRequest.jl:44 [inlined]
[12] request(::Type{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:reached_redirect_limit,),Tuple{Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/MessageRequest.jl:51
[13] request(::Type{BasicAuthLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; kw::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:reached_redirect_limit,),Tuple{Bool}}}) at /home/david/.julia/packages/HTTP/atT5q/src/BasicAuthRequest.jl:28
[14] request(::Type{RedirectLayer{BasicAuthLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/david/.julia/packages/HTTP/atT5q/src/RedirectRequest.jl:24
[15] request(::Type{RedirectLayer{BasicAuthLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at /home/david/.julia/packages/HTTP/atT5q/src/RedirectRequest.jl:21
[16] request(::String, ::String, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}; headers::Array{Pair{SubString{String},SubString{String}},1}, body::Array{UInt8,1}, query::Nothing, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/david/.julia/packages/HTTP/atT5q/src/HTTP.jl:314
[17] request at /home/david/.julia/packages/HTTP/atT5q/src/HTTP.jl:314 [inlined] (repeats 2 times)
[18] top-level scope at REPL[8]:1
julia> r = HTTP.request("GET", "https://www.amazon.com/")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: Server
Date: Mon, 17 Aug 2020 13:46:47 GMT
x-amz-rid: AFPE4ZKB9HX8C16BE1RT
Set-Cookie: session-id=137-8852803-5109410; Domain=.amazon.com; Expires=Tue, 17-Aug-2021 13:46:47 GMT; Path=/
Set-Cookie: session-id-time=2082787201l; Domain=.amazon.com; Expires=Tue, 17-Aug-2021 13:46:47 GMT; Path=/
Set-Cookie: i18n-prefs=USD; Domain=.amazon.com; Expires=Tue, 17-Aug-2021 13:46:47 GMT; Path=/
Set-Cookie: sp-cdn="L5Z9:AU"; Version=1; Domain=.amazon.com; Max-Age=31536000; Expires=Tue, 17-Aug-2021 13:46:47 GMT; Path=/; Secure; HttpOnly
Set-Cookie: skin=noskin; path=/; domain=.amazon.com
Accept-CH: ect,rtt,downlink
Accept-CH-Lifetime: 86400
X-UA-Compatible: IE=edge
Content-Language: en-US
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
X-XSS-Protection: 1;
X-Content-Type-Options: nosniff
Vary: Accept-Encoding,User-Agent,Content-Type,Cookie,Referer,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
Strict-Transport-Security: max-age=47474747; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
X-Cache: Miss from cloudfront
Via: 1.1 52fa887ba82513d16e3f586c3db681fe.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: SYD1-C2
X-Amz-Cf-Id: 6j1hfOUNdBTCmGOBLPPUOxVRfyuWjw5w1DPal1RUCbD2CessJb21pQ==
<!doctype html><html lang="en-us" class="a-no-js" data-19ax5a9jf="dingo"><!-- sp:feature:head-start -->
<head><script>var aPageStart = (new Date()).getTime();</script><meta charset="utf-8"/>
<script>var ue_t0=ue_t0||+new Date();</script>
<!-- sp:feature:cs-optimization -->
<meta http-equiv='x-dns-prefetch-control' content='on'>
<link rel="preconnect" href="https://images-na.ssl-images-amazon.com" crossorigin>
<link rel="preconnect" href="https://m.media-amazon.com" crossorigin>
<link rel="preconnect" href="https://completion.amazon.com" crossorigin>
<script>
(function(a){var b=a.performance;a.ue_mark=b&&"function"===typeof b.mark?function(a){b.mark(a)}:function(){}})(window);
ue_mark('csm:ho:ob');
</script>
<script>
window.ue_ihb = (window.ue_ihb || window.ueinit || 0) + 1;
if (window.ue_ihb === 1) {
var ue_csm = window,
ue_hob = +new Date();
(function(d){var e=d.ue=d.ue||{},f=Date.now||function(){return+new Date};e.d=function(b){return f()-(b?0:d.ue_t0)};e.stub
⋮
274645-byte body
"""
So I’m trying to still understand what is happening here.