Facing issue in HTTP.jl Proxy support

Sijun · November 12, 2019, 8:43am

First of all, I would like to thank everyone who helped me to overcome the obstacles I faced in learning this beautiful and powerful programming language.

This time, the issue is regarding proxy in HTTP.get in HTTP.jl.

r=HTTP.get("https://github.com/JuliaWeb", proxy="http://168.219.61.252:8080");

ERROR: HTTP.ExceptionRequest.StatusError(400, “CONNECT”, “github.com:”, HTTP.Messages.Response:
“”"
HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: close
Connection: close
Content-Length: 733

“”")

I googled to have discovered some similar postings but I still can’t figure it out.
I am doing this on Windows 10
I set HTTP_PROXY and HTTPS_PROXY to env vars expecting HTTP.jl to pick up any of these

With Python requests, it was successful:

import requests

proxies = {
 "http": "http://168.219.61.252:8080",
 "https": "http://168.219.61.252:8080",
}

s = requests.Session()
s.proxies = proxies

t = s.get("https://github.com/JuliaWeb")

It looks like that somehow the server recognizes the request as malformed when the request is assembled by HTTP.get() or request()

What else could I try?

oheil · November 12, 2019, 10:40am

So far I can only confirm. Maybe he @samoconnor can help?
Same settings, same errror:

http proxy

julia> r=HTTP.get("https://github.com/JuliaWeb"; verbose=3, proxy="http://our-proxy-server:3128");
ERROR: HTTP.ExceptionRequest.StatusError(400, "CONNECT", "github.com:", HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Proxy-Connection: close
Connection: close
Content-Length: 691

""")
Stacktrace:
 [1] #request#1(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}}, ::typeof(HTTP.request), ::Type{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\ExceptionRequest.jl:22
 [2] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}, ::typeof(HTTP.request), ::Type{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [3] (::getfield(Base, Symbol("###50#51#53")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)})(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}}, ::getfield(Base, Symbol("##50#52")){getfield(Base, Symbol("###50#51#53")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)}}, ::Type, ::Vararg{Any,N} where N) at .\error.jl:249
 [4] #request#1(::VersionNumber, ::String, ::Nothing, ::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}}, ::typeof(HTTP.request), ::Type{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [5] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}, ::typeof(HTTP.request), ::Type{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [6] #request#1(::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}}, ::typeof(HTTP.request), ::Type{RedirectLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\RedirectRequest.jl:24
 [7] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}, ::typeof(HTTP.request), ::Type{RedirectLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [8] #request#5(::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}, ::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}}, ::typeof(HTTP.request), ::String, ::String, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\HTTP.jl:308
 [9] (::getfield(HTTP, Symbol("#kw##get")))(::NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}, ::typeof(HTTP.get), ::String) at .\none:0
 [10] top-level scope at REPL[3]:1

julia>

Tried also ] add HTTP#jq/proxy , see https://github.com/JuliaWeb/HTTP.jl/issues/403

No success.

Same for https-proxy. error is:

https proxy

julia> r=HTTP.get("https://github.com/JuliaWeb"; verbose=3, proxy="https://our-proxy-server:3128");
ERROR: IOError(MbedTLS error code -29184: SSL - An invalid SSL record was received during request(https://our-proxy-server:3128))

Stacktrace:
 [1] handshake(::MbedTLS.SSLContext) at C:\Users\oheil\.julia\packages\MbedTLS\a1JFn\src\ssl.jl:83
 [2] handshake! at C:\Users\oheil\.julia\packages\MbedTLS\a1JFn\src\MbedTLS.jl:141 [inlined]
 [3] #sslconnection#18(::Bool, ::MbedTLS.SSLConfig, ::Base.Iterators.Pairs{Symbol,Union{Nothing, Int64},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :verbose),Tuple{Nothing,Int64}}}, ::typeof(HTTP.ConnectionPool.sslconnection), ::Sockets.TCPSocket, ::SubString{String}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\ConnectionPool.jl:637
 [4] #sslconnection at .\none:0 [inlined]
 [5] #getconnection#17 at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\ConnectionPool.jl:621 [inlined]
 [6] #getconnection at .\none:0 [inlined]
 [7] #getconnection#11(::Int64, ::Int64, ::Int64, ::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,Union{Nothing, Int64},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :verbose),Tuple{Nothing,Int64}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\ConnectionPool.jl:517
 [8] (::getfield(HTTP.ConnectionPool, Symbol("#kw##getconnection")))(::NamedTuple{(:reuse_limit, :iofunction, :verbose),Tuple{Int64,Nothing,Int64}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}, ::SubString{String}, ::SubString{String}) at .\none:0
 [9] #request#1(::String, ::Type, ::Int64, ::Base.Iterators.Pairs{Symbol,Union{Nothing, Int64},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :verbose),Tuple{Nothing,Int64}}}, ::typeof(HTTP.request), ::Type{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\ConnectionRequest.jl:41
 [10] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}, ::typeof(HTTP.request), ::Type{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [11] #request#1(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}}, ::typeof(HTTP.request), ::Type{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\ExceptionRequest.jl:19
 [12] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}, ::typeof(HTTP.request), ::Type{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [13] (::getfield(Base, Symbol("###50#51#53")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)})(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :proxy),Tuple{Nothing,Int64,String}}}, ::getfield(Base, Symbol("##50#52")){getfield(Base, Symbol("###50#51#53")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)}}, ::Type, ::Vararg{Any,N} where N) at .\error.jl:262
 [14] #request#1(::VersionNumber, ::String, ::Nothing, ::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}}, ::typeof(HTTP.request), ::Type{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [15] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}, ::typeof(HTTP.request), ::Type{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [16] #request#1(::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}}, ::typeof(HTTP.request), ::Type{RedirectLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\RedirectRequest.jl:24
 [17] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}, ::typeof(HTTP.request), ::Type{RedirectLayer{MessageLayer{RetryLayer{ExceptionLayer{ConnectionPoolLayer{DebugLayer{StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [18] #request#5(::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}, ::Nothing, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}}, ::typeof(HTTP.request), ::String, ::String, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\oheil\.julia\packages\HTTP\6Smhf\src\HTTP.jl:308
 [19] (::getfield(HTTP, Symbol("#kw##get")))(::NamedTuple{(:verbose, :proxy),Tuple{Int64,String}}, ::typeof(HTTP.get), ::String) at .\none:0
 [20] top-level scope at REPL[7]:1

julia>

oheil · November 12, 2019, 1:59pm

When I connect through proxy with openssl (v1.1) it works:

echo -e "GET /JuliaWeb HTTP/1.1\r\nHost:github.com\r\n\r\n" | ./apps/openssl s_client -quiet -connect github.com:443 -proxy our-proxy:80

results in:

HTTP/1.1 200 OK
Date: Tue, 12 Nov 2019 13:56:02 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Server: GitHub.com
Status: 200 OK
Vary: X-Requested-With
ETag: W/"d9197fcdfba91b4c5f5ea4e627774b44"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: has_recent_activity=1; path=/; expires=Tue, 12 Nov 2019 14:56:01 -0000
Set-Cookie: _octo=GH1.1.1425403097.1573566962; domain=.github.com; path=/; expires=Fri, 12 Nov 2021 13:56:02 -0000
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Sat, 12 Nov 2039 13:56:02 -0000; secure; HttpOnly
Set-Cookie: _gh_sess=Q0xLTjk1Wk94YTl4WG9IclkzbllpRjRKL0Y5czNodzBIaFk4VitYeHVxcmtwdlowWHhObUZGZEpDNlF3Mm1NWDJRL3lXTWhMVnBCQlhKRUUyV0dRZk9qVUJ2MWRzK2w0eTJQb1BmN25hTjVuK3dxZm1kVFRUNHV2TWtTMVJyc2Jld0xIUnNHVHhQQk9nZEtMdFF0MDBnPT0tLUNkaWlQRVNwN0hxRVo3djhYbnRyZHc9PQ%3D%3D--cbb807e8901c3ea136b41594bce49345ca297f22; path=/; secure; HttpOnly
X-Request-Id: b8f895b5-0887-494e-8814-4aa4e77bb9a1
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
Vary: Accept-Encoding
X-GitHub-Request-Id: D061:4513D:10CE9EE:1935D6A:5DCAB9F1

2c13


<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://github.githubassets.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">

...

Sijun · November 12, 2019, 2:32pm

Thanks! So, both openssl and python requests module work.

I tried:
``HTTP.get(“https://github.com/JuliaWeb:443”, proxy=“http://168.219.61.252:8080”);`

but it didn’t work. Maybe I will have to try some other experiments tomorrow.

oheil · November 12, 2019, 2:44pm

The following give some non error results:

r=HTTP.get("https://github.com:443/JuliaWeb"; proxy="http://our-proxy:3128");

See the explicit port 443 after github.com

(oops, you tried it)

For me this gives in r:

julia> r
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Date: Tue, 12 Nov 2019 14:42:16 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Server: GitHub.com
Status: 200 OK
Vary: X-Requested-With
ETag: W/"fea18ac6845f03dce34bff2158432afb"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: has_recent_activity=1; path=/; expires=Tue, 12 Nov 2019 15:42:16 -0000
Set-Cookie: _octo=GH1.1.1116754122.1573569736; domain=.github.com; path=/; expires=Fri, 12 Nov 2021 14:42:16 -0000
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Sat, 12 Nov 2039 14:42:16 -0000; secure; HttpOnly
Set-Cookie: _gh_sess=RHpOYkM0Yi92eVZ6TjNkUXBuNHRIaFVWekFkWkZ6Q0hOYzdlelhNQ1JES0hwVHI1L2R2YUtac3BrU0ZOWWVjcitEY1lvSFpiK2p3RFA2T2dvOUdLV2Fha29RMW9OR2pHYXEwd0FqTXlQTkZxd1VrNUJtZEZBTGtEbHJEMi9BRHpMODNrbG9yd3FVeGl5L0lpKzBLNmFnPT0tLXdVVnEzZVBXdmN3NjB6VDB4YzZDVkE9PQ%3D%3D--31300cfa24a0d234e5d0abf3dd4ae3909136697d; path=/; secure; HttpOnly
X-Request-Id: d6330459-d2de-4ac2-bcb9-43b1a0ebff47
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
Vary: Accept-Encoding
X-GitHub-Request-Id: F5CC:45142:4D18260:7467F3C:5DCAC4C7



<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://github.githubassets.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">



  <link crossorigin="anonymous" media="all" integrity="sha512-/YEVWs7BzxfKyUd6zVxjEQcXRWsLbcEjv045Rq8DSoipySmQblhVKxlXLva2GtNd5DhwCxHwW1RM0N9I7S2Vew==" rel="stylesheet" href="https://github.githubassets.com/assets/frameworks-481a47a96965f6706fb41bae0d14b09a.css" />
  <link crossorigin="anonymous" media="all" integrity="sha512-rqlwdaP+8oV7JwHySo0RpslRQAvrs57OQxH/1vmIliL2vqPijry+bYuYzVw9yZoZMZI0Lbe3tJeMlcDPvISB
⋮
180787-byte body
"""

julia>

Sijun · November 12, 2019, 2:46pm

@oheil, Oh my… I made silly mistake. I will try that surely. Thank you very much!

oheil · November 12, 2019, 2:52pm

Yes, now I see, your port is at the wrong place. After the host it should be.

Looking into all these codes and errors makes me somehow dizzy

oheil · November 12, 2019, 3:15pm

Opened issue:

https://github.com/JuliaWeb/HTTP.jl/issues/472

Sijun · November 12, 2019, 3:25pm

Oh it’s great. thank you so much.