OpenSSL.jl eof() blocks with Keep-Alive Connections

General Usage
,
1

I am struggling with reading from an sslstream (or even just a socket) with a keep alive connection (that is, ones where an HTTP request I send contains the ‘Connection: Keep-Alive header’). Specifically, it appears that the sslstream will present no bytes available until a call to eof(sslstream) is made. So if I expect a long response from the server requiring multiple reads I would write something like…

while(!eof(sslstream))
   write(someBuffer,readavailable(sslstream))
end

However, no EOF signal is sent during a keep-alive connection so eof() blocks. The code from the OpenSSL.jl github README page (GitHub - JuliaWeb/OpenSSL.jl) modified to include the code above will suffice as an example.

tcp = connect("www.nghttp2.org", 443)
# wrap tcp socket in OpenSSL.SSLStream
ssl = SSLStream(tcp)
# inject host for cert verification
OpenSSL.hostname!(ssl, "www.nghttp2.org")
# perform TLS handshake
OpenSSL.connect(ssl)
# can now write/read from ssl
request_str = "GET / HTTP/1.1\r\n"*
	"Host: www.nghttp2.org\r\n"*
	"User-Agent: curl\r\n"*
	"Accept: */*\r\n"*
	"Connection: Keep-Alive\r\n\r\n"
written = write(ssl, request_str)
io = IOBuffer()
# ********** BLOCKS  ************* 
while( !eof(ssl) )
	write(io, readavailable(ssl))
end
# ********** BLOCKS  ************* 
response = String(take!(io))
println(response)

Some potential solutions I see:

  1. making the eof() call asynchronous using @async, which is fine if there’s just one HTTP request, but the whole reason we “keep-alive” is to make a bunch more requests. This means that later in my application code I will make another request and there’s going to be a growing pile of idle eof() tasks in the scheduler that will muck with any buffers I’m writing to or reading from for the current request.

  2. Don’t use Keep-Alive connections. In this case the server will close the connection and an EOF signal will get sent and all will work as expected. However, now we have to reestablish a new connection for all requests which includes all the TLS handshake bits, which seems inefficient.

  3. The server will send a Content-Length: ### header for the response message body that I can check against what I’ve currently read to see if the server is done sending data. Again I have to use eof() calls to get the sslstream to present any bytes for reading and be super sure I don’t call eof() when there are no bytes left in the sslstream but you have to call eof() to check that, i.e. catch22

All my answers seem jerry-rigged and there are much smarter people than me that use this package in a professional and reliable way so my question is what is the accepted way to get data from the stream without needing to make blocking eof() calls?

Many thanks for any insight in advance!!