Last year I did a significant amount of work on HTTP2.jl which I intended as a replacement to HTTP.jl with full support for both the HTTP 2.0 and HTTP 1.1 protocols.
I believe that the “core logic” of this package is largely complete (more on this below), however a significant amount of work remains and I have been trying to get myself away from projects that I deem merely “useful” and back toward projects which are interesting and fun (i.e. I don’t want to spend much of my free time on anything that’s not math or physics related anymore). Therefore, I don’t want to undertake the implemention of the major missing features (connection pools) or the enormous amount of “smoothing and polishing” that making a worthy replacement for HTTP.jl would require.
I am announcing my abandoned package in the hope that it will prove useful to someone, as I do think that it would be a very good thing for Julia to have a native HTTP 2.0 implementation and that HTTP.jl, originally developed in the very early days of Julia, is crying out for a comprehensive replacement.
I will attempt to give a brief summary of what has been done and what remains to be done.
What has been done
the HPACK header compression protocol
This is the most complete part of the package and it includes unit testing. I considered publishing this as a separate package and registering it, but the protocol is almost absurdly specific (it even includes its own header-specific dictionary), so it’s hard to imagine it having any use except for HTTP 2.0.
concurrency safe connections with read and write loops
One of my major goals when I started HTTP2.jl was to fully take advantage of the streaming and concurrency features of HTTP 2.0. The central abstraction of the package are Connection objects which by default include separate threads for reading and writing which block when there is no input data, but they can also be run sequentially if need be. These connections are also “substrate agnostic” in that you can use them over any IO object, not necessarily a TCP connection (the major use case for this being testing).
the protocol is generalized in such a way that HTTP 1.0 is treated as a special case rather than a completely separate protocol
Another of my major goals was to make using HTTP 1.0 as painless as possible and without the need for a separate package. I did this mainly by treating HTTP 1.0 as a special HTTP 2.0 frame type. Because it is possible to upgrade an active connection from HTTP 1.0 to HTTP 2.0 it is important for them to work together seamlessly
1st class low-level API
One of my biggest complaints about HTTP.jl is that it is rather opaque. I wrote HTTP 2.0 so that packages would have the option to extend it in interesting ways, in particular
- Servers which may implement more complicated concurrency models, may need to handle their own connections, or use obscure features.
- The possibility of extending HTTP 2.0 with new frame types can potentially have interesting uses for networked IPC.
What remains to be done
connection pools
HTTP.jl references a global state for its connections called the connection pool. It is one of my major complaints about HTTP.jl that it doesn’t really leave you any other options, however there is no question this is an absolutely necessary feature. Without it, programs must create and manage their own Connection objects. This might not be such a bad thing if HTTP were to be used in a local, non-networked context, but over a network it’s potentially a nightmare (e.g. where exactly you declare your Connection compared to where you use it may be important as this might cause the remote host to drop the connection in the case of latency). Beside this, everyone of course wants to have the “fire and forget” HTTP2.get("https://julialang.org")-like methods because HTTP is so ubiquitous and sometimes you just need to call something and you don’t want to have to care about how that happens.
tons of polishing
Writing network code is a gigantic pain in the ass, as is evidenced by the startling number of try catch blocks in HTTP.jl. While I have gotten HTTP2.jl to work, getting it to work reliably in a wide variety of realistic cases is potentially a huge amount of work. In particular, it is likely that my HTTP2.jl code requires extensive error-handling code which is barely hinted at in the existing repo. This, more than anything else is what made me decide to move on from this project. If it were merely a matter of implementing connection pools, I’d have finished it already, but I’m not looking forward to ensuring that everything is working perfectly and have decided that life is too short to spend precious free time on this when I could be doing something that’s actually interesting.
How do you expect people to use this?
I have no idea. I assume that at some point HTTP.jl will be rewritten, there’s really no way around it. I think it would be really great if we had a solid Julia implementation rather than just wrapping libcurl or whatever, but perhaps this is something that falls into the category of “life is too short”. So, I’d like to throw my HTTP2.jl experiment out to fate, and hope that someone somewhere finds it useful. What I’ve already done was quite a lot of work.