Publishing my HTTP2.jl work

Last year I did a significant amount of work on HTTP2.jl which I intended as a replacement to HTTP.jl with full support for both the HTTP 2.0 and HTTP 1.1 protocols.

I believe that the “core logic” of this package is largely complete (more on this below), however a significant amount of work remains and I have been trying to get myself away from projects that I deem merely “useful” and back toward projects which are interesting and fun (i.e. I don’t want to spend much of my free time on anything that’s not math or physics related anymore). Therefore, I don’t want to undertake the implemention of the major missing features (connection pools) or the enormous amount of “smoothing and polishing” that making a worthy replacement for HTTP.jl would require.

I am announcing my abandoned package in the hope that it will prove useful to someone, as I do think that it would be a very good thing for Julia to have a native HTTP 2.0 implementation and that HTTP.jl, originally developed in the very early days of Julia, is crying out for a comprehensive replacement.

I will attempt to give a brief summary of what has been done and what remains to be done.

What has been done

the HPACK header compression protocol

This is the most complete part of the package and it includes unit testing. I considered publishing this as a separate package and registering it, but the protocol is almost absurdly specific (it even includes its own header-specific dictionary), so it’s hard to imagine it having any use except for HTTP 2.0.

concurrency safe connections with read and write loops

One of my major goals when I started HTTP2.jl was to fully take advantage of the streaming and concurrency features of HTTP 2.0. The central abstraction of the package are Connection objects which by default include separate threads for reading and writing which block when there is no input data, but they can also be run sequentially if need be. These connections are also “substrate agnostic” in that you can use them over any IO object, not necessarily a TCP connection (the major use case for this being testing).

the protocol is generalized in such a way that HTTP 1.0 is treated as a special case rather than a completely separate protocol

Another of my major goals was to make using HTTP 1.0 as painless as possible and without the need for a separate package. I did this mainly by treating HTTP 1.0 as a special HTTP 2.0 frame type. Because it is possible to upgrade an active connection from HTTP 1.0 to HTTP 2.0 it is important for them to work together seamlessly

1st class low-level API

One of my biggest complaints about HTTP.jl is that it is rather opaque. I wrote HTTP 2.0 so that packages would have the option to extend it in interesting ways, in particular

  • Servers which may implement more complicated concurrency models, may need to handle their own connections, or use obscure features.
  • The possibility of extending HTTP 2.0 with new frame types can potentially have interesting uses for networked IPC.

What remains to be done

connection pools

HTTP.jl references a global state for its connections called the connection pool. It is one of my major complaints about HTTP.jl that it doesn’t really leave you any other options, however there is no question this is an absolutely necessary feature. Without it, programs must create and manage their own Connection objects. This might not be such a bad thing if HTTP were to be used in a local, non-networked context, but over a network it’s potentially a nightmare (e.g. where exactly you declare your Connection compared to where you use it may be important as this might cause the remote host to drop the connection in the case of latency). Beside this, everyone of course wants to have the “fire and forget” HTTP2.get("https://julialang.org")-like methods because HTTP is so ubiquitous and sometimes you just need to call something and you don’t want to have to care about how that happens.

tons of polishing

Writing network code is a gigantic pain in the ass, as is evidenced by the startling number of try catch blocks in HTTP.jl. While I have gotten HTTP2.jl to work, getting it to work reliably in a wide variety of realistic cases is potentially a huge amount of work. In particular, it is likely that my HTTP2.jl code requires extensive error-handling code which is barely hinted at in the existing repo. This, more than anything else is what made me decide to move on from this project. If it were merely a matter of implementing connection pools, I’d have finished it already, but I’m not looking forward to ensuring that everything is working perfectly and have decided that life is too short to spend precious free time on this when I could be doing something that’s actually interesting.

How do you expect people to use this?

I have no idea. I assume that at some point HTTP.jl will be rewritten, there’s really no way around it. I think it would be really great if we had a solid Julia implementation rather than just wrapping libcurl or whatever, but perhaps this is something that falls into the category of “life is too short”. So, I’d like to throw my HTTP2.jl experiment out to fate, and hope that someone somewhere finds it useful. What I’ve already done was quite a lot of work.

24 Likes

FWIW, I’ve been working pretty steadily over the last 8 months on overhauling HTTP.jl (for the 2nd time!), and it’s very close to a 1.0 release (like days away). We’ve closed something like ~100 issues in that time, overhauled large parts of the codebase, and improved polish and robustness significantly. It also helps that I’m now employed full-time to work on things like the robustness of foundational packages in the ecosystem like HTTP.jl! My goal is for HTTP.jl to get back to a quality reputation as robust and flexible pure-Julia.

I would love to get solid HTTP2 support in Julia and it’s on my list. I would love to look into the code you’re sharing and see if we can incorporate it into HTTP.jl.

21 Likes

Looking forward to that!

Just thought I should clarify especially for anyone not familiar with HTTP.jl, anything I say in the OP about HTTP.jl is based on my reading of it when I was last working on HTTP2.jl, I haven’t been following it so what I said may not reflect the current state of the package.

2 Likes

Thanks for your initiative @ExpandingMan !
Hope this helps for more people to contribute towards Julia having better web libraries