Good news for interfacing with the web in Julia: a new package HTTP.jl has been released (technically, re-released after an original package from 3 years ago was deprecated).
Julia’s webstack has languished a bit as of late and this is a concerted effort to revive, modernize, and improve the state of webstack code for Julia. HTTP.jl is actually a consolidation of the various web packages to ensure code consistency, reduction in code duplication, and to streamline the modernization process. Certain packages have gone through complete re-writes (URIParser.jl, Requests.jl, HttpServer.jl), with also an effort to remove certain binary dependencies (HttpParser.jl, rewritten in native Julia).
The consolidation is a short-term arrangement to encourage and focus development, and as things mature, specific parts can be split out into separate packages as deemed useful.
It’d be great for everyone to take it for a spin! Please file issues with discovered bugs, ideas for how things should work, enhancements, and any issues with performance.
It’s worth testing at least, to see if there are any gaps in the new package’s capabilities. It’s a lot less mature, and not as modular if you only need client code and not server functionality. Having the HTTP parser components be in pure Julia instead of C will be useful, if it turns out to be comparably robust and reliable.
I’d say look over the code, compare it to what’s in the existing web stack and judge for yourself whether the old code or the new would be easier to understand and fix if it came to that. Currently if something breaks in HTTPServer.jl and you’re not using that piece, then it doesn’t matter much for you. Whereas with all of the code in a single package, some types of problems even in code you’re not using could get in your way.
Here are a couple of thoughts as the current maintainer of HTTP.jl:
It was started by combining the existing webstack code (Requests, HttpServer, HttpParser), removing quite a bit of code that was duplicated across packages, and simplifying a few APIs
I then went through efforts to modernize the codebase: a lot of the existing webstack code was developed pre-Julia 0.3, and so new features/functionality weren’t being leveraged
I then went through and ported code that we were calling out to fairly simple C libraries over to pure Julia; this has lead to actually some great performance improvements (avoiding the need FFI), as well as increased flexibility (i.e. we can bake cookie detection right into the parser, which the C library doesn’t support).
There have since been efforts to add functionality that was lacking in the old webstack packages: client-side auto session and cookie management, more robust and concurrent server code, a more simple multipart-form API, etc.
Yes, the package is still a bit young, but it’s actively developed and progressing quite quickly. There are definite plans to eventually separate out functionality (server vs. client, uri functionality), but for now, it’s drastically simplified and improved the development process to keep everything together.
I don’t think we need to rush deprecating things for now; in my mind, there’ll probably be a natural transition as the older webstack packages are not very actively maintained or developed, packages w/ dependencies will want to update to get the updated, maintained code in HTTP.jl.
My plan is to continue advertising and talking about it on forums and such to encourage usage. I’m also aiming to address and fix issues with a very quick turn-around to ensure it’s as usable as possible for folks looking to migrate.
For anyone who’s directly interested in migrating a package over, I’m happy to chat and pitch in code to make the change. Feel free to open an issue and I’m happy to help.
I checked out this package and it definitely looks like a step in the right direction (no more type pirating Base.get). However, I can’t figure out what the equivalent to Requests.json is. JSON.parse(HTTP.body(HTTP.get(...))) is failing with a strange error:
julia> JSON.parse(HTTP.body(res))
ERROR: MethodError: Cannot `convert` an object of type Tuple{UInt8,Bool} to an object of type UInt8
This may have arisen from a call to the constructor UInt8(...),
since type constructors fall back to convert methods.
Stacktrace:
[1] byteat at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:51 [inlined]
[2] current at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:62 [inlined]
[3] chomp_space! at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:106 [inlined]
[4] parse_value(::JSON.Parser.StreamingParserState{HTTP.FIFOBuffer}, ::Type{T} where T) at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:149
[5] #parse#2(::Type{Dict{String,Any}}, ::Function, ::HTTP.FIFOBuffer) at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:384
[6] parse(::HTTP.FIFOBuffer) at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:383
Probably the problem is here; either this method is returning the wrong type (should be just UInt8 according to the docstring of read) or this is an appropriate return value and JSON is being too intolerant.
A Response body type is HTTP.FIFOBuffer, a new type that handles asynchronous first-in-first-out byte buffering. There are convenience methods provided to get the response data as a byte vector or string like take!(response) and take!(String, response), which can then be passed to JSON.parse.
Thanks for your input, indeed JSON.parse(take!(String, HTTP.get(...))) works.
I’m aware the same issue existed with Requests, but are there any plans to make take!(String, ...) encoding-aware? ISO 8859-1 is still quite common on the web.
Could you open an issue about it at HTTP.jl? It’d be helpful to have an example response w/ ISO 8859-1, where Base.String falls short/doesn’t work and suggested solution to support it. It seems like something that would be useful in HTTP.jl itself.
@quinnj – What sort of relationship do you anticipate for this package and Mux going forward? Will it take over some of the same higher-level functionality, or is there any plan to migrate Mux from the older set of Http packages to the new one?
Yes, I think it’d be ideal for Mux.jl to switch to using HTTP.jl for the core server functionality. It’s on my list to help migrate packages, but the current preparation for 0.6 release is taking precedence.
Yeah, it still might need a little refinement to get just right, but once I found myself pretty much always needing it in production scripts, I figured it was probably useful enough to bake in.
). However, I can’t figure out what the equivalent to 