# ANN: New HTTP.jl package

**URL:** <https://discourse.julialang.org/t/ann-new-http-jl-package/2048>\
**Category:** Community\
**Tags:** package, web\
**Created:** [February 12, 2017, 12:30am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048 "2017-02-12T00:30:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [February 12, 2017, 12:30am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/1 "2017-02-12T00:30:59Z")

</div>

Hey everyone!

Good news for interfacing with the web in Julia: a new package HTTP.jl has been released (technically, re-released after an original package from 3 years ago was deprecated).

Julia’s webstack has languished a bit as of late and this is a concerted effort to revive, modernize, and improve the state of webstack code for Julia. HTTP.jl is actually a consolidation of the various web packages to ensure code consistency, reduction in code duplication, and to streamline the modernization process. Certain packages have gone through complete re-writes (URIParser.jl, Requests.jl, HttpServer.jl), with also an effort to remove certain binary dependencies (HttpParser.jl, rewritten in native Julia).

The consolidation is a short-term arrangement to encourage and focus development, and as things mature, specific parts can be split out into separate packages as deemed useful.

It’d be great for everyone to take it for a spin! Please [file issues](https://github.com/JuliaWeb/HTTP.jl/issues) with discovered bugs, ideas for how things should work, enhancements, and any issues with performance.

Cheers!

-Jacob

---

<div class="post-metadata">

**Author:** ![tkelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkelman/32/692_2.png) [@tkelman](https://discourse.julialang.org/u/tkelman)\
**Post date:** [February 12, 2017, 12:40am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/2 "2017-02-12T00:40:51Z")

</div>

Reduced code duplication, by duplicating code…

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [February 12, 2017, 1:00am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/3 "2017-02-12T01:00:19Z")

</div>

Well, it actually involved way more rewriting and throwing out than actually re-using. 😛

---

<div class="post-metadata">

**Author:** ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)\
**Post date:** [March 15, 2017, 4:07pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/4 "2017-03-15T16:07:42Z")

</div>

Sounds very cool. I left my laptop in the office and limited to my iPhone. This is what the docs look like in Chrome? Anything we can do about it?

 ![](https://global.discourse-cdn.com/julialang/original/3X/8/8/88953f951e9a106ac9cbfd85db449b7a83d48f58.png)

---

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [March 15, 2017, 9:12pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/5 "2017-03-15T21:12:19Z")

</div>

So are folks intended to use HTTP.jl in place of Requests.jl, HttpServer.jl, etc. from here on out?

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 16, 2017, 5:34am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/6 "2017-03-16T05:34:08Z")

</div>

I second that question. Should I migrate any code that uses `Requests` to `HTTP`?

---

<div class="post-metadata">

**Author:** ![tkelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkelman/32/692_2.png) [@tkelman](https://discourse.julialang.org/u/tkelman)\
**Post date:** [March 16, 2017, 7:04am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/7 "2017-03-16T07:04:16Z")

</div>

It’s worth testing at least, to see if there are any gaps in the new package’s capabilities. It’s a lot less mature, and not as modular if you only need client code and not server functionality. Having the HTTP parser components be in pure Julia instead of C will be useful, if it turns out to be comparably robust and reliable.

I’d say look over the code, compare it to what’s in the existing web stack and judge for yourself whether the old code or the new would be easier to understand and fix if it came to that. Currently if something breaks in HTTPServer.jl and you’re not using that piece, then it doesn’t matter much for you. Whereas with all of the code in a single package, some types of problems even in code you’re not using could get in your way.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 16, 2017, 4:01pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/8 "2017-03-16T16:01:56Z")

</div>

Here are a couple of thoughts as the current maintainer of HTTP.jl:

- It was started by combining the existing webstack code (Requests, HttpServer, HttpParser), removing quite a bit of code that was duplicated across packages, and simplifying a few APIs
- I then went through efforts to modernize the codebase: a lot of the existing webstack code was developed pre-Julia 0.3, and so new features/functionality weren’t being leveraged
- I then went through and ported code that we were calling out to fairly simple C libraries over to pure Julia; this has lead to actually some great performance improvements (avoiding the need FFI), as well as increased flexibility (i.e. we can bake cookie detection right into the parser, which the C library doesn’t support).
- There have since been efforts to add functionality that was lacking in the old webstack packages: client-side auto session and cookie management, more robust and concurrent server code, a more simple multipart-form API, etc.

Yes, the package is still a bit young, but it’s actively developed and progressing quite quickly. There are definite plans to eventually separate out functionality (server vs. client, uri functionality), but for now, it’s drastically simplified and improved the development process to keep everything together.

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 16, 2017, 6:27pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/9 "2017-03-16T18:27:39Z")

</div>

Removing the C dependencies sounds like a big step in the right direction. I will certainly check this out.

Are there plans to eventually deprecate Requests, HttpServer, HttpParser, etc. in favor of HTTP?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 16, 2017, 7:36pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/10 "2017-03-16T19:36:46Z")

</div>

I don’t think we need to rush deprecating things for now; in my mind, there’ll probably be a natural transition as the older webstack packages are not very actively maintained or developed, packages w/ dependencies will want to update to get the updated, maintained code in HTTP.jl.

My plan is to continue advertising and talking about it on forums and such to encourage usage. I’m also aiming to address and fix issues with a very quick turn-around to ensure it’s as usable as possible for folks looking to migrate.

For anyone who’s directly interested in migrating a package over, I’m happy to chat and pitch in code to make the change. Feel free to [open an issue](https://github.com/JuliaWeb/HTTP.jl/issues/new) and I’m happy to help.

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 17, 2017, 4:11am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/11 "2017-03-17T04:11:18Z")

</div>

I checked out this package and it definitely looks like a step in the right direction (no more type pirating `Base.get` 😄). However, I can’t figure out what the equivalent to `Requests.json` is. `JSON.parse(HTTP.body(HTTP.get(...)))` is failing with a strange error:

```julia
julia> JSON.parse(HTTP.body(res))
ERROR: MethodError: Cannot `convert` an object of type Tuple{UInt8,Bool} to an object of type UInt8
This may have arisen from a call to the constructor UInt8(...),
since type constructors fall back to convert methods.
Stacktrace:
 [1] byteat at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:51 [inlined]
 [2] current at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:62 [inlined]
 [3] chomp_space! at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:106 [inlined]
 [4] parse_value(::JSON.Parser.StreamingParserState{HTTP.FIFOBuffer}, ::Type{T} where T) at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:149
 [5] #parse#2(::Type{Dict{String,Any}}, ::Function, ::HTTP.FIFOBuffer) at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:384
 [6] parse(::HTTP.FIFOBuffer) at /home/fengyang/.julia/v0.6/JSON/src/Parser.jl:383

```

Probably the problem is [here](https://github.com/JuliaWeb/HTTP.jl/blob/f2f68f1a5f788c4cb98a00d59f5e8fa3ffb9791d/src/fifobuffer.jl#L161); either this method is returning the wrong type (should be just `UInt8` according to the docstring of `read`) or this is an appropriate return value and `JSON` is being too intolerant.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 17, 2017, 4:14am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/12 "2017-03-17T04:14:11Z")

</div>

A Response body type is `HTTP.FIFOBuffer`, a new type that handles asynchronous first-in-first-out byte buffering. There are convenience methods provided to get the response data as a byte vector or string like `take!(response)` and `take!(String, response)`, which can then be passed to JSON.parse.

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 17, 2017, 4:35am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/13 "2017-03-17T04:35:57Z")

</div>

Thanks for your input, indeed `JSON.parse(take!(String, HTTP.get(...)))` works.

I’m aware the same issue existed with Requests, but are there any plans to make `take!(String, ...)` encoding-aware? ISO 8859-1 is still quite common on the web.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 17, 2017, 2:43pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/14 "2017-03-17T14:43:06Z")

</div>

Could you open an issue about it at HTTP.jl? It’d be helpful to have an example response w/ ISO 8859-1, where Base.String falls short/doesn’t work and suggested solution to support it. It seems like something that would be useful in HTTP.jl itself.

---

<div class="post-metadata">

**Author:** ![malmaud](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/malmaud/32/29_2.png) [@malmaud](https://discourse.julialang.org/u/malmaud)\
**Post date:** [March 17, 2017, 4:02pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/15 "2017-03-17T16:02:34Z")

</div>

Probably couldn’t hurt to add a convenience method for this common task.

---

<div class="post-metadata">

**Author:** ![fengyang.wang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fengyang.wang/32/104_2.png) [@fengyang.wang](https://discourse.julialang.org/u/fengyang.wang)\
**Post date:** [March 17, 2017, 4:53pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/16 "2017-03-17T16:53:07Z")

</div>

Sure, I just opened

[https://github.com/JuliaWeb/HTTP.jl/issues/38](https://github.com/JuliaWeb/HTTP.jl/issues/38)

---

<div class="post-metadata">

**Author:** ![amellnik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amellnik/32/137_2.png) [@amellnik](https://discourse.julialang.org/u/amellnik)\
**Post date:** [March 26, 2017, 2:57am UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/17 "2017-03-26T02:57:16Z")

</div>

@quinnj – What sort of relationship do you anticipate for this package and Mux going forward? Will it take over some of the same higher-level functionality, or is there any plan to migrate Mux from the older set of Http packages to the new one?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 27, 2017, 3:13pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/18 "2017-03-27T15:13:11Z")

</div>

Yes, I think it’d be ideal for Mux.jl to switch to using HTTP.jl for the core server functionality. It’s on my list to help migrate packages, but the current preparation for 0.6 release is taking precedence.

---

<div class="post-metadata">

**Author:** ![jwu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jwu/32/3775_2.png) [@jwu](https://discourse.julialang.org/u/jwu)\
**Post date:** [March 27, 2017, 8:51pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/19 "2017-03-27T20:51:27Z")

</div>

after seeing the retry parameter, I added a star immediately!🙂

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [March 27, 2017, 8:58pm UTC](https://discourse.julialang.org/t/ann-new-http-jl-package/2048/20 "2017-03-27T20:58:00Z")

</div>

Yeah, it still might need a little refinement to get just right, but once I found myself pretty much always needing it in production scripts, I figured it was probably useful enough to bake in. 🙂

[Next page](https://discourse.julialang.org/t/ann-new-http-jl-package/2048.md?page=2)