Julia HTTP.jl performance vs python requests

General Usage
1

Hello,

I am currently trying to port some of my python code to Julia.
And after some benchmarking, I’ve noticed that Julia’s HTTP.jl performance is somewhat worse than python’s requests (370ms vs 240ms). This is probably due to the fact that I use “persistent connection” in python via requests.session and just plain HTTP requests for HTTP.jl. So, is there any way to get a similar performance in Julia?

2

Try the Downloads package (a standard library since Julia 1.6). It wraps libcurl and should be quite efficient. If you want to download multiple things concurrently, you can use tasks like this:

using Downloads: download

@sync for url in urls
    @async download(url)
end

This example isn’t terribly useful as is since it downloads each url to a temporary file and then discards the path of that file, but hopefully you get the idea.

2 Likes