Goose Attack against a Genie App

[StatisticalMouse]

I made a small Genie App, and I created a Goose Attack with Rust that tests it. I used Julia 1.6.1.

The app is essentially (which is routes.jl)

using Genie.Router, Genie.Renderer, Genie.Exceptions, JSON3

route("/") do
  serve_static_file("welcome.html")
end

route("/greeting", named = :greeting) do
  "Welcome to Genie! This site is served by Julia $(Base.VERSION) with $(Threads.nthreads()) threads."
end

route("/random", named = :random) do
 num = 1
 if haskey(@params, :num)
   num = parse(Int64, @params(:num))
 end
 JSON3.write(rand(Int, num))
end

I’m getting many errors like this in Genie

┌ Error: 2021-06-25 13:22:17 Base.IOError("stream is closed or unusable", 0)
└ @ Genie.AppServer ~/.julia/packages/Genie/Uvtzb/src/AppServer.jl:93

The Rust app gets timeouts.

The code for both is at GitHub - StatisticalMice/performance-tests: Web based performance tests

The full HTML report is here GitHub & BitBucket HTML Preview

Comments? Suggestions on improvement?

[StatisticalMouse]

Goose Attack: A Locust-inspired Load Testing Tool In Rust

[pfitzseb]

FWIW, a simple HTTP server (code below) yields

for me, which doesn’t seem too bad.

The biggest difference is disabling logging of those “stream is closed or unusable” errors with e.g.

Logging.global_logger(NullLogger())

As a comparison: “normal” average request time in Genie is 47ms on my machine, whereas disabling the global logger gets it down to 21ms. The HTTP.jl server with logging took something like 35ms on average, IIRC. Not sure where the disparity comes from, tbh, but maybe Genie has some internal log handling.

HTTP server

using HTTP, Sockets, JSON, Logging
using HTTP.URIs: queryparams, URI

const ROUTER = HTTP.Router()

function serveWelcome(req::HTTP.Request)
    return HTTP.Response(200, read(joinpath(@__DIR__, "welcome.html"), String))
end
function serveRandom(req::HTTP.Request)
    num = parse(Int, get(queryparams(URI(req.target)), "num", "0"))

    return HTTP.Response(200, JSON.json(rand(Int, num)))
end

HTTP.@register(ROUTER, "GET", "/", serveWelcome)
HTTP.@register(ROUTER, "GET", "/random", serveRandom)

Logging.global_logger(NullLogger())
HTTP.serve(ROUTER, Sockets.localhost, 8000)

[StatisticalMouse]

@pfitzseb

Logging.global_logger(NullLogger())

I put this at the end of the routes.jl, the log statements went away, but the performance didn’t increase.

[pfitzseb]

I just disabled the logger in the Genie REPL. Not sure why that would be different, but then again this is the first time I’ve used Genie.

[StatisticalMouse]

I tried to put this HTTP server code in the root of my repo, and serve content from GenieApp/public(/welcome.html). I can get it to serve that html page, but all content it tries to load is broken.

[pfitzseb]

Sure, that’s not what it is set up for. If you want a static file server (+ that one /random route), use something like

using HTTP, Sockets, JSON, Logging
using HTTP.URIs: queryparams, URI

const ROUTER = HTTP.Router()

function serveStatic(req::HTTP.Request)
    return HTTP.Response(200, read(joinpath(@__DIR__, "public", split(URI(req.target).path, '/')...), String))
end
function serveRandom(req::HTTP.Request)
    num = parse(Int, get(queryparams(URI(req.target)), "num", "0"))

    return HTTP.Response(200, JSON.json(rand(Int, num)))
end

HTTP.@register(ROUTER, "GET", "/random", serveRandom)
HTTP.@register(ROUTER, "GET", "/*", serveStatic)

Logging.global_logger(NullLogger())

HTTP.serve(ROUTER, Sockets.localhost, 8000)

[StatisticalMouse]

I’m using a different version:

using HTTP, Sockets, JSON3, Logging
using HTTP.URIs: queryparams, URI

const ROUTER = HTTP.Router()

function serveWelcome(req::HTTP.Request)
    return HTTP.Response(200, "Welcome to Genie! This site is served by Julia $(Base.VERSION) with $(Threads.nthreads()) threads.")
end

function serveRandom(req::HTTP.Request)
    num = parse(Int, get(queryparams(URI(req.target)), "num", "1"))

    return HTTP.Response(200, JSON3.write(rand(Int, num)))
end

HTTP.@register(ROUTER, "GET", "/", serveWelcome)
HTTP.@register(ROUTER, "GET", "/random", serveRandom)

Logging.global_logger(NullLogger())
HTTP.serve(ROUTER, Sockets.localhost, 8000)

It’s still not working properly, though. I’m running on a Mac, which might affect things.
It’s in git.

[StatisticalMouse]

The performance I get with this HTTP.jl version with code I have in git is

[pfitzseb]

Ok, that looks much better, no? If you disregard the 99th and 100th percentile (or compile your code by hitting each route manually before starting the benchmark) then you get pretty good times.

[StatisticalMouse]

The normal speed is better, but I still get a lot of errors during execution.

I added a Python Flask app to the git repo. It also produces connection errors on the Rust side. This makes me think that this is a problem with my Mac.

[StatisticalMouse]

Most curious. The Python Flask app stopped producing any errors when I totally turned off the Mullvad VPN client. This didn’t help the two Julia apps, which still produce errors.

[StatisticalMouse]

The test is running 30 concurrent users, which should not be too much for either my Mac or Julia, so something odd is happening.

If I use one client, there are no errors visible to goose connecting to Julia, but if I use two, there are.

I think it’s quite possible that the fast speed is actually too fast for the Mac.

[jzr]

IME, a single “hello world” response in a Genie app takes 15 ms compared to 5ms for both Flask and HTTP.jl. I don’t know why.

[essenciary]

Great stuff, thanks for setting these up and sharing!

Can you please try Genie v2 (currently on #master) it should be faster.

I’m also just cloned the repo and running the tests locally. But I’m curious what it looks like on your machine.

[StatisticalMouse]

The problem was Mac. I re-ran the tests on Google GCP on a e2-standard-4 (4 vCPUs, 16 GB memory). The VM is running Debian 10.
Flask:

Genie.jl:

HTTP.jl:

The error messages from Genie were still coming until I turned off all logging. The test was run without logging.

[StatisticalMouse]

Flask is actually doing pretty well with 52 ms average time. (It can probably be optimised further.)

HTTP.jl wins with roughly 6.6 ms average against Genie.jl’s 22 ms.

[essenciary]

With Genie 1.18 configured for production env (no Revise, no logging, etc) the timing on my machine is about identical with the timing for HTTP.jl (but both much worse than the ones on your server, I’m running on a Windows laptop):

Genie prod:

HTTP.jl

Can you please try my Genie fork configured for prod on your server?

---

I’m not sure about the errors, seem to be coming from Rust, Goose outputs a lot of:

21:58:29 [WARN] "/": error sending request for url (http://127.0.0.1:8000/): error trying to connect: tcp connect error: Only one usage of each socket address (protocol/network address/port) is normally permitted. (os error 10048)

[StatisticalMouse]

Please make a PR.
Mac was giving different errors, but similar.

[essenciary]

Using Genie’s own HTML and JSON renderers it actually beats HTTP.jl times on my machine, so it looks like that’s the best way to run Genie

Genie prod with own renderers