TechEmpower/FrameworkBenchmarks R20 deadline 12/28 + first julia "JSON serialization" results

TLDR:

Details:

The good news, somebody ( not me ) has been added the first Julia “JSON Serialization” test to the TechEmpower/FrameworkBenchmarks
and the latest Unofficial( continuous benchmarking ) results can be seen here:

image

  • rank: 149 ( from 166 )
  • 1.3% performance ( compare the best results )

If I check only the “Julia+Python” results ( see “Show filters panel”)

then ( Julia+Python) results:

3 Likes

The first response will be extremely slow because of the JIT compiler, is that time included in the results?

It’s using the ancient but functional JSON.jl
It really should be using the modern and faster JSON3.jl,
especially since the message to serve has a fixed structure

Also this really should be using a Manifest and a Project.toml
Or at least specifying versions during add

3 Likes

How much does that improve the speed?

It’s about 5x faster.

julia> @btime JSON3.write((;message ="Hello, World!"))
  134.465 ns (5 allocations: 304 bytes)
"{\"message\":\"Hello, World!\"}"

julia> @btime JSON.json(Dict(:message => "Hello, World!"))
  665.146 ns (11 allocations: 944 bytes)
"{\"message\":\"Hello, World!\"}"

though wether it is JSON or HTTP dominatings idk.

8 Likes

It also looks like it might be single core/thread based on the run.sh

could try changing that to:

for i in $(seq 0 $(($(nproc --all)-1))); do
	julia --threads auto server.jl &
done

while : ; do sleep 1 ; done
3 Likes

I started a PR based on the comments here: https://github.com/TechEmpower/FrameworkBenchmarks/pull/6215

Locally, I saw a 5-10x performance improvement.

Let me know if there’s anything else that could be improved! There is a warmup before the real benchmark, so I don’t think we’re being penalized for the JIT compile time.

9 Likes

Thanks!

maybe we can use the latest stable Julia ( in the Dockerfile ) … julia-1.5.2 --> julia-1.5.3

1 Like

The PR was merged. The latest preliminary results show a 10x improvement!

There are still a couple days left for PRs for round 20 if anyone has more ideas for improvements.

3 Likes

Thanks!

as I see the other frameworks using all dirty/legal tricks :slight_smile:

so maybe some of the julia --help parameters can be useful … for example

  • --optimize=3
  • --cpu-target=native
  • -g0 ( no debug information )
  • --check-bounds=no
  • --inline=yes
  • --math-mode=fast
  • --code-coverage=none ( default : “user” )
  • --track-allocation=none ( default : “user” )

( I have not tested ; just checked the parameters … so don’t follow my suggestions blindly )

important: the Citrine test hardware is “28 HT Cores Intel® Xeon® Gold 5120 CPU @ 2.20GHz”

As I see now 2 test has been implemented ( all=7) :

  • JSON serialization (214,458)
  • Plaintext (242,892) ( ‘a “Hello, World” message rendered as plain text.’ )

so it is a good progress :slight_smile:

1 Like

R20 Gen has been started

so I am close this thread …
Continue in R21 … :slight_smile:

2 Likes

R20 -(2021-02-08) Released :