TechEmpower/FrameworkBenchmarks R20 deadline 12/28 + first julia "JSON serialization" results

Specific Domains Web Stack
,
1

TLDR:

Details:

The good news, somebody ( not me ) has been added the first Julia “JSON Serialization” test to the TechEmpower/FrameworkBenchmarks
and the latest Unofficial( continuous benchmarking ) results can be seen here:

image

  • rank: 149 ( from 166 )
  • 1.3% performance ( compare the best results )

If I check only the “Julia+Python” results ( see “Show filters panel”)

image
image704×287 34 KB

then ( Julia+Python) results:

image
image1455×708 187 KB

3 Likes
2

The first response will be extremely slow because of the JIT compiler, is that time included in the results?

3

It’s using the ancient but functional JSON.jl
It really should be using the modern and faster JSON3.jl,
especially since the message to serve has a fixed structure
https://github.com/TechEmpower/FrameworkBenchmarks/blob/8a0612c6154c11f483608b0879aa8b3ed8f21793/frameworks/Julia/Http.jl/server.jl

Also this really should be using a Manifest and a Project.toml
Or at least specifying versions during add
https://github.com/TechEmpower/FrameworkBenchmarks/blob/8a0612c6154c11f483608b0879aa8b3ed8f21793/frameworks/Julia/Http.jl/http-jl.dockerfile#L25-L29

3 Likes
4

How much does that improve the speed?

5

It’s about 5x faster.

julia> @btime JSON3.write((;message ="Hello, World!"))
  134.465 ns (5 allocations: 304 bytes)
"{\"message\":\"Hello, World!\"}"

julia> @btime JSON.json(Dict(:message => "Hello, World!"))
  665.146 ns (11 allocations: 944 bytes)
"{\"message\":\"Hello, World!\"}"

though wether it is JSON or HTTP dominatings idk.

8 Likes
6

It also looks like it might be single core/thread based on the run.sh

could try changing that to:

for i in $(seq 0 $(($(nproc --all)-1))); do
	julia --threads auto server.jl &
done

while : ; do sleep 1 ; done
3 Likes
7

I started a PR based on the comments here: https://github.com/TechEmpower/FrameworkBenchmarks/pull/6215

Locally, I saw a 5-10x performance improvement.

Let me know if there’s anything else that could be improved! There is a warmup before the real benchmark, so I don’t think we’re being penalized for the JIT compile time.

9 Likes
8

Thanks!

maybe we can use the latest stable Julia ( in the Dockerfile ) … julia-1.5.2 → julia-1.5.3

1 Like
9

The PR was merged. The latest preliminary results show a 10x improvement!

image
image856×52 5.4 KB

There are still a couple days left for PRs for round 20 if anyone has more ideas for improvements.

3 Likes
10

Thanks!

as I see the other frameworks using all dirty/legal tricks :slight_smile:

so maybe some of the julia --help parameters can be useful … for example

  • --optimize=3
  • --cpu-target=native
  • -g0 ( no debug information )
  • --check-bounds=no
  • --inline=yes
  • --math-mode=fast
  • --code-coverage=none ( default : “user” )
  • --track-allocation=none ( default : “user” )

( I have not tested ; just checked the parameters … so don’t follow my suggestions blindly )

important: the Citrine test hardware is “28 HT Cores Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz”

As I see now 2 test has been implemented ( all=7) :

  • JSON serialization (214,458)
  • Plaintext (242,892) ( ‘a “Hello, World” message rendered as plain text.’ )

so it is a good progress :slight_smile:

1 Like
11

R20 Gen has been started

so I am close this thread …
Continue in R21 … :slight_smile:

2 Likes
12

R20 -(2021-02-08) Released :

image
image1411×458 99.4 KB