# TechEmpower/FrameworkBenchmarks R20 deadline 12/28 + first julia "JSON serialization" results

**URL:** <https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936>\
**Category:** Web Stack\
**Tags:** benchmark, webapps\
**Created:** [December 16, 2020, 3:17pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936 "2020-12-16T15:17:40Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [December 16, 2020, 3:17pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/1 "2020-12-16T15:17:40Z")

</div>

TLDR:

- [Techempower Framework benchmarks](https://www.techempower.com/benchmarks/) is a collaborative, open web framework benchmarking project
- The next release: [Round 20 "Last day for PR review is 12/28. "](https://github.com/TechEmpower/FrameworkBenchmarks/issues/5709#issuecomment-741995793)
- First Julia results ( “http.jl” ) … current **“rank: 149/166”** not the best…
- 🙂 So if anybody has an experience in performance tuning ;
  - then please check the current code. ( [http.jl](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Julia/Http.jl) )
  - and/or please [add a new/better/alternative Julia code](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Julia)

Details:

The good news, somebody ( not me ) has been added the first Julia **“JSON Serialization”** test to the TechEmpower/FrameworkBenchmarks  
and the latest Unofficial( [continuous benchmarking](https://tfb-status.techempower.com/) ) results can be seen here:

- [TechEmpower Framework Benchmarks](https://www.techempower.com/benchmarks/#section=test&runid=996d0239-a18a-4054-b496-3506868f0723&hw=ph&test=json&a=2)

![image](https://global.discourse-cdn.com/julialang/original/3X/d/4/d4db3d36111cd61b58ba92a2fd2ba33d04ed7da9.png)

- rank: 149 ( from 166 )
- 1.3% performance ( compare the best results )

If I check only the “Julia+Python” results ( see **“Show filters panel”** )

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/4/f43ab6e63d243d9644f83bbf4f06cdcea6c9f2c8.png)

then ( Julia+Python) results:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/3/1/3101c4c2bf06e308ee9f3c54d4aea8624dc514b7.png)

---

<div class="post-metadata">

**Author:** ![fonsp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fonsp/32/222349_2.png) [@fonsp](https://discourse.julialang.org/u/fonsp)\
**Post date:** [December 16, 2020, 8:59pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/2 "2020-12-16T20:59:02Z")

</div>

The first response will be extremely slow because of the JIT compiler, is that time included in the results?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [December 16, 2020, 9:33pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/3 "2020-12-16T21:33:49Z")

</div>

It’s using the ancient but functional JSON.jl  
It really should be using the modern and faster JSON3.jl,  
especially since the message to serve has a fixed structure  
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/8a0612c6154c11f483608b0879aa8b3ed8f21793/frameworks/Julia/Http.jl/server.jl](https://github.com/TechEmpower/FrameworkBenchmarks/blob/8a0612c6154c11f483608b0879aa8b3ed8f21793/frameworks/Julia/Http.jl/server.jl)

Also this really should be using a Manifest and a Project.toml  
Or at least specifying versions during `add`  
[https://github.com/TechEmpower/FrameworkBenchmarks/blob/8a0612c6154c11f483608b0879aa8b3ed8f21793/frameworks/Julia/Http.jl/http-jl.dockerfile#L25-L29](https://github.com/TechEmpower/FrameworkBenchmarks/blob/8a0612c6154c11f483608b0879aa8b3ed8f21793/frameworks/Julia/Http.jl/http-jl.dockerfile#L25-L29)

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 16, 2020, 9:40pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/4 "2020-12-16T21:40:54Z")

</div>

How much does that improve the speed?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [December 16, 2020, 10:13pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/5 "2020-12-16T22:13:29Z")

</div>

It’s about 5x faster.

```julia
julia> @btime JSON3.write((;message ="Hello, World!"))
  134.465 ns (5 allocations: 304 bytes)
"{\"message\":\"Hello, World!\"}"

julia> @btime JSON.json(Dict(:message => "Hello, World!"))
  665.146 ns (11 allocations: 944 bytes)
"{\"message\":\"Hello, World!\"}"

```

though wether it is JSON or HTTP dominatings idk.

---

<div class="post-metadata">

**Author:** ![mcmcgrath13](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcmcgrath13/32/7788_2.png) [@mcmcgrath13](https://discourse.julialang.org/u/mcmcgrath13)\
**Post date:** [December 16, 2020, 11:01pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/6 "2020-12-16T23:01:20Z")

</div>

It also looks like it might be single core/thread based on the run.sh

could try changing that to:

```julia
for i in $(seq 0 $(($(nproc --all)-1))); do
	julia --threads auto server.jl &
done

while : ; do sleep 1 ; done

```

---

<div class="post-metadata">

**Author:** ![mcmcgrath13](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcmcgrath13/32/7788_2.png) [@mcmcgrath13](https://discourse.julialang.org/u/mcmcgrath13)\
**Post date:** [December 16, 2020, 11:45pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/7 "2020-12-16T23:45:50Z")

</div>

I started a PR based on the comments here: [https://github.com/TechEmpower/FrameworkBenchmarks/pull/6215](https://github.com/TechEmpower/FrameworkBenchmarks/pull/6215)

Locally, I saw a 5-10x performance improvement.

Let me know if there’s anything else that could be improved! There is a warmup before the real benchmark, so I don’t think we’re being penalized for the JIT compile time.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [December 17, 2020, 12:33am UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/8 "2020-12-17T00:33:21Z")

</div>

> [@mcmcgrath13](#):
>
> I started a PR based on the comments here:

Thanks!

> [@mcmcgrath13](#):
>
> Let me know if there’s anything else that could be improved!

maybe we can use [the latest stable Julia](https://julialang.org/downloads/) ( in the Dockerfile ) … `julia-1.5.2` → `julia-1.5.3`

---

<div class="post-metadata">

**Author:** ![mcmcgrath13](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcmcgrath13/32/7788_2.png) [@mcmcgrath13](https://discourse.julialang.org/u/mcmcgrath13)\
**Post date:** [December 26, 2020, 3:56pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/9 "2020-12-26T15:56:05Z")

</div>

The PR was merged. The [latest preliminary results](https://www.techempower.com/benchmarks/#section=test&runid=3ac967d8-1894-4158-b981-24f6eb768112&hw=ph&test=json&a=2) show a 10x improvement!

 ![image](https://global.discourse-cdn.com/julialang/original/3X/9/b/9bda3914bd2a33992c8e794b20ad6c6d7aba9d3b.png)

There are still a couple days left for PRs for round 20 if anyone has more ideas for improvements.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [December 26, 2020, 11:58pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/10 "2020-12-26T23:58:01Z")

</div>

Thanks!

> [@mcmcgrath13](#):
>
> if anyone has more ideas for improvements.

as I see the other frameworks using all dirty/legal tricks 🙂

- march=native [https://github.com/TechEmpower/FrameworkBenchmarks/search?q=march%3Dnative&type=code](https://github.com/TechEmpower/FrameworkBenchmarks/search?q=march%3Dnative&type=code)

- [NIM httpbeast: ` -d:danger`](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Nim/httpbeast/httpbeast.dockerfile#L12) ( _“danger: **Turns off all runtime checks** and turns on the optimizer.”_)

so maybe some of the `julia --help` parameters can be useful … for example

- `--optimize=3`
- `--cpu-target=native`
- `-g0` ( no debug information )
- `--check-bounds=no`
- `--inline=yes`
- `--math-mode=fast`
- `--code-coverage=none` ( default : “user” )
- `--track-allocation=none` ( default : “user” )

( I have not tested ; just checked the parameters … so don’t follow my suggestions blindly )

important: the Citrine test hardware is [“28 HT Cores Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz”](https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Environment)

- [ark.intel.com - intel-xeon-gold-5120-processor](https://ark.intel.com/content/www/us/en/ark/products/120474/intel-xeon-gold-5120-processor-19-25m-cache-2-20-ghz.html) ( Skylake Q3’17 AVX-512 )

As I see now 2 test has been implemented ( all=7) :

- JSON serialization (214,458)
- Plaintext (242,892) ( ‘a “Hello, World” message rendered as plain text.’ )

so it is a good progress 🙂

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [December 30, 2020, 2:57am UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/11 "2020-12-30T02:57:15Z")

</div>

R20 Gen has been started

- [https://github.com/TechEmpower/FrameworkBenchmarks/issues/5709#issuecomment-752213443](https://github.com/TechEmpower/FrameworkBenchmarks/issues/5709#issuecomment-752213443)

so I am close this thread …  
Continue in R21 … 🙂

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [February 8, 2021, 6:27pm UTC](https://discourse.julialang.org/t/techempower-frameworkbenchmarks-r20-deadline-12-28-first-julia-json-serialization-results/51936/12 "2021-02-08T18:27:54Z")

</div>

R20 -(2021-02-08) Released :

- see “http.jl” [https://www.techempower.com/benchmarks/#section=data-r20](https://www.techempower.com/benchmarks/#section=data-r20)  
\* in the “JSON serialisation” + " Plaintext" tab

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/7/c750f4b61ae4ab58df0dfe59267d8bd6745601db.png)