Funny Benchmark with Julia (no longer) at the bottom

General Usage Performance
141

I monitored the results of the last runs of the benchmark on the virtual machine, see e.g. https://github.com/jinyus/related_post_gen/blob/main/raw_results.md or https://github.com/jinyus/related_post_gen/blob/697198a2bab8c9b9e2f90fe6569cc6afbc80f772/raw_results.md, what I see is strange to me. Julia is the more performance-inconsistent between runs compared with all the other languages e.g. compare the results of Rust with the ones of Julia for the 5k posts run:

Rust

Benchmark 1: ./target/release/rust
Processing time (w/o IO): 23.541084ms
Processing time (w/o IO): 23.466582ms
Processing time (w/o IO): 23.441882ms
Processing time (w/o IO): 23.468383ms
Processing time (w/o IO): 23.629785ms
Processing time (w/o IO): 23.484583ms
Processing time (w/o IO): 23.434282ms
Processing time (w/o IO): 23.425681ms
Processing time (w/o IO): 23.420081ms
Processing time (w/o IO): 23.649885ms
Processing time (w/o IO): 23.493782ms
Processing time (w/o IO): 23.469783ms
Processing time (w/o IO): 23.456882ms
  Time (mean ± σ):      39.6 ms ±   0.8 ms    [User: 32.2 ms, System: 7.4 ms]
  Range (min … max):    39.1 ms …  41.7 ms    10 runs

Julia

Benchmark 1: julia -O3 --project=Related -e "using Related; main()"
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 25 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 24 milliseconds
  Time (mean ± σ):      3.351 s ±  0.047 s    [User: 3.200 s, System: 0.254 s]
  Range (min … max):    3.280 s …  3.447 s    10 runs

Interestingly this happens also on my machine by running in the REPL using Related; main(), restarting Julia can downgrade performance forever for that REPL session. And restarting again, can make the performance again better.

All other languages except for NodeJS (which has some less large oscillations) don’t exhibit these variations, so I’m asking myself what is special about Julia, the code run on the VM seems to be on this file https://github.com/jinyus/related_post_gen/blob/main/run.sh:

run_julia() {
    echo "Running Julia" &&
        cd ./julia &&
        julia -e 'using Pkg; Pkg.activate("Related"); Pkg.instantiate()' &&
        if [ $HYPER == 1 ]; then

            capture "Julia" hyperfine -r $runs -w $warmup --show-output "julia --project=Related -e \"using Related; main()\""
        else
            command ${time} -f '%es %Mk' julia --project=Related -e "using Related; main()"

        fi

    check_output "related_posts_julia.json"
}

I’m not sure what can cause this difference, do someone have an explaination for this behaviour?

2 Likes