# \[RFC-ANN\] SystemBenchmark.jl

**URL:** <https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982>\
**Category:** Package Announcements\
**Tags:** performance, benchmark\
**Created:** [May 7, 2020, 7:25pm UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982 "2020-05-07T19:25:44Z")\
**Posts on this page:** 6\
**Page:** 3

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [May 9, 2020, 10:47pm UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982/42 "2020-05-09T22:47:49Z")

</div>

Small issue when testing a system without a discrete GPU:  
julia\> comparison\_df = compareToRef(sysbenchmark())  
[ Info: CuArrays.functional() == false. No usable GPU detected  
Disk IO tests100%|█████████████████████████████████████████| Time: 0:01:06  
ERROR: ArgumentError: Column names :ms and :reswere found in only one of the passed data frames and passed `cols=:setequal`  
Stacktrace:  
[1] append!(::DataFrames.DataFrame, ::DataFrames.DataFrame; cols::Symbol) at /home/ufechner/.julia/packages/DataFrames/S3ZFo/src/dataframe/dataframe.jl:1222  
[2] append! at /home/ufechner/.julia/packages/DataFrames/S3ZFo/src/dataframe/dataframe.jl:1191 [inlined]  
[3] sysbenchmark(; printsysinfo::Bool) at /home/ufechner/.julia/packages/SystemBenchmark/SXV31/src/SystemBenchmark.jl:125  
[4] sysbenchmark() at /home/ufechner/.julia/packages/SystemBenchmark/SXV31/src/SystemBenchmark.jl:78  
[5] top-level scope at REPL[3]:1

julia\>

---

<div class="post-metadata">

**Author:** ![ianshmean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ianshmean/32/216042_2.png) [@ianshmean](https://discourse.julialang.org/u/ianshmean)\
**Post date:** [May 9, 2020, 10:50pm UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982/43 "2020-05-09T22:50:04Z")

</div>

Yes, sorry, just fixed that on master. Will register and release proper versions soon

---

<div class="post-metadata">

**Author:** ![imcinerney](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imcinerney/32/9882_2.png) [@imcinerney](https://discourse.julialang.org/u/imcinerney)\
**Post date:** [May 9, 2020, 11:25pm UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982/44 "2020-05-09T23:25:21Z")

</div>

After some work, I was able to get the benchmarks from several ARM boards I had sitting around:

Raspberry Pi 3B+ (aarch64 - Quad-core Cortex A-53):

```julia
Reference system ----------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
  GPU: GeForce GTX 1650 with Max-Q Design

Test system ---------------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (aarch64-unknown-linux-gnu)
  CPU: unknown
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, cortex-a53)

14×5 DataFrames.DataFrame
│ Row │ cat │ testname │ ref_ms │ test_ms │ factor │
│ │ String │ String │ Float64 │ Float64 │ Float64 │
├─────┼─────────────┼─────────────────┼─────────────┼─────────────┼─────────┤
│ 1 │ cpu │ FloatMul │ 1.134e-6 │ 5.104e-6 │ 4.50088 │
│ 2 │ cpu │ FloatSin │ 4.048e-6 │ 5.51524e-5 │ 13.6246 │
│ 3 │ cpu │ VecMulBroad │ 2.94935e-5 │ 0.000324703 │ 11.0093 │
│ 4 │ cpu │ CPUMatMul │ 0.018796 │ 0.207395 │ 11.034 │
│ 5 │ cpu │ MatMulBroad │ 0.0042468 │ 0.033177 │ 7.81224 │
│ 6 │ cpu │ 3DMulBroad │ 0.0010416 │ 0.008993 │ 8.63383 │
│ 7 │ cpu │ FFMPEGH264Write │ 105.579 │ 1013.53 │ 9.59979 │
│ 8 │ mem │ DeepCopy │ 0.000177585 │ 0.0016198 │ 9.12129 │
│ 9 │ diskio │ DiskWrite1KB │ 0.031872 │ 1.2978 │ 40.7192 │
│ 10 │ diskio │ DiskWrite1MB │ 0.650659 │ 57.1556 │ 87.8427 │
│ 11 │ diskio │ DiskRead1KB │ 0.00681275 │ 0.134114 │ 19.6857 │
│ 12 │ diskio │ DiskRead1MB │ 0.144945 │ 1.97855 │ 13.6504 │
│ 13 │ loading │ JuliaLoad │ 91.1325 │ 790.011 │ 8.66882 │
│ 14 │ compilation │ compilecache │ 110.3 │ 870.46 │ 7.89177 │

```

Xilinx Pynq-v1 (armv7l - Zynq 7020 with dual-core Cortex A9):

```julia
Reference system ----------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
  GPU: GeForce GTX 1650 with Max-Q Design

Test system ---------------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (arm-linux-gnueabihf)
  CPU: ARMv7 Processor rev 0 (v7l)
  WORD_SIZE: 32
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, cortex-a9)

14×5 DataFrames.DataFrame
│ Row │ cat │ testname │ ref_ms │ test_ms │ factor │
│ │ String │ String │ Float64 │ Float64 │ Float64 │
├─────┼─────────────┼─────────────────┼─────────────┼─────────────┼─────────┤
│ 1 │ cpu │ FloatMul │ 1.134e-6 │ 1.98255e-5 │ 17.4828 │
│ 2 │ cpu │ FloatSin │ 4.048e-6 │ 0.0001094 │ 27.0258 │
│ 3 │ cpu │ VecMulBroad │ 2.94935e-5 │ 0.000971211 │ 32.9297 │
│ 4 │ cpu │ CPUMatMul │ 0.018796 │ 1.2337 │ 65.6363 │
│ 5 │ cpu │ MatMulBroad │ 0.0042468 │ 0.156975 │ 36.9631 │
│ 6 │ cpu │ 3DMulBroad │ 0.0010416 │ 0.025769 │ 24.7398 │
│ 7 │ cpu │ FFMPEGH264Write │ 105.579 │ 3097.85 │ 29.3416 │
│ 8 │ mem │ DeepCopy │ 0.000177585 │ 0.0049185 │ 27.6967 │
│ 9 │ diskio │ DiskWrite1KB │ 0.031872 │ 1.15174 │ 36.1364 │
│ 10 │ diskio │ DiskWrite1MB │ 0.650659 │ 52.8955 │ 81.2953 │
│ 11 │ diskio │ DiskRead1KB │ 0.00681275 │ 0.143813 │ 21.1093 │
│ 12 │ diskio │ DiskRead1MB │ 0.144945 │ 4.3458 │ 29.9824 │
│ 13 │ loading │ JuliaLoad │ 91.1325 │ 1444.04 │ 15.8455 │
│ 14 │ compilation │ compilecache │ 110.3 │ 2714.8 │ 24.6129 │

```

Beaglebone Black (armv7l - Single core Cortex A8):

```julia
Reference system ----------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
  GPU: GeForce GTX 1650 with Max-Q Design

Test system ---------------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (arm-linux-gnueabihf)
  CPU: ARMv7 Processor rev 2 (v7l)
  WORD_SIZE: 32
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, cortex-a8)

14×5 DataFrames.DataFrame
│ Row │ cat │ testname │ ref_ms │ test_ms │ factor │
│ │ String │ String │ Float64 │ Float64 │ Float64 │
├─────┼─────────────┼─────────────────┼─────────────┼─────────────┼─────────┤
│ 1 │ cpu │ FloatMul │ 1.134e-6 │ 2.21073e-5 │ 19.495 │
│ 2 │ cpu │ FloatSin │ 4.048e-6 │ 0.000355889 │ 87.9172 │
│ 3 │ cpu │ VecMulBroad │ 2.94935e-5 │ 0.00128817 │ 43.6763 │
│ 4 │ cpu │ CPUMatMul │ 0.018796 │ 21.7096 │ 1155.01 │
│ 5 │ cpu │ MatMulBroad │ 0.0042468 │ 0.254076 │ 59.8276 │
│ 6 │ cpu │ 3DMulBroad │ 0.0010416 │ 0.035999 │ 34.5613 │
│ 7 │ cpu │ FFMPEGH264Write │ 105.579 │ 4004.5 │ 37.9291 │
│ 8 │ mem │ DeepCopy │ 0.000177585 │ 0.0102587 │ 57.7681 │
│ 9 │ diskio │ DiskWrite1KB │ 0.031872 │ 1.738 │ 54.5307 │
│ 10 │ diskio │ DiskWrite1MB │ 0.650659 │ 48.5503 │ 74.6171 │
│ 11 │ diskio │ DiskRead1KB │ 0.00681275 │ 0.346284 │ 50.8288 │
│ 12 │ diskio │ DiskRead1MB │ 0.144945 │ 9.43058 │ 65.0631 │
│ 13 │ loading │ JuliaLoad │ 91.1325 │ 2048.39 │ 22.4771 │
│ 14 │ compilation │ compilecache │ 110.3 │ 5665.95 │ 51.3687 │

```

---

<div class="post-metadata">

**Author:** ![ianshmean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ianshmean/32/216042_2.png) [@ianshmean](https://discourse.julialang.org/u/ianshmean)\
**Post date:** [May 10, 2020, 5:52am UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982/45 "2020-05-10T05:52:17Z")

</div>

Awesome, thanks so much @imcinerney.

I’ve just merged additional tests and changes to the way information is reported. Anyone else who is able/willing to share benchmark results, please submit them [here](https://github.com/ianshmean/SystemBenchmark.jl/issues/8) and I’ll add them to the repo for group analysis

---

<div class="post-metadata">

**Author:** ![ianshmean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ianshmean/32/216042_2.png) [@ianshmean](https://discourse.julialang.org/u/ianshmean)\
**Post date:** [May 10, 2020, 6:19pm UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982/46 "2020-05-10T18:19:41Z")

</div>

**PSA: Don’t submit any more results here**

Please submit results over at [Submit benchmark files here · Issue #8 · IanButterworth/SystemBenchmark.jl · GitHub](https://github.com/ianshmean/SystemBenchmark.jl/issues/8)

* * *

I’ve added a function to grab all the results posted to [SystemBenchmark.jl/issues/8](https://github.com/ianshmean/SystemBenchmark.jl/issues/8) and return them as a single DataFrame.

```julia
using SystemBenchmark
getsubmittedbenchmarks()

```

Given the additional tests, changes to the way data is collected in the package, and how well the data collection in that issue seems to be going, I’d really appreciate if people that have submitted results here could re-run with the latest master and submit to [that issue](https://github.com/ianshmean/SystemBenchmark.jl/issues/8), using the instructions. Specifically I need the `.txt` files to be linked to.

Then we’ll have a nice easily query-able database. Thanks!

p.s. Sorry if that means booting a different OS / starting up that raspberry pi again…

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [May 28, 2020, 8:00pm UTC](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982/47 "2020-05-28T20:00:33Z")

</div>

New lappytoppy!

```julia
32×6 DataFrames.DataFrame
│ Row │ cat │ testname │ units │ ref_res │
│ │ String │ String │ String? │ Any │
├─────┼─────────────┼───────────────────────────┼─────────┼──────────────────────────────────────────┤
│ 1 │ info │ SysBenchVer │ │ 0.3.0 │
│ 2 │ info │ JuliaVer │ │ 1.4.1 │
│ 3 │ info │ OS │ │ Linux (x86_64-pc-linux-gnu) │
│ 4 │ info │ CPU │ │ Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz │
│ 5 │ info │ WORD_SIZE │ │ 64 │
│ 6 │ info │ LIBM │ │ libopenlibm │
│ 7 │ info │ LLVM │ │ libLLVM-8.0.1 (ORCJIT, skylake) │
│ 8 │ info │ GPU │ │ GeForce GTX 1650 with Max-Q Design │
│ 9 │ cpu │ FloatMul │ ms │ 1.1339999999999999e-6 │
│ 10 │ cpu │ FusedMulAdd │ ms │ 1.1339999999999999e-6 │
│ 11 │ cpu │ FloatSin │ ms │ 4.051e-6 │
│ 12 │ cpu │ VecMulBroad │ ms │ 2.9823115577889445e-5 │
│ 13 │ cpu │ CPUMatMul │ ms │ 0.018683 │
│ 14 │ cpu │ MatMulBroad │ ms │ 0.0042223 │
│ 15 │ cpu │ 3DMulBroad │ ms │ 0.0010529 │
│ 16 │ cpu │ peakflops │ flops │ 1.8061545733047305e11 │
│ 17 │ cpu │ FFMPEGH264Write │ ms │ 107.751337 │
│ 18 │ mem │ DeepCopy │ ms │ 0.00018790257558790594 │
│ 19 │ mem │ Bandwidth10kB │ MiB/s │ 104871.77985698299 │
│ 20 │ mem │ Bandwidth100kB │ MiB/s │ 58579.50346475738 │
│ 21 │ mem │ Bandwidth1MB │ MiB/s │ 33439.963407070725 │
│ 22 │ mem │ Bandwidth10MB │ MiB/s │ 5551.476774539766 │
│ 23 │ mem │ Bandwidth100MB │ MiB/s │ 6910.9558841033295 │
│ 24 │ diskio │ DiskWrite1KB │ ms │ 0.031895 │
│ 25 │ diskio │ DiskWrite1MB │ ms │ 0.920052 │
│ 26 │ diskio │ DiskRead1KB │ ms │ 0.006683666666666667 │
│ 27 │ diskio │ DiskRead1MB │ ms │ 0.143755 │
│ 28 │ loading │ JuliaLoad │ ms │ 91.08045 │
│ 29 │ compilation │ compilecache │ ms │ 208.532099 │
│ 30 │ compilation │ success_create_expr_cache │ ms │ 235.4855485 │
│ 31 │ compilation │ create_expr_cache │ ms │ 0.894126 │
│ 32 │ compilation │ output-ji-substart │ ms │ 32.639706000000004 │

│ Row │ test_res │ factor │
│ │ Any │ Any │
├─────┼───────────────────────────────────────────┼───────────┤
│ 1 │ 0.3.2 │ Not Equal │
│ 2 │ 1.4.2 │ Not Equal │
│ 3 │ Linux (x86_64-pc-linux-gnu) │ Equal │
│ 4 │ Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz │ Not Equal │
│ 5 │ 64 │ Equal │
│ 6 │ libopenlibm │ Equal │
│ 7 │ libLLVM-8.0.1 (ORCJIT, icelake-client) │ Not Equal │
│ 8 │ missing │ Not Equal │
│ 9 │ 1.205e-6 │ 1.06261 │
│ 10 │ 1.198e-6 │ 1.05644 │
│ 11 │ 4.056e-6 │ 1.00123 │
│ 12 │ 3.05965e-5 │ 1.02593 │
│ 13 │ 0.071705 │ 3.83798 │
│ 14 │ 0.00392787 │ 0.930269 │
│ 15 │ 0.0011296 │ 1.07285 │
│ 16 │ 2.8156e10 │ 0.155889 │
│ 17 │ 195.38 │ 1.81325 │
│ 18 │ 0.000139657 │ 0.743239 │
│ 19 │ 159868.0 │ 1.52442 │
│ 20 │ 49199.0 │ 0.839868 │
│ 21 │ 24273.3 │ 0.725877 │
│ 22 │ 12538.3 │ 2.25856 │
│ 23 │ 12419.3 │ 1.79704 │
│ 24 │ 0.0278665 │ 0.873695 │
│ 25 │ 1.34645 │ 1.46345 │
│ 26 │ 0.00617983 │ 0.924617 │
│ 27 │ 0.125835 │ 0.875343 │
│ 28 │ 99.5474 │ 1.09296 │
│ 29 │ 313.758 │ 1.5046 │
│ 30 │ 321.333 │ 1.36455 │
│ 31 │ 5.66792 │ 6.33906 │
│ 32 │ 56.1592 │ 1.72058 │

```

[Previous page](https://discourse.julialang.org/t/rfc-ann-systembenchmark-jl/38982.md?page=2)