Small issue when testing a system without a discrete GPU:
julia> comparison_df = compareToRef(sysbenchmark())
[ Info: CuArrays.functional() == false. No usable GPU detected
Disk IO tests100%|█████████████████████████████████████████| Time: 0:01:06
ERROR: ArgumentError: Column names :ms and :reswere found in only one of the passed data frames and passed cols=:setequal
Stacktrace:
[1] append!(::DataFrames.DataFrame, ::DataFrames.DataFrame; cols::Symbol) at /home/ufechner/.julia/packages/DataFrames/S3ZFo/src/dataframe/dataframe.jl:1222
[2] append! at /home/ufechner/.julia/packages/DataFrames/S3ZFo/src/dataframe/dataframe.jl:1191 [inlined]
[3] sysbenchmark(; printsysinfo::Bool) at /home/ufechner/.julia/packages/SystemBenchmark/SXV31/src/SystemBenchmark.jl:125
[4] sysbenchmark() at /home/ufechner/.julia/packages/SystemBenchmark/SXV31/src/SystemBenchmark.jl:78
[5] top-level scope at REPL[3]:1
julia>
Yes, sorry, just fixed that on master. Will register and release proper versions soon
3 Likes
After some work, I was able to get the benchmarks from several ARM boards I had sitting around:
Raspberry Pi 3B+ (aarch64 - Quad-core Cortex A-53):
Reference system ----------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
GPU: GeForce GTX 1650 with Max-Q Design
Test system ---------------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (aarch64-unknown-linux-gnu)
CPU: unknown
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, cortex-a53)
14×5 DataFrames.DataFrame
│ Row │ cat │ testname │ ref_ms │ test_ms │ factor │
│ │ String │ String │ Float64 │ Float64 │ Float64 │
├─────┼─────────────┼─────────────────┼─────────────┼─────────────┼─────────┤
│ 1 │ cpu │ FloatMul │ 1.134e-6 │ 5.104e-6 │ 4.50088 │
│ 2 │ cpu │ FloatSin │ 4.048e-6 │ 5.51524e-5 │ 13.6246 │
│ 3 │ cpu │ VecMulBroad │ 2.94935e-5 │ 0.000324703 │ 11.0093 │
│ 4 │ cpu │ CPUMatMul │ 0.018796 │ 0.207395 │ 11.034 │
│ 5 │ cpu │ MatMulBroad │ 0.0042468 │ 0.033177 │ 7.81224 │
│ 6 │ cpu │ 3DMulBroad │ 0.0010416 │ 0.008993 │ 8.63383 │
│ 7 │ cpu │ FFMPEGH264Write │ 105.579 │ 1013.53 │ 9.59979 │
│ 8 │ mem │ DeepCopy │ 0.000177585 │ 0.0016198 │ 9.12129 │
│ 9 │ diskio │ DiskWrite1KB │ 0.031872 │ 1.2978 │ 40.7192 │
│ 10 │ diskio │ DiskWrite1MB │ 0.650659 │ 57.1556 │ 87.8427 │
│ 11 │ diskio │ DiskRead1KB │ 0.00681275 │ 0.134114 │ 19.6857 │
│ 12 │ diskio │ DiskRead1MB │ 0.144945 │ 1.97855 │ 13.6504 │
│ 13 │ loading │ JuliaLoad │ 91.1325 │ 790.011 │ 8.66882 │
│ 14 │ compilation │ compilecache │ 110.3 │ 870.46 │ 7.89177 │
Xilinx Pynq-v1 (armv7l - Zynq 7020 with dual-core Cortex A9):
Reference system ----------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
GPU: GeForce GTX 1650 with Max-Q Design
Test system ---------------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (arm-linux-gnueabihf)
CPU: ARMv7 Processor rev 0 (v7l)
WORD_SIZE: 32
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, cortex-a9)
14×5 DataFrames.DataFrame
│ Row │ cat │ testname │ ref_ms │ test_ms │ factor │
│ │ String │ String │ Float64 │ Float64 │ Float64 │
├─────┼─────────────┼─────────────────┼─────────────┼─────────────┼─────────┤
│ 1 │ cpu │ FloatMul │ 1.134e-6 │ 1.98255e-5 │ 17.4828 │
│ 2 │ cpu │ FloatSin │ 4.048e-6 │ 0.0001094 │ 27.0258 │
│ 3 │ cpu │ VecMulBroad │ 2.94935e-5 │ 0.000971211 │ 32.9297 │
│ 4 │ cpu │ CPUMatMul │ 0.018796 │ 1.2337 │ 65.6363 │
│ 5 │ cpu │ MatMulBroad │ 0.0042468 │ 0.156975 │ 36.9631 │
│ 6 │ cpu │ 3DMulBroad │ 0.0010416 │ 0.025769 │ 24.7398 │
│ 7 │ cpu │ FFMPEGH264Write │ 105.579 │ 3097.85 │ 29.3416 │
│ 8 │ mem │ DeepCopy │ 0.000177585 │ 0.0049185 │ 27.6967 │
│ 9 │ diskio │ DiskWrite1KB │ 0.031872 │ 1.15174 │ 36.1364 │
│ 10 │ diskio │ DiskWrite1MB │ 0.650659 │ 52.8955 │ 81.2953 │
│ 11 │ diskio │ DiskRead1KB │ 0.00681275 │ 0.143813 │ 21.1093 │
│ 12 │ diskio │ DiskRead1MB │ 0.144945 │ 4.3458 │ 29.9824 │
│ 13 │ loading │ JuliaLoad │ 91.1325 │ 1444.04 │ 15.8455 │
│ 14 │ compilation │ compilecache │ 110.3 │ 2714.8 │ 24.6129 │
Beaglebone Black (armv7l - Single core Cortex A8):
Reference system ----------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
GPU: GeForce GTX 1650 with Max-Q Design
Test system ---------------------------
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Linux (arm-linux-gnueabihf)
CPU: ARMv7 Processor rev 2 (v7l)
WORD_SIZE: 32
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, cortex-a8)
14×5 DataFrames.DataFrame
│ Row │ cat │ testname │ ref_ms │ test_ms │ factor │
│ │ String │ String │ Float64 │ Float64 │ Float64 │
├─────┼─────────────┼─────────────────┼─────────────┼─────────────┼─────────┤
│ 1 │ cpu │ FloatMul │ 1.134e-6 │ 2.21073e-5 │ 19.495 │
│ 2 │ cpu │ FloatSin │ 4.048e-6 │ 0.000355889 │ 87.9172 │
│ 3 │ cpu │ VecMulBroad │ 2.94935e-5 │ 0.00128817 │ 43.6763 │
│ 4 │ cpu │ CPUMatMul │ 0.018796 │ 21.7096 │ 1155.01 │
│ 5 │ cpu │ MatMulBroad │ 0.0042468 │ 0.254076 │ 59.8276 │
│ 6 │ cpu │ 3DMulBroad │ 0.0010416 │ 0.035999 │ 34.5613 │
│ 7 │ cpu │ FFMPEGH264Write │ 105.579 │ 4004.5 │ 37.9291 │
│ 8 │ mem │ DeepCopy │ 0.000177585 │ 0.0102587 │ 57.7681 │
│ 9 │ diskio │ DiskWrite1KB │ 0.031872 │ 1.738 │ 54.5307 │
│ 10 │ diskio │ DiskWrite1MB │ 0.650659 │ 48.5503 │ 74.6171 │
│ 11 │ diskio │ DiskRead1KB │ 0.00681275 │ 0.346284 │ 50.8288 │
│ 12 │ diskio │ DiskRead1MB │ 0.144945 │ 9.43058 │ 65.0631 │
│ 13 │ loading │ JuliaLoad │ 91.1325 │ 2048.39 │ 22.4771 │
│ 14 │ compilation │ compilecache │ 110.3 │ 5665.95 │ 51.3687 │
1 Like
Awesome, thanks so much @imcinerney.
I’ve just merged additional tests and changes to the way information is reported. Anyone else who is able/willing to share benchmark results, please submit them here and I’ll add them to the repo for group analysis
PSA: Don’t submit any more results here
Please submit results over at Submit benchmark files here · Issue #8 · IanButterworth/SystemBenchmark.jl · GitHub
I’ve added a function to grab all the results posted to SystemBenchmark.jl/issues/8 and return them as a single DataFrame.
using SystemBenchmark
getsubmittedbenchmarks()
Given the additional tests, changes to the way data is collected in the package, and how well the data collection in that issue seems to be going, I’d really appreciate if people that have submitted results here could re-run with the latest master and submit to that issue, using the instructions. Specifically I need the .txt files to be linked to.
Then we’ll have a nice easily query-able database. Thanks!
p.s. Sorry if that means booting a different OS / starting up that raspberry pi again…
5 Likes
New lappytoppy!
32×6 DataFrames.DataFrame
│ Row │ cat │ testname │ units │ ref_res │
│ │ String │ String │ String? │ Any │
├─────┼─────────────┼───────────────────────────┼─────────┼──────────────────────────────────────────┤
│ 1 │ info │ SysBenchVer │ │ 0.3.0 │
│ 2 │ info │ JuliaVer │ │ 1.4.1 │
│ 3 │ info │ OS │ │ Linux (x86_64-pc-linux-gnu) │
│ 4 │ info │ CPU │ │ Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz │
│ 5 │ info │ WORD_SIZE │ │ 64 │
│ 6 │ info │ LIBM │ │ libopenlibm │
│ 7 │ info │ LLVM │ │ libLLVM-8.0.1 (ORCJIT, skylake) │
│ 8 │ info │ GPU │ │ GeForce GTX 1650 with Max-Q Design │
│ 9 │ cpu │ FloatMul │ ms │ 1.1339999999999999e-6 │
│ 10 │ cpu │ FusedMulAdd │ ms │ 1.1339999999999999e-6 │
│ 11 │ cpu │ FloatSin │ ms │ 4.051e-6 │
│ 12 │ cpu │ VecMulBroad │ ms │ 2.9823115577889445e-5 │
│ 13 │ cpu │ CPUMatMul │ ms │ 0.018683 │
│ 14 │ cpu │ MatMulBroad │ ms │ 0.0042223 │
│ 15 │ cpu │ 3DMulBroad │ ms │ 0.0010529 │
│ 16 │ cpu │ peakflops │ flops │ 1.8061545733047305e11 │
│ 17 │ cpu │ FFMPEGH264Write │ ms │ 107.751337 │
│ 18 │ mem │ DeepCopy │ ms │ 0.00018790257558790594 │
│ 19 │ mem │ Bandwidth10kB │ MiB/s │ 104871.77985698299 │
│ 20 │ mem │ Bandwidth100kB │ MiB/s │ 58579.50346475738 │
│ 21 │ mem │ Bandwidth1MB │ MiB/s │ 33439.963407070725 │
│ 22 │ mem │ Bandwidth10MB │ MiB/s │ 5551.476774539766 │
│ 23 │ mem │ Bandwidth100MB │ MiB/s │ 6910.9558841033295 │
│ 24 │ diskio │ DiskWrite1KB │ ms │ 0.031895 │
│ 25 │ diskio │ DiskWrite1MB │ ms │ 0.920052 │
│ 26 │ diskio │ DiskRead1KB │ ms │ 0.006683666666666667 │
│ 27 │ diskio │ DiskRead1MB │ ms │ 0.143755 │
│ 28 │ loading │ JuliaLoad │ ms │ 91.08045 │
│ 29 │ compilation │ compilecache │ ms │ 208.532099 │
│ 30 │ compilation │ success_create_expr_cache │ ms │ 235.4855485 │
│ 31 │ compilation │ create_expr_cache │ ms │ 0.894126 │
│ 32 │ compilation │ output-ji-substart │ ms │ 32.639706000000004 │
│ Row │ test_res │ factor │
│ │ Any │ Any │
├─────┼───────────────────────────────────────────┼───────────┤
│ 1 │ 0.3.2 │ Not Equal │
│ 2 │ 1.4.2 │ Not Equal │
│ 3 │ Linux (x86_64-pc-linux-gnu) │ Equal │
│ 4 │ Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz │ Not Equal │
│ 5 │ 64 │ Equal │
│ 6 │ libopenlibm │ Equal │
│ 7 │ libLLVM-8.0.1 (ORCJIT, icelake-client) │ Not Equal │
│ 8 │ missing │ Not Equal │
│ 9 │ 1.205e-6 │ 1.06261 │
│ 10 │ 1.198e-6 │ 1.05644 │
│ 11 │ 4.056e-6 │ 1.00123 │
│ 12 │ 3.05965e-5 │ 1.02593 │
│ 13 │ 0.071705 │ 3.83798 │
│ 14 │ 0.00392787 │ 0.930269 │
│ 15 │ 0.0011296 │ 1.07285 │
│ 16 │ 2.8156e10 │ 0.155889 │
│ 17 │ 195.38 │ 1.81325 │
│ 18 │ 0.000139657 │ 0.743239 │
│ 19 │ 159868.0 │ 1.52442 │
│ 20 │ 49199.0 │ 0.839868 │
│ 21 │ 24273.3 │ 0.725877 │
│ 22 │ 12538.3 │ 2.25856 │
│ 23 │ 12419.3 │ 1.79704 │
│ 24 │ 0.0278665 │ 0.873695 │
│ 25 │ 1.34645 │ 1.46345 │
│ 26 │ 0.00617983 │ 0.924617 │
│ 27 │ 0.125835 │ 0.875343 │
│ 28 │ 99.5474 │ 1.09296 │
│ 29 │ 313.758 │ 1.5046 │
│ 30 │ 321.333 │ 1.36455 │
│ 31 │ 5.66792 │ 6.33906 │
│ 32 │ 56.1592 │ 1.72058 │
1 Like