# Julia performance on the same task on different CPUs and OSs

**URL:** <https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916>\
**Category:** Performance\
**Tags:** dataframes, os, cpu\
**Created:** [August 7, 2024, 12:37pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916 "2024-08-07T12:37:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kobusherbst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kobusherbst/32/19682_2.png) [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Post date:** [August 7, 2024, 12:37pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/1 "2024-08-07T12:37:08Z")

</div>

## The workload

Arrow files with 250K individuals and 3M education status observations  
Read into memory as DataFrames, join individual records with education status observations.  
Group resulting data frame by individual and process the on average 12 observations per individual.  
Process individual education records to eliminate retrogression in educational attainment, and fill gaps in annual educational attainment using linear interpolation.  
Write out the resulting cleaned dataset as an arrow file.  
No multi-threading is used by the code to, for example, process individuals independently

Julia version: 1.10.4  
Packages: DataFrames, Arrow, Dates, StatsBase  
Code available here: [Source code](https://github.com/kobusherbst/VMPerformance.jl.git)

## Question

Why the major differences in performance for the exact same code in the different environments in the table below:

## Results

| Time (hh:mm:ss) | OS | CPU | Cores | Machine |
| --- | --- | --- | --- | --- |
| 00:26:42 | Windows 11 | Xeon Gold 6430 | 8 | VMWare virtual machine 128GB RAM |
| 00:29:17 | Win Server 2019 | Xeon Cascadelake | 25 | Openstack virtual machine 256GB RAM |
| 00:12:12 | Windows 11 | i7-13700H | 14 | Dell XPS 17 64GB RAM |
| 00:11:47 | RaspianOs | Cortex A76 | 4 | Raspberry Pi 5 8GB |
| 00:07:53 | Windows 11 | Ryzen 7 3800 | 8 | Desktop 128GB RAM |
| 00:01:51 | MacOs 14.5 | M3 Max | 14 | MacBook Pro 36GB RAM |

I realise that the virtual environment may have different workloads in the background, but the results were fairly consistent over several different runs at different times and the differences are large! 99% of the execution time is spent in the code containing no file reads or writes.

Thank you for pointing me to ways in which I can improve performance in the virtual environment, because that is the common resource in my environment.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [August 7, 2024, 12:46pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/2 "2024-08-07T12:46:45Z")

</div>

this is likely a combination of windows having a slow filesystem, the servers having slow drives, and antivirus software that is slowing down the windows machines.

---

<div class="post-metadata">

**Author:** ![kobusherbst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kobusherbst/32/19682_2.png) [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Post date:** [August 7, 2024, 12:52pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/3 "2024-08-07T12:52:54Z")

</div>

The Raspberry Pi is using an SD card for storage!

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [August 7, 2024, 4:43pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/4 "2024-08-07T16:43:14Z")

</div>

Well, Linux is faster than Windows… If you use virtual machines anyways, why not using a virtual machine with Linux?

---

<div class="post-metadata">

**Author:** ![kobusherbst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kobusherbst/32/19682_2.png) [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Post date:** [August 8, 2024, 8:25am UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/5 "2024-08-08T08:25:37Z")

</div>

I did test a Linux virtual machine:

| Time | OS | CPU | Cores | Machine |
| --- | --- | --- | --- | --- |
| 00:25:54 | Ubuntu | Xeon Gold 6430 | 8 | VMWare virtual machine |

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [August 8, 2024, 8:35am UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/6 "2024-08-08T08:35:36Z")

</div>

Perhaps there is something wrong with the configuration of your virtual machines?

- how much RAM do you assign to the VM?
- how many cores do you assign to the VM?
- with how many threads to you start Julia?

And then there is the option “Virtualize Intel VT-x/EPT or AMD-V/RVI” which you can check or not.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/d/4d9d6eee070efdc2ccef25697e0d37d10c03db0e.png)

---

<div class="post-metadata">

**Author:** ![BrendanG](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/brendang/32/211189_2.png) [@BrendanG](https://discourse.julialang.org/u/BrendanG)\
**Post date:** [August 8, 2024, 8:42pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/7 "2024-08-08T20:42:26Z")

</div>

Ran the application on bare metal server installed with Ubuntu 22.04LTS and used wine as compatibility layer.

Started cleaning education status  
=== Finished cleaning education status after 46 minutes, 48 seconds, 524 milliseconds

$ lscpu  
Architecture: x86\_64  
CPU op-mode(s): 32-bit, 64-bit  
Address sizes: 46 bits physical, 48 bits virtual  
Byte Order: Little Endian  
CPU(s): 48  
On-line CPU(s) list: 0-47  
Vendor ID: GenuineIntel  
Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz  
CPU family: 6  
Model: 85  
Thread(s) per core: 2  
Core(s) per socket: 12  
Socket(s): 2  
Stepping: 4  
BogoMIPS: 5200.00  
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant\_tsc art arch\_perfmon pebs bts rep\_good nopl xtopology nonstop\_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds\_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4\_1 sse4\_2 x2apic movbe popcnt tsc\_deadline\_timer aes xsave avx f16c rdrand lahf\_lm abm 3dnowprefetch cpuid\_fault epb cat\_l3 cdp\_l3 invpcid\_single pti intel\_ppin ssbd mba ibrs ibpb stibp tpr\_shadow vnmi flexpriority ept vpid ept\_ad fsgsbase tsc\_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt\_a avx512f avx512dq rdseed adx smap clflushopt clwb intel\_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm\_llc cqm\_occup\_llc cqm\_mbm\_total cqm\_mbm\_local dtherm ida arat pln pts pku ospke md\_clear flush\_l1d arch\_capabilities  
Virtualization features:  
Virtualization: VT-x  
Caches (sum of all):  
L1d: 768 KiB (24 instances)  
L1i: 768 KiB (24 instances)  
L2: 24 MiB (24 instances)  
L3: 38.5 MiB (2 instances)  
NUMA:  
NUMA node(s): 2  
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46  
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47

Machine RAM: 315GB

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [August 13, 2024, 1:55pm UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/8 "2024-08-13T13:55:58Z")

</div>

If you have the latest Windows 11 you may activate the [Dev Drive](https://learn.microsoft.com/en-us/windows/dev-drive).  
It creates a working space which is much more efficient with handling I/O tasks on the HD.  
It is based on [ReFS](https://en.wikipedia.org/wiki/ReFS).

By the way, the reason Windows’ performance with HD I/O calls is slower is due to a design choice of allowing 3rd party API on calls.  
It is a feature of the OS and its modular structure. The price is higher overhead.  
A good example of the benefits of such architecture is [voidtools - Everything](https://www.voidtools.com) - Locate files and folders by name instantly.

---

<div class="post-metadata">

**Author:** ![kobusherbst](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kobusherbst/32/19682_2.png) [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Post date:** [August 18, 2024, 4:57am UTC](https://discourse.julialang.org/t/julia-performance-on-the-same-task-on-different-cpus-and-oss/117916/9 "2024-08-18T04:57:22Z")

</div>

Although the test workload used here is not IO constrained, I do have other real world workloads that are much more disk intensive. I will give this a try.