Different memory usage under Windows and linux?

General Usage
1

I am trying to run a simulation using EvoDynamics.jl, a package I wrote myself. The simulation is quite complicated, so I am putting the code here.

The problem I am facing is different memory usage under Windows vs Ubuntu or Debian. On my Windows laptop with 32 GB of RAM, the simulation ends in about 2 hours. Using the same laptop that is running Ubuntu, the code gets killed because of using too much memory. I even tried it on a Debian workstation with more that 300 GB of RAM and all of it gets used until the jobs is killed.

I can assure that the initial conditions are identical. The simulations are seeded to the same value, and they use the same Julia environment, meaning the packages and their versions are the same. Basically, I have a git repo that is cloned everywhere. The directory has its own environment that is activated before running the code, and the simulations have a seed to remove any stochasticity.

What do you think could explain this difference?

3 Likes
2

It sounds like a memory leak that is platform-dependent for some reason. Would you be free to share the Git repo to help us inverstigate?

1 Like
3

Yes, I’ll prepare and share it here. I will have to remove other stuff from the repo.

4

Here is the repo: GitHub - kavir1698/debug: temp . maincor.jl runs the simulation.

1 Like
5

I can reproduce the apparently unbounded memory growth on my Linux computer but I don’t have time right now to dig into it sorry.

Maybe @Datseris or someone else that knows Agents.jl can say if they have any idea why the memory behavior would be different between Windows and Linux…

6

Could you try Julia 1.10-rc1 and see if the problem still appears?

7

I’ll test and report back here.

1 Like
8

Unhelpful reply. Have you run top or vmstat when this program is running?

I guess on Linux the system log reports that the OOM killer is killing the program?

9

I had to add:

if Sys.free_memory()/2^30 < 6.0
    GC.gc()
end

to large functions that are called by pmap to avoid OOM on Linux when using Julia 1.9.

4 Likes
10

Running with Julia 1.10.0-rc1 the code in your debug repo has reached 10% so far.
Resident Set Sizeis holding constant at around 4.3 Gbytes on a 16Gbyte laptop

Maybe another stage will be reached and things will get out of hand.
Now at 26% - I must go to choir practice so am stopping the simulation.

3 Likes
11

Interesting. I tried it with Julia 1.10-rc1 on WSL vs 1.9.3 on WSL. The memory usage was very different. Using Julia 1.10-rc1, it about 5 GB of RAM. Using Julia 1.9.3, it used less than 16 GB, but didn’t get killed.

I am now running the code on the Debian workstation. So far (11%), memory usage seems to have reached a constant level of 4 GB. With Julia 1.9, memory usage kept increasing all the time. It sounds like this is indeed an issue with Julia 1.9.

Update:

The simulation finished using no more than 4.3 Gb with Julia 1.10-rc1 on Debian.

1 Like
12

Yes, I followed memory usage with htop. Haven’t checked the system log.