Hi, I have some code that can do some performance optimizations provided that an allocation of some matrix NxN is possible. If N is too large, I get an out of memory error. In this case I can use an alternative algorithm that doesn’t do this allocation but is slower. Is there a way to programmatica…

try/catch? Though if the user doesn’t have enough memory for this, I’m not sure their original code would finish…

There’s not really a reliable, cross-platform way to do this, because of Linux: Linux malloc doesn’t guarantee that you can actually use the allocated memory without triggering the OOM killer Even if you malloc and touch every page of memory to ensure it’s currently available, the OOM killer could…

@jpsamaroo Good reply. I have delved into the OOM killer quite a lot in the past. Stupid response from me - what if you disable memory overcommit on Linux? does this give you a more determinable OOM situation? ps. We should also consider running the code in a cgroup here - surely you know how much…

[image] johnh: what if you disable memory overcommit on Linux? I suspect disabling memory overcommit would break other applications that depend on it, but it’s worth a try if it’s critical to avoiding OOMs for this specific application. [image] johnh: We should also consider running the …

I had to try it: julia> try rand(Int(1e15)) catch OutOfMemoryError rand(10) end 10-element Vector{Float64}: 0.8514994866896233 0.6930665449494327 0.570494518398375 0.4205381912313253 0.05554694657914827 0.8990128840646228 0.11956078951581128 0.29042646526…

I was using a try/catch block before I opened this issue, perhaps I should have said in the opening :stuck_out_tongue: But I thought, or I vaguely remember, that try/catch are not good for source code because they have performance and/or stability drawbacks…?

[image] Datseris: But I thought, or I vaguely remember, that try/catch are not good for source code because they have performance and/or stability drawbacks…? This is exactly what try catch was invented for, so I don’t think you should shy away from it. What I think you want to avoid is throw…

Okay thanks people, I’ll go with the try/catch then because the other suggestion with malloc stuff is a bit too advanced for me.

try/catch is designed for trying something and falling back to some other code path if the first breaks horribly/irrecoverably (e.g. your memory limitation, where you can’t continue on the first path at all if it fails). If you only ever hit the try path, you’ll barely notice at all. Hitting the cat…

How to programmatically check if an allocation is possible

General Usage

rafael.guerra September 4, 2021, 3:49pm 2

Could you check the system free memory like here?

Topic		Replies	Views
Memory sizing and GC behavior General Usage	6	4945	July 23, 2020
Instead of OutOfMemory Error being thrown, my OS (Ubuntu) is killing the julia process General Usage	6	1055	August 18, 2021
Add testing for memory allocations Performance memory-allocation	13	1310	July 27, 2021
Feature request: unsafe_free! Internals & Design	25	1468	March 21, 2024
Stopping code if a given RAM limit is reached General Usage memory	8	1973	December 19, 2018

How to programmatically check if an allocation is possible

Related topics