I use JuMP.jl with HiGHS.jl to solve an optimisation problem, then generate a shared library for this using JuliaC.jl. The shared library is used within a larger software. It is loaded once (at startup), and then always the same Base.@ccallable function is called, which executes the optimisation.
It does the usual stuff, i.e.
model = Model(HiGHS.Optimizer)
# add variables, constraints, and an objective
optimize!(model)
# get values of some variables, termination status, etc.
# return extracted information
Now, we’re dealing with segfault issues.
It came to my mind that HiGHS itself again loads (via HiGHS_jll.jl) a library. Then, probably when creating the julia-level model, it creates a HiGHS instance via Highs_create. The usual C-API workflow would then be to call Highs_destroy once everything is done. Otherwise, one would just leak memory by keeping all those old HiGHS instances around, right?
How is this handled throughout the chain from JuMP.jl through MathOptInterface.jl and HiGHS.jl, down to HiGHS_jll.jl? Is there a high-level interface to free allocated memory (by destroying the HiGHS instance) before I return from the function in the shared library I create?
Some tests indicate that the finalizers are run once the GC collected (or while it is doing so) all references. The description in the docs is not very detailed / explicit about it (Essentials · The Julia Language).
The question still remains when that might happen if all is in a shared library that is kept loaded. Would it, in that case, be advisable to manually call Base.finalize on the model (or alternatively forcing the garbage collector to run via GC.gc()), before returning from the current call into the library, to make sure there is no unused, allocated memory possibly stacking up before the next automatic run of the GC?
Hi @asprionj, I’ve moved this to the optimization section of the forum.
Yes, Highs_destroy is called during the GC finaliser. You should almost certainly not call finalise yourself.
What’s the segfault? Do you have a reproducible example? I’ve never tried JuliaC with JuMP or HiGHS. (I’ve added this as an item to next week’s JuMP-dev hackathon)
We’re still trying to figure out where the problem (the segfault) actually comes from. It manifests itself not at the time when the memory corruption occurs, it seems, but we’re still trying to isolate/pinpoint it exactly. Could very well be that it is not related to JuMP/HiGHS at all. For example, we’re just getting the segfaults on Linux, not on Windows. Also, we get it when we do not run any optimisation at all; so it seems more like an interface (handing forth and back data via pointers).
Or does JuMP and/or HiGHS allocate any memory, just by the using statement?
BTW, the segfault occurs after something like 3200 calls, and then happens when some C-library invoked on Java side (we assume a webserver component) is called and probably accesses the memory or so. So, not when actually inside the shared library created by JuliaC.
We currently try to use memory-inspecting / tracking tools to pinpoint the problem. Or then have to resort to start with a minimal setup (just an empty function), then add functionality step by step and see what steps introduces the problem.
I’ll keep you posted via this thread on whatever we find out.
FWIW, here’s the two segfaults, the first when we did call the optimisation:
[1] signal 11 (1): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x73bf944fe5a8) at (unknown file)
Allocations: 21587875 (Pool: 21587861; Big: 14); GC: 52
And that’s the one when we did not invoke the solver:
[1] signal 11 (1): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x7d0c9c4d9b88) at (unknown file)
Allocations: 1 (Pool: 1; Big: 0); GC: 0
Update: we stripped down to an as-minimal example as possible. The @ccallable now is just a completely void function, not receiving or returning anything, and not performing anything at all.
With that, we can even --trim=safe it, so there’s nothing dynamic going on anymore.
Still, we get a segfault when calling the function in the library from Java, on a single thread, under Linux, after something like 5000 to 8000 calls. We do not get segfaults under Windows, and neither when calling the library (for 100k times) from Go (instead of Java).
So, there seems to be something in how Java handles C shared libs that does not play nicely with Julia. And it also seems that there is not problem with how JuMP/HiGHS handle memory in the shared libs they interface as artifacts.
The conclusion from discussions at JuMP-dev 2025 is that we’re going to write tests, tutorials, and documentation for using JuMP and various solvers with PackageCompiler and JuliaC. But from your comment, it doesn’t seem like this is JuMP-related, so I’m not sure I can help, sorry.