Hello, I posted this on Stackoverflow (https://stackoverflow.com/q/78087392/23508386) a few days ago but got ignored :c. So I decided to try my luck by copying it (inlcuding the Updates to the post I made on stack overflow so far) here:
My plan is to first calculate allot of expressions symbolically, safe them, then use them later in several extensive Metropolis Monte-Carlo-simulations. While I initially tried Sympy and then Symbolics, I am now intending to use SymEngine as it just appears to be significantly faster at all relevant steps. When using SymEngine a big part of the time used to generate the functions is actually not the symbolic computations but the lambdifying. Therefore I wish to save the functions generated by lambdify and not the symbolic expressions. However I seem to be incapable of finding a way to successfulyl load them into a new Session.
I initially tried Using BSON as proposed here: Is there a way to write a function into JLD or other h5 file? - #2 by marius311. But sadly reloading only works in the same Session. This can for example be reproduced by doing:
using SymEngine
using BSON: @save, @load
function mwe()
@vars X
f = X
return(lambdify(f,[X]))
end
f = mwe()
@save "test.bson" f
then (in a new Session):
using SymEngine
using BSON: @save, @load
function mwe()
@vars X
f = X
return(lambdify(f,[X]))
end
@load "test.bson" f
println(f(1))
f = "nofunction"
@load "test.bson" f
println(f(1))
If I do both in the same Session the output is
1
1
as expected.
However if I run the second part in a new Session I get:
ERROR: UndefVarError: ###312 not defined
I also tried a naive (maybe there are more elaborate ways of applying these?) aproach of using Serialization, FileIO as well as JLD2. All of them resulting in similar behavior working fine if loaded in same session, but either erroring out or warning and then behaving unexpectedly when loading in a different Session.
What did work is using Symbolics write("test.jl", string(f)) as described here:
Using Serialization to store then load a lambdified function? - #2 by SteffenPL. However since Symbolics is -according to all benchamrking I did- much slower then SymEnginge this does not really seem to be the ideal way to go to me.
Update: I tried to find a work around by generating a sysimage with PackageCompiler. The Idea was to create a module in which all the symbolic calculations are done then a constant containing a list of all the lamdifyed expressions is created. Then create a sysimage of that module and hope that I could use the lambdified expression in a new Session when started using the sysimage.
If I do not generate the constant, creating the sysimage works fine. But once I add the lines that should create the list of functions
const Bookofterms = generate_terms(3,3)
I get a somewhat lengthy error message containing:
ERROR: The following 1 direct dependency failed to precompile:
TimeInt [14fa5fd4-ce9e-41c9-8e49-66abaaa3fbcd]
(TimeInt is the custom package)
Failed to precompile TimeInt [14fa5fd4-ce9e-41c9-8e49-66abaaa3fbcd] to C:\Users\...
ERROR: LoadError: ArgumentError: Expression does not lambdify
caused by: Evaluation into the closed module `SymEngine` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `SymEngine` with `eval` during precompilation - don't do this.
I am honestly quite confused on whats going on. Is SymEngine applying some weird side effects to what ever thing during lambdification that get lost when one tries to safe the functions? Does that mean it is impossible to safe functions created by SymEngines lambdify?