Workaround for BSON "Inexact: trunc" error for saving very large files?

Specific Domains Data
1

Hi, I’m experiencing an (apparently known) error using BSON to try to save very large files, where it throws an error: InexactError: trunc(Int32, 2299858984), when the file to be saved is too large. This has been discussed previously, does anyone know if a fix has been developed?

An obvious workaround would be to disassemble the file into pieces when writing it out, and reassemble it back together when reading it in… unfortunately, I don’t know how to do this with my data set(s), which is not a simple array, but an ODESolution object. In particular, it is the result of solving a 2-dimensional partial differential equation (PDE), which I have broken into several thousand coupled ODE’s, each of which is continuous for interpolation in time t, with each located at a different grid point with discrete values of x.

Does anyone know how to breakdown, write, read, and reassemble this ODESolution object correctly, without losing the interpolation properties of the ODE’s in time? (Or better yet, a way to just fix the BSON problem?)

Here is a sample of the code generating the error:

prob = ODEProblem(Equation!, InitCondMatrix, tspan, Params)
soln = DifferentialEquations.solve(prob, Rodas5(), dense=true, 
            reltol=1e-12, abstol=1e-12, maxiters = 1e7, progress=true)
BSON.@save string(FilesDir, "soln.bson") soln

Note that InitCondMatrix contains initial Position & Velocity data for each discrete value of x, and is type Array{Float64,2} with size (NpointsInX, 2).

Thanks for any info!

2

I doubt it, since

is open.

I suggest just using a different format, like HDF5.

3

One option is JLSO. JLSO is not a serializer, it is a container arround serialized data, that adds metadata and compression.
This is now fixed in JLSO, if not using the :bson backend.
Used to be a problem for JLSO, where even if you were using the :julia_serializer backend, it would still fail, because the top layer containing metadata was BSON and stored the data with in it in what ever serialized form was selected.
Now it storeds the data after the BSON metadata.

What ever you use, remember that what ever serializer you use if you store things other than the plain supported types it will be just as unportable as using the julia serializer (which these days it not actually that bad); because it effectlyly falls back to the julia serializer for that.
For JLD2/JLD backed by HDF5 this means basically storing anything other than arrays of numbers.
For BSON this basically means arrays of numbers, and datetimes.

4

Thanks for the suggestions, guys. I’m trying JLSO, just having a little problem with the implementation. Based off the few examples given in the documentation, I’m trying things like:

JLSO.save(string(FilesDir, "MyFileName.jlso"), :stuff => ODEsoln)

which gives me the same truncation error for this huge data object as with direct BSON, so I tried:
JLSO.save(string(FilesDir, "MyFileName.jlso"), "format" => :julia_serialize, :stuff => ODEsoln)
which gives me a MethodError, and didn’t work. So per the documentation examples I tried:

JLSOFile(ODEsoln; format=:julia_serialize, compression=:gzip)

but that gave me a MethodError, and I’m not sure what to do with that command anyway.

I was able to save a regular array (one not too large to trigger the “trunc” error); it seems the easiest way to save & load it back was:

JLSO.save(string(FilesDir, "MyFIleName.jlso"), :ArrStuff => DataArray)
LoadedArr = JLSO.load(string(FilesDir, "MyFIleName.jlso"))[:ArrStuff]

But that doesn’t help me solve the huge ODE solution data read/write problem.
I’m probably missing something obvious with the syntax, any ideas?
Thanks again!

1 Like
5

Please open issues on the repo. Including issues with the documentation. :slightly_smiling_face:
It’s impossible to track issues on discourse etc etc once you are managing more than a dozen projects.
Everything in the documentation should work. (it should also be clear, so if things are unclear please do open an issue and we can fix it)

Re oversized though, it appears that hasn’t been released yet for some reason.
I openned an issue about it
https://github.com/invenia/JLSO.jl/issues/85
For now you can try ] add JLSO#master

2 Likes
6

Hi, I managed to get the syntax to save JLSO files, but I still got the same error as I had with straight BSON, when trying to save a very large file:
InexactError: trunc(Int32, 5374807941)

Using JLSO with the :julia_serializer backend was supposed to eliminate this problem, but I still can’t save the files. Any ideas? A sample of my code is below. Thanks…

prob = ODEProblem(MyEquation!, InitialConditions, tspan, Params)
soln = DifferentialEquations.solve(prob,Rodas5(), dense=true, reltol=1e-12, 
                                   abstol=1e-12, maxiters = 1e7, progress=true)
###    BSON.@save string(FilesDir, "soln.bson") soln
JLSO.save(string(FilesDir, "soln.jlso"), format=:julia_serialize, 
                   compression=:gzip, :SavedODESoln => soln)