Loading a large complex matrix or array from a *.jl file

New to Julia
1

Dear all,

I hope to find some help here regarding loading a (relatively) large complex array stored together with some other parameters within a .jl file.
The .jl data file has the following structure:

# A comment with a short explanation of what the data is
scalar parameter  = 1
tuple = (var1 =1, var2 = 2)
energy_array = ComplexF64[...]
wavefunction_array = [...]

Some more explanations for the variables: energy_array has size (7,60) and contains complex values.
wavefunction_array has size (7,20000,60) and contains complex values.

The first dimension (of size 7) is a numerical parameter, for which I would like to check convergence. The last dimension (of size 60) is the number of eigenvalues/eigenfunctions which I saved. The dimension of size 20000 is the size of my eigenvectors.

Before, I only saved the eigenvalues (energy_array) leading to a file size of some kB, and loading the file via include(filename) did the job, so that I could further analyze the results. Now, recently I also calculated the eigenfunctions resulting in a file size of up to a few hundret Megabytes. Attempting to load the file via include(filename) leads to ERROR: LoadErrow: StackOverflowError.

Any insight, how I can load the file? I dont need the full wavefunction_array at once, it would be ok if I can do something like load(filename, wavefunction_array[num_param_val,:,state_index]).

Maybe another fileformat istead of “.jl” would be more suitable for this task? I can redo my current calculations (will take 1-2 days) if you tell me the “.jl” fileformat is not really suitable for larger files.

Many thanks.

2

You probably want a JLD file:

3

Thank you, can you tell me if I will be able to convert my filename.jl directly to filename.jld, or should I redo my calculation and save the results into a .jld file?

After a quick glance, I could not find this information in the general info of the linked package.

Best regards

4

A .jl file is a text file. If you cannot include it directly, then you’re going to have to parse it a different way.

A .jld file is a binary HDF5 file.

I do not know how expensive your calculation is. In terms of your time, it probably is easier just to redo your calculation and save it via JLD.jl.