Can't read old JLD2 file

bjarthur · February 12, 2019, 4:02pm

i did an ] update and now i can’t read a .jld2 file made yesterday. a diff on the current and previous Manifest.toml files shows that only the DiffEqDiffTools, LsqFit, NLSolversBase, NaturalSort, Rotations, and WeakRefStrings packages have changed. activating a new empty test environment and installing JLD2 shows that it depends on none of these packages. can anyone explain how JLD2 is so fragile?

crbinz · February 12, 2019, 4:29pm

What error message are you getting?

bjarthur · February 12, 2019, 6:26pm

**ERROR:** BoundsError: attempt to access 8249-element Array{Int64,1} at index [8250]

Stacktrace:

 [1] **getindex** at **./array.jl:731** [inlined]

 [2] **getindex** at **./multidimensional.jl:412** [inlined]

 [3] **read_heap_object(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.GlobalHeapID, ::JLD2.ReadRepresentation{UInt8,UInt8} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/global_heaps.jl:130**

 [4] **jlconvert** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/data.jl:700** [inlined]

 [5] **jlconvert** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/data.jl:766** [inlined]

 [6] **macro expansion** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/dataio.jl:70** [inlined]

 [7] **macro expansion** at **./simdloop.jl:73** [inlined]

 [8] **read_array!** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/dataio.jl:68** [inlined]

 [9] **read_array(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{String,JLD2.Vlen{String}}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:323**

 [10] **read_data(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{String,JLD2.Vlen{String}}, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:173**

 [11] **macro expansion** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datatypes.jl:76** [inlined]

 [12] **read_data(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:152**

 [13] **load_dataset(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:92**

 [14] **jlconvert** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/data.jl:674** [inlined]

 [15] **macro expansion** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/dataio.jl:70** [inlined]

 [16] **macro expansion** at **./simdloop.jl:73** [inlined]

 [17] **read_array!** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/dataio.jl:68** [inlined]

 [18] **read_array(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{AbstractArray{T,1} where T,JLD2.RelOffset}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:323**

 [19] **macro expansion** at **./logging.jl:309** [inlined]

 [20] **read_data(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Any,JLD2.RelOffset}, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:194**

 [21] **macro expansion** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:154** [inlined] (repeats 2 times)

 [22] **macro expansion** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datatypes.jl:72** [inlined]

 [23] **read_data(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:152**

 [24] **load_dataset(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:92**

 [25] **jlconvert** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/data.jl:674** [inlined]

 [26] **macro expansion** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/data.jl:1285** [inlined]

 [27] **jlconvert(** ::JLD2.ReadRepresentation{getfield(JLD2.ReconstructedTypes, Symbol("##DataFrames.DataFrame#373")),JLD2.OnDiskRepresentation{(0, 8),Tuple{Any,getfield(JLD2.ReconstructedTypes, Symbol("##DataFrames.Index#372"))},Tuple{JLD2.RelOffset,JLD2.OnDiskRepresentation{(0, 8),Tuple{Dict{Symbol,Int64},Any},Tuple{JLD2.CustomSerialization{Array,JLD2.RelOffset},JLD2.RelOffset}}()}}()}, ::JLD2.JLDFile{JLD2.MmapIO}, ::Ptr{Nothing}, ::JLD2.RelOffset **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/data.jl:1232**

 [28] **read_data(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{getfield(JLD2.ReconstructedTypes, Symbol("##DataFrames.DataFrame#373")),JLD2.OnDiskRepresentation{(0, 8),Tuple{Any,getfield(JLD2.ReconstructedTypes, Symbol("##DataFrames.Index#372"))},Tuple{JLD2.RelOffset,JLD2.OnDiskRepresentation{(0, 8),Tuple{Dict{Symbol,Int64},Any},Tuple{JLD2.CustomSerialization{Array,JLD2.RelOffset},JLD2.RelOffset}}()}}()}, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/dataio.jl:37**

 [29] **read_data(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1} **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:149**

 [30] **load_dataset(** ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/datasets.jl:92**

 [31] **getindex(** ::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, ::String **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/groups.jl:108**

 [32] **read(** ::JLD2.JLDFile{JLD2.MmapIO}, ::String **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/JLD2.jl:326**

 [33] **#3** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/loadsave.jl:77** [inlined]

 [34] **#jldopen#31(** ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Main, Symbol("##3#4")), ::String **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/loadsave.jl:4**

 [35] **jldopen(** ::Function, ::String **)** at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/loadsave.jl:2**

 [36] top-level scope at **/groups/scicompsoft/home/arthurb/.julia/packages/JLD2/KjBIK/src/loadsave.jl:76*

bjarthur · February 18, 2019, 10:55pm

it could be this. am investigating…

Tamas_Papp · February 19, 2019, 8:02am

FWIW, I am under the impression that JLD2 is effectively abandonned (major outstanding bugs with data corruption are have not been fixed for a long time).

So your best bet is probably reproducing the previous environment, reading back your data, and then saving it in a more timeproof format (eg plain HDF5).

carstenbauer · February 19, 2019, 8:45am

… which is very sad (and bad), IMHO.

First, JLD seemed to be the Julia file format. Then, during the Julia 1.0 transition, JLD was basically abandoned (it was only complaints from a bunch of users that lead to fixes for the most essential things) and JLD2 was praised as the future. And now, JLD2 seems abandoned.

jonathanBieler · February 19, 2019, 8:56am

That’s not ideal to say the least, specially since HDF5 tries to build gcc form source on mac OS (it’s an issue with homebrew), taking hours to install. That said I’ve never had an issue with JLD2, I can even read v0.6 files with v1.0, but maybe I’ve been lucky.

Tamas_Papp · February 19, 2019, 9:05am

I think that something like JLD2 would have been nice, but in general, Julia’s type system is too rich for something like this to work reliably without a lot of effort.

A native HDF5 implementation would be useful though, but that is also a lot of work.

I think that the existence of JLD2 in its current state misleads users into considering it as a reliable solution for long- or medium-term data externalization, with unpleasant surprises and costly debugging and data reproduction later on.

tobias.knopp · February 19, 2019, 9:43am

Installation of HDF5 is kindof an orthogonal issue. HDF5 is well known to be the file format of choice for data and I have not seen compatibility issues with that. JLD can work fine but it depends on the types that are being used.

What should be clear is that plain HDF5 can require manual conversion (e.g. for complex numbers). What would be great if we get better support for compound datatypes which will improve the situation there. Also a stable way of storing Dicts (and restoring them) would be great.

Tamas_Papp · February 19, 2019, 9:57am

I disagree with this — I think the best solution is to support datatypes which have a native or near-native correspondence with what is supported by HDF5, and require explicit conversion by the user for the rest.

This seems like an unnecessary overhead, but making the mapping automatic is not a strategy that works well even in the medium run, as such mappings inevitable break down, sometimes silently (with no error, just corrupted data).

IMO this has been the source of a lot of grief for users who thought that they can just use JLD2 or similar as a fast, native, and reasonably robust data storage format. I think that there are two viable approaches:

emphemeral data one can regenerate at little cost: Serialization.serialize and friends or Mmap.mmap, depending on the details,
structured data that is costly to regenerate: map to something that is supported by HDF5, and for long-term storage write a lot of relevant metadata, including strings with explanations, and hope for the best.

tobias.knopp · February 19, 2019, 10:00am

I have not said automatic. Currently the compound datatype support in HDF5.jl is pretty limited. One gets a binary blob when reading compound data.

carstenbauer · February 19, 2019, 10:09am

FWIW, I always used/saw JLD as HDF5 + convenience conversions. I’d be fine if we restrict the convenience part to a basic subset of types, ComplexF64 being the most important I guess.

Juan · February 19, 2019, 12:08pm

What should we use then?
JLD2, HDF5, Feather, Parquet, JLD, fst…?

Some people say JLD produce memory leakage.

It would be great to have something like TileDB. Or something fast, escalable and able to add rows and columns on disk.

bjarthur · February 19, 2019, 12:11pm

perhaps between the seven of us we can work to fix and improve JLD2. volunteers are the nature of open source projects like julia afterall. just need to get familiar with the code, and have someone reliable with write access to merge the PRs.

Tamas_Papp · February 19, 2019, 12:26pm

I detailed my suggestions above. TL;DR: HDF5 or serialization.

I am afraid that the problem is more difficult than this. Julia has a very rich type system which also maps to low-level building blocks very efficiently, and mapping complex nested data into any representation while keeping it fast is inherently very difficult. Almost all similar packages ran into this broad issue.

And while being able to save and load very complex data effortlessly and in a way that promises long-run compatibility is inherently appealing, I am not sure it is possible or even desirable. Again, for short-term storage we have Serialization, and for long-term storage, can we guarantee that something like

NamedTuple{(:a, :b), Tuple{Union{Missing,Integer}, AbstractString}}[(a = 1, b = "foo")]

will even make sense 5 years from now? If no, do we want to reconstruct the exact container types, or should they not matter?

I find it is worth mapping the data to arrays of primitives like Float64, and save that. For this, HDF5 is perfect.

Tomas_Pevny · February 19, 2019, 12:56pm

I agree with the complex times. At the moment, we rely on FlatBuffers instead of hdf5, as the problem with memory leak was on the side of hdf5.

mkborregaard · February 19, 2019, 2:55pm

Edited - possibly misleading

tobias.knopp · February 19, 2019, 4:21pm

I think this is a very promising step forward. HDF5.jl as the low level interface and then some HDF5Extension.jl where we implement some automatic or semiautomatic conversion rule. But in contrast to JLD, HDF5Extension.jl would only support conversions where we are totally sure that they will work independently of the Julia version. For instance, here is code to store complex arrays:

github.com

MagneticParticleImaging/MPIFiles.jl/blob/master/src/MPIFiles.jl#L384


      
          module MPIFiles
          
          using UUIDs
          using Graphics: @mustimplement
          
          using LinearOperatorCollection
          using FFTW
          using AxisArrays
          const axes = Base.axes
          using Interpolations
          using HDF5
          using Dates
          using DelimitedFiles
          using ImageMetadata
          using ImageAxes
          using LinearAlgebra
          using Random
          using Mmap
          using Statistics
          using Unitful
          using CodecZlib
          using Tar
          using Pkg.PlatformEngines
          using Pkg.GitTools
          using Pkg.Artifacts
          using Unitful
          using InteractiveUtils
          using UnitfulAngles
          using Inflate, SHA
          using StableRNGs
          using REPL: fielddoc
          using DocStringExtensions
          
          ### global import list ###
          
          import Base: convert, get, getindex, haskey, iterate, length, ndims, range, read, show, time, write, close
          import FileIO: save
          import HDF5: h5read
          import Interpolations: interpolate
          
          ### export list ###
          
          export MPIFile
          
          # general parameters
          export version, uuid
          
          # study parameters
          export studyName, studyNumber, studyUuid, studyDescription, studyTime
          
          # experiment parameters
          export experimentName, experimentNumber, experimentUuid, experimentDescription, experimentSubject,
            experimentIsSimulation, experimentIsCalibration,
            experimentHasMeasurement, experimentHasReconstruction
          
          # tracer parameters
          export tracerName, tracerBatch, tracerVolume, tracerConcentration,
            tracerSolute, tracerInjectionTime, tracerVendor
          
          # scanner parameters
          export scannerFacility, scannerOperator, scannerManufacturer, scannerName,
            scannerTopology
          
          # acquisition parameters
          export acqStartTime, acqNumFrames, acqNumAverages,
            acqGradient, acqOffsetField, acqNumPeriodsPerFrame
          
          # drive-field parameters
          export dfNumChannels, dfStrength, dfPhase, dfBaseFrequency, dfCustomWaveform,
            dfDivider, dfWaveform, dfCycle
          
          # receiver parameters
          export rxNumChannels, rxBandwidth, rxNumSamplingPoints,
            rxTransferFunction, rxTransferFunctionFileName, rxHasTransferFunction, rxUnit,
            rxDataConversionFactor, rxInductionFactor
          
          # measurements
          export measData, measDataTDPeriods, measIsFourierTransformed, measIsTFCorrected,
            measIsTransferFunctionCorrected,
            measIsBGCorrected, measIsBackgroundCorrected, measIsFastFrameAxis,
            measIsFramePermutation, measIsFrequencySelection,
            measIsBGFrame, measIsBackgroundFrame, measIsSpectralLeakageCorrected, measFramePermutation,
            measFrequencySelection, measIsSparsityTransformed, measIsCalibProcessed
          
          # calibrations
          export calibSNR, calibSnr, calibFov, calibFieldOfView, calibFovCenter,
            calibFieldOfViewCenter, calibSize, calibOrder, calibPositions,
            calibOffsetFields, calibDeltaSampleSize,
            calibMethod, calibIsMeanderingGrid
          
          # reconstruction results
          export recoData, recoFov, recoFieldOfView, recoFovCenter, recoFieldOfViewCenter,
            recoSize, recoOrder, recoPositions
          
          # additional functions that should be implemented by an MPIFile
          export filepath, systemMatrixWithBG, systemMatrix
          
          export selectedChannels
          ### Interface of an MPIFile ###
          
          abstract type MPIFile end
          
          # general parameters
          @mustimplement version(f::MPIFile)
          @mustimplement uuid(f::MPIFile)
          @mustimplement time(f::MPIFile)
          
          # study parameters
          @mustimplement studyName(f::MPIFile)
          @mustimplement studyNumber(f::MPIFile)
          @mustimplement studyUuid(f::MPIFile)
          @mustimplement studyDescription(f::MPIFile)
          @mustimplement studyTime(f::MPIFile)
          
          # experiment parameters
          @mustimplement experimentName(f::MPIFile)
          @mustimplement experimentNumber(f::MPIFile)
          @mustimplement experimentUuid(f::MPIFile)
          @mustimplement experimentDescription(f::MPIFile)
          @mustimplement experimentSubject(f::MPIFile)
          @mustimplement experimentIsSimulation(f::MPIFile)
          @mustimplement experimentIsCalibration(f::MPIFile)
          @mustimplement experimentHasReconstruction(f::MPIFile)
          @mustimplement experimentHasMeasurement(f::MPIFile)
          
          # tracer parameters
          @mustimplement tracerName(f::MPIFile)
          @mustimplement tracerBatch(f::MPIFile)
          @mustimplement tracerVolume(f::MPIFile)
          @mustimplement tracerConcentration(f::MPIFile)
          @mustimplement tracerSolute(f::MPIFile)
          @mustimplement tracerInjectionTime(f::MPIFile)
          
          # scanner parameters
          @mustimplement scannerBoreSize(f::MPIFile)
          @mustimplement scannerFacility(f::MPIFile)
          @mustimplement scannerOperator(f::MPIFile)
          @mustimplement scannerManufacturer(f::MPIFile)
          @mustimplement scannerName(f::MPIFile)
          @mustimplement scannerTopology(f::MPIFile)
          
          # acquisition parameters
          @mustimplement acqStartTime(f::MPIFile)
          @mustimplement acqNumAverages(f::MPIFile)
          @mustimplement acqNumPeriodsPerFrame(f::MPIFile)
          @mustimplement acqNumFrames(f::MPIFile)
          @mustimplement acqGradient(f::MPIFile)
          @mustimplement acqOffsetField(f::MPIFile)
          
          # drive-field parameters
          @mustimplement dfNumChannels(f::MPIFile)
          @mustimplement dfStrength(f::MPIFile)
          @mustimplement dfPhase(f::MPIFile)
          @mustimplement dfBaseFrequency(f::MPIFile)
          @mustimplement dfCustomWaveform(f::MPIFile)
          @mustimplement dfDivider(f::MPIFile)
          @mustimplement dfWaveform(f::MPIFile)
          @mustimplement dfCycle(f::MPIFile)
          
          # receiver properties
          @mustimplement rxNumChannels(f::MPIFile)
          @mustimplement rxBandwidth(f::MPIFile)
          @mustimplement rxNumSamplingPoints(f::MPIFile)
          @mustimplement rxTransferFunction(f::MPIFile)
          @mustimplement rxTransferFunctionFileName(f::MPIFile)
          @mustimplement rxHasTransferFunction(f::MPIFile)
          @mustimplement rxInductionFactor(f::MPIFile)
          @mustimplement rxUnit(f::MPIFile)
          @mustimplement rxDataConversionFactor(f::MPIFile)
          
          # measurements
          @mustimplement measData(f::MPIFile)
          @mustimplement measDataTD(f::MPIFile)
          @mustimplement measDataTDPeriods(f::MPIFile, periods)
          @mustimplement measFramePermutation(f::MPIFile)
          @mustimplement measFrequencySelection(f::MPIFile)
          @mustimplement measIsBGCorrected(f::MPIFile)
          @mustimplement measIsBGFrame(f::MPIFile)
          @mustimplement measIsFastFrameAxis(f::MPIFile)
          @mustimplement measIsFourierTransformed(f::MPIFile)
          @mustimplement measIsFramePermutation(f::MPIFile)
          @mustimplement measIsFrequencySelection(f::MPIFile)
          @mustimplement measIsSparsityTransformed(f::MPIFile)
          @mustimplement measIsSpectralLeakageCorrected(f::MPIFile)
          @mustimplement measIsTFCorrected(f::MPIFile)
          @mustimplement measSparsityTransformation(f::MPIFile)
          @mustimplement measSubsamplingIndices(f::MPIFile)
          @mustimplement measIsCalibProcessed(b::MPIFile)
          
          # calibrations
          @mustimplement calibSNR(f::MPIFile)
          @mustimplement calibFov(f::MPIFile)
          @mustimplement calibFovCenter(f::MPIFile)
          @mustimplement calibSize(f::MPIFile)
          @mustimplement calibOrder(f::MPIFile)
          @mustimplement calibPositions(f::MPIFile)
          @mustimplement calibOffsetFields(f::MPIFile)
          @mustimplement calibDeltaSampleSize(f::MPIFile)
          @mustimplement calibMethod(f::MPIFile)
          @mustimplement calibIsMeanderingGrid(f::MPIFile)
          
          # reconstruction results
          @mustimplement recoData(f::MPIFile)
          @mustimplement recoFov(f::MPIFile)
          @mustimplement recoFovCenter(f::MPIFile)
          @mustimplement recoSize(f::MPIFile)
          @mustimplement recoOrder(f::MPIFile)
          @mustimplement recoPositions(f::MPIFile)
          @mustimplement recoIsOverscanRegion(f::MPIFile)
          
          # additional functions that should be implemented by an MPIFile
          @mustimplement filepath(f::MPIFile)
          
          
          include("Derived.jl")
          include("Custom.jl")
          include("Utils.jl")
          include("FramePermutation.jl")
          
          ### Concrete implementations ###
          include("MDF.jl")
          include("MDFInMemory.jl")
          include("MDFCommon.jl")
          include("Brukerfile.jl")
          include("IMT.jl")
          
          # This dispatches on the file extension and automatically
          # generates the correct type
          function MPIFile(filename::AbstractString; kargs...)
            filenamebase, ext = splitext(filename)
            if ext == ".mdf" || ext == ".hdf" || ext == ".h5"
              file = h5open(filename, "r")
              if haskey(file, "/version")
                return MDFFile(filename, file) # MDFFile currently has no kargs
              else
                return IMTFile(filename, file; kargs...)
              end
            else
              if isfile(joinpath(filename, "mdf"))
                filenameMDF = readline(joinpath(filename, "mdf"))
                return MDFFile(filenameMDF)
              else
                return BrukerFile(filename; kargs...)
              end
            end
          end
          
          function show(io::IO, f::MPIFile)
            print(io, supertype(typeof(f)))
            print(io, "\n\tStudy: ")
            show(io, studyName(f))
            print(io, ", ")
            show(io, studyTime(f))
            print(io, "\n\tExperiment: ")
            show(io, experimentName(f))
            print(io, ", ")
            show(io, acqStartTime(f))
            print(io, "\n")
          end
          
          # Opens a set of MPIFiles
          function MPIFile(filenames::Vector)
            return map(x -> MPIFile(x), filenames)
          end
          
          # For the do block
          function MPIFile(h::Function, args...; kargs...)
            f = MPIFile(args...; kargs...)
            try
              h(f)
            finally
              close(f)
            end
          end
          
          Base.length(f::MPIFile) = 1
          Base.close(f::MPIFile) = nothing
          
          include("TransferFunction.jl")
          include("MultiMPIFile.jl")
          include("Measurements.jl")
          include("SystemMatrix.jl")
          include("FrequencyFilter.jl")
          include("Conversion.jl")
          include("RecoData.jl")
          include("DatasetStore/DatasetStore.jl")
          include("MixingFactors.jl")
          include("Positions/Positions.jl")
          include("MagneticFieldMeasurement.jl")
          
          end # module

This file has been truncated. show original

It uses the same convention as Octave, where re/im are put into a compound data type. So all is allocation free and layed out in memory as is the Julia array.