Reading sequential fortran files with variable record length

New to Julia
1

Hello, new to Julia, coming from python. I have data files produced by sequential fortran write with records of the following format:

  • First goes a header with metadata, an array of 14 float32 numbers;
  • The next record is a big 2d array of float32 data, which correspond to the header above.

Each file contains a number of such pairs of records. I read these files with FortranFiles.jl package. My question is what would be the most convenient and efficient data structure to store array of such records in memory besides keeping separate float32 arrays for headers and data, such that I could do things like

println(var["header"][end-1][5])
contour(var["data"][end-1][:,:])

etc? Something like numpy associative arrays? Also, could someone criticize this code, i.e. tell me if it is flawed in some way or could be improved:

using FortranFiles
function readforcing(path)

  # Get Forcing file metadata
  file=FortranFile(path)
  hsize=14
  header=read(file, (Float32,hsize))
  im=convert(Int, header[end-1])
  jm=convert(Int, header[end])

  # Read data
  rewind(file)
  header=Array{Float64}[]
  data=Array{Float64}[]
  while !eof(file)
    push!(header,read(file,(Float32,hsize)))
    push!(data,read(file,(Float32,im,jm)))
  end

  close(file)

  return header, data
end

? I appreciate all responses.

2

I would do something like the following, which puts all of the headers and data into a single vector:

(Edited to correct error noted by OP)

using FortranFiles, StaticArrays
const HSIZE = 14  # length of Float32 header 

struct FData
    header::SVector{HSIZE, Float32}
    data::Matrix{Float32}
end

function readforcing(path)
    fdata = FData[]
    open(path) do io
        file = FortranFile(io)
        im = jm = 0
        while !eof(file)
            header = read(file, (Float32, HSIZE))
            im == 0 && (im, jm = convert.(Int, header[end-1:end]))
            data = read(file, (Float32, im, jm))
            push!(fdata, FData(header, data))
        end
    end
    
    return fdata
end

Usage would be like this:

...
fdata = readforcing(path)
dat = fdata[end-1]
println(dat.header[5])
contour(dat.data)
...
1 Like
3

Fantastic! Thanks for tips (StaticArrays, do block, short circuit, etc). Result is identical to my less advanced variant. Performance wise everything is also good (about same as my variant). The only thing which I needed to correct is to set jm=0 outside the while loop too, otherwise it is undefined after first iteration.

4

Yes, sorry I forgot about that. Otherwise it is local to the loop and as you say, becomes undefined after the first iteration.