How do you read structures from a file?

[Zweta_Fuze]

Good wishes for the New year.
I want to read data that are coming from a device. These data are organized in frames with a fixed structure each one. To give an example I want to read a file “Data.dat” with 100 frames. I intend to do something like
‘’'struct Frame # frame consists of a string with 13 chars, 10 Int16, and 20 Float32=113Byte
s::String{13}
n::Vector{Int16}(undef,10)
x::Vector{Float32}(undef,20)
end

f=open(“Data.dat”,“r”) # open file
for i=1:100 # read 100 frames
read!(f,Frame) # read one frame to have access to Frame.s, Frame.n, and Frame.x
end
close(f) # close file
‘’’
Of course, this snippet of code doesn’t work. But I hope I could express my intention.
How must the code look like to work?

Best regards Zweta

[aditi_dey29]

Maybe you can try this:

struct Frame # frame consists of a string with 13 chars, 10 Int16, and 20 Float32=113Byte
    s::String
    n::Vector{Int16}
    x::Vector{Float32}
end

function readframe(f::IO)
    s = readstring(f, 13)
    n = read(f, Vector{Int16}, 10)
    x = read(f, Vector{Float32}, 20)
    return Frame(s, n, x)
end

f=open("Data.dat", "r") # open file
for i=1:100 # read 100 frames
    frame = readframe(f) # read one frame to have access to Frame.s, Frame.n, and Frame.x
end
close(f) # close file

This code will read the file “Data.dat” and for each of the 100 frames it will read a string of 13 characters, followed by 10 Int16 values, and finally 20 Float32 values. It will then store these values in a Frame struct and loop to the next frame.

[GunnarFarneback]

@aditi_dey29

function readframe(f::IO)
    s = readstring(f, 13)
    n = read(f, Vector{Int16}, 10)
    x = read(f, Vector{Float32}, 20)
    return Frame(s, n, x)
end

Where does readstring come from? Where do the read(f, Vector{T}, n) methods come from?

[giordano]

An example, not necessarily the best solution:

julia> struct Frame
           s::NTuple{13,Char}
           n::NTuple{10,Int16}
           x::NTuple{20,Float32}
       end

julia> Frame() = Frame(
           (rand(Char, 13)...,),
           (rand(Int16, 10)...,),
           (rand(Float32, 20)...,),
       )
Frame

julia> function write_frame(filename::String, frame::Frame)
           open(filename, "w") do io
               for field in fieldnames(Frame)
                   for x in getfield(frame, field)
                       write(io, x)
                   end
               end
           end
       end
write_frame (generic function with 1 method)

julia> function read_frame(filename::String)
           return open(filename, "r") do io
               s = ntuple(_ -> read(io, Char),    13)
               n = ntuple(_ -> read(io, Int16),   10)
               x = ntuple(_ -> read(io, Float32), 20)
               Frame(s, n, x)
           end
       end
read_frame (generic function with 1 method)

julia> a = Frame()
Frame(('\Ue2c3c', '\U6ce34', '𫁝', '\Ucde12', '\Ue9496', '\U7fb68', '\U12baf', '\U35114', '\U5b688', '𔑿', '𐤐', '\U6b4ef', '\U47830'), (-26877, -29246, -15065, -30020, -26097, -11079, -7182, -25570, 31616, -2110), (0.7977007f0, 0.3373096f0, 0.74674666f0, 0.5593646f0, 0.23270303f0, 0.72000647f0, 0.1837818f0, 0.67431325f0, 0.43312347f0, 0.3558365f0, 0.9129793f0, 0.62474954f0, 0.013182282f0, 0.68477046f0, 0.37474865f0, 0.6750861f0, 0.81095505f0, 0.556222f0, 0.10530347f0, 0.5351166f0))

julia> write_frame("file.dat", a)

julia> read_frame("file.dat") === a
true

[Zweta_Fuze]

Thank you very much for your quick answer. The problem is that the size of the Frame hasn’t the size of 113Byte but 160Byte instead. I was looking for a method to have it 113Byte. I’m sorry, I should have pointed this out.
Best regards Zweta

[Zweta_Fuze]

Hello giordano,
thank you for your quick response. The code works fine. I’m sorry for not having pointed out the fact that I need the structure to be 113Byte of size. So in this example I would read 113Byte of data and want them to be interpreted as 113Byte String + 10 numbers of Int16 + 20 numbers of Float32. That’s the difficulty. I haven’t found anything about this in the documentation.
Best regards Zweta

[stevengj]

@aditi_dey29

function readframe(f::IO)
    s = readstring(f, 13)
    n = read(f, Vector{Int16}, 10)
    x = read(f, Vector{Float32}, 20)
    return Frame(s, n, x)
end

Right idea, but these aren’t valid methods. To make this approach work, you want something more like:

function readframe(f::IO)
    s = String(read(f, 13)) # read 13 bytes and interpret them as a String
    n = Vector{Int16}(undef, 10)
    x = Vector{Float32}(undef, 20)
    if sizeof(s) != 13 ||
       readbytes!(f, reinterpret(UInt8, n)) != sizeof(n) ||
       readbytes!(f, reinterpret(UInt8, x)) != sizeof(x)
        error("failed to read a complete frame")
    end
    return Frame(s, n, x)
end

[stevengj]

@giordano

struct Frame
           s::NTuple{13,Char}
           n::NTuple{10,Int16}
           x::NTuple{20,Float32}
       end

From the OP’s description, @Zweta_Fuze is reading strings consisting of 13 ASCII characters (or 13 UTF-8 code units), and not 13 Chars`. So you need:

struct Frame
    s::NTuple{13,UInt8}
    n::NTuple{10,Int16}
    x::NTuple{20,Float32}
end

But you will still run into a problem with struct padding — the compiler pads Frame with 3 extra bytes so that n starts on a 32-bit word boundary.

[giordano]

@stevengj

But you will still run into a problem with struct padding — the compiler pads Frame with 3 extra bytes so that n starts on a 32-bit word boundary.

Yes, I should have used UInt8 instead of Char, but how’s padding a problem here?

julia> struct Frame
           s::NTuple{13,UInt8}
           n::NTuple{10,Int16}
           x::NTuple{20,Float32}
       end

julia> Frame() = Frame(
           (rand(UInt8, 13)...,),
           (rand(Int16, 10)...,),
           (rand(Float32, 20)...,),
       )
Frame

julia> function write_frame(filename::String, frame::Frame)
           open(filename, "w") do io
               for field in fieldnames(Frame)
                   for x in getfield(frame, field)
                       write(io, x)
                   end
               end
           end
       end
write_frame (generic function with 1 method)

julia> function read_frame(filename::String)
           return open(filename, "r") do io
               s = ntuple(_ -> read(io, UInt8),   13)
               n = ntuple(_ -> read(io, Int16),   10)
               x = ntuple(_ -> read(io, Float32), 20)
               Frame(s, n, x)
           end
       end
read_frame (generic function with 1 method)

julia> a = Frame()
Frame((0xb7, 0x6c, 0x28, 0x23, 0x0c, 0xb8, 0x20, 0xfb, 0x07, 0xb2, 0xaf, 0xff, 0xec), (-13167, -21111, -26729, -5595, -30379, -19431, 25308, -17026, -14928, 8709), (0.8693912f0, 0.520382f0, 0.95749295f0, 0.86897546f0, 0.4634325f0, 0.33964092f0, 0.55640644f0, 0.64960533f0, 0.7045306f0, 0.2612428f0, 0.66503394f0, 0.75041735f0, 0.5394455f0, 0.44677263f0, 0.41727632f0, 0.2482816f0, 0.50361246f0, 0.49845558f0, 0.97010577f0, 0.47037232f0))

julia> write_frame("file.dat", a)

julia> read_frame("file.dat") === a
true

[stevengj]

@giordano

Yes, I should have used UInt8 instead of Char, but how’s padding a problem here?

julia> struct Frame
           s::NTuple{13,UInt8}
           n::NTuple{10,Int16}
           x::NTuple{20,Float32}
       end

It’s padded with 3 extra bytes, so each frame is not 113 bytes:

julia> sizeof(Frame)
116

Update Sorry, I misunderstood you — I was thinking that you just wanted to do read(io, Frame), which would fail with padding.

The padding is problem if you want to read the bytes manually (similar to the readframe(f) function above), but in that case I would probably store it as a String (or some related type) and not a tuple, to be more natural to work with in Julia once you have read it.

[stevengj]

@Zweta_Fuze

‘’'struct Frame # frame consists of a string with 13 chars, 10 Int16, and 20 Float32=113Byte

There used to be a nice package called StrPack.jl for this sort of thing, but it stopped being maintained. But @Keno put together a replacement called StructIO.jl, although it doesn’t seem to have gotten a lot of attention in the last couple of years. StructIO is a registered package and its tests still pass, however, so it would be worth trying.

Also, it could give a cleaner API if you use StaticArrays.jl and StaticStrings.jl instead of tuples for the struct fields.

For example:

using StructIO, StaticStrings, StaticArrays

@io struct Frame
    s::StaticString{13}
    n::SVector{10, Int16}
    x::SVector{20, Float32}
end align_packed

which gives an on-disk size of 113 bytes:

julia> packed_sizeof(Frame)
113

Then you should be able to read from a file f with simply:

f = open("data.dat", "r")
data = [unpack(f, Frame) for i = 1:100]

@Keno, it would be nice if StructIO.jl could be transferred to JuliaIO and given a little TLC, especially more documentation.

[aditi_dey29]

Yep right! Thanks

[Zweta_Fuze]

Wow, that’s just amazing. Thank you very much. It works exactly as I wanted.
Just one little more question that might be easy for you.

How is it possible to fill the structure with data in Julia and how can I write this to a file?

That’s all I would need.

Best regards Zweta

[stevengj]

@Zweta_Fuze

How is it possible to fill the structure with data in Julia and how can I write this to a file?

With my StructIO example above, you can do e.g.:

d = Frame("foo", 1:10, rand(20)) # initialize with some data
open("bar.dat", "w") do f    # open file for writing as f
    pack(f, d) # write d as packed binary struct to file
end

[Zweta_Fuze]

@stevengj

Frame("foo", 1:10, rand(20))

Thank you very much for your quick response. This completes all my open questions. I was hoping to find these examples in the Julia documentation but I didn’t find any. S o I thank you very much.
Best regards Zweta

[Zweta_Fuze]

@everybody
The answering of the above question is successfully completed. I thank everybody for the quick response. Thank you very much.
Best regards Zweta

[Zweta_Fuze]

Thanks from the help of
aditi_dey29, GunnarFarneback, giordano, and stevengj
I can summarize the following answer to the question above.
The following text can be copied in the REPL and is supposed to run
without error messages.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# Example how to write and read a DataStructure to and from a file "bar.dat"

# Packages needed
using StructIO,StaticStrings,StaticArrays

# determine structure to be read as an example of a data format
@io struct Frame 
       s::StaticString{13}
       n::SVector{10,Int16}
       x::SVector{20,Float32}
    end align_packed

# show size of that structure       
println(packed_sizeof(Frame)) # sizeof Frame to be written

# generate some data as an example -> Fw
Fw=Frame("up to max 13",1:10,rand(20))

# write the contents of the frame to the file "bar.dat"
f=open("bar.dat","w")
pack(f,Fw)
close(f)

# show the written data
println(Fw)

# show the size of the file "bar.dat"
# this is supposed to be 13*8+10*2+20*4 = 113 byte
println(run(`ls -l bar.dat`)) # there we go

# read one frame of the file "bar.dat"
f=open("bar.dat","r")
Fr=unpack(f,Frame)
close(f)

# show the contents of the frame read
println(Fr) # Frame read

# compare the contents written and read which is supposed to be true as one can see
println(Fw==Fr)

[Dan]

Final tip: Use triple quotes (```) before and after multiline code section to format it properly in Discourse. Pressing edit on the last post above, is a good place to try it.

[Zweta_Fuze]

Well, thanks Dan. Tried it first but it didn’t work because I didn’t use the backquotes.