# How do you read structures from a file?

**URL:** <https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646>\
**Category:** New to Julia\
**Tags:** binaryio\
**Created:** [January 7, 2023, 7:59pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646 "2023-01-07T19:59:53Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 7, 2023, 7:59pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/1 "2023-01-07T19:59:53Z")

</div>

Good wishes for the New year.  
I want to read data that are coming from a device. These data are organized in frames with a fixed structure each one. To give an example I want to read a file “Data.dat” with 100 frames. I intend to do something like  
‘’'struct Frame # frame consists of a string with 13 chars, 10 Int16, and 20 Float32=113Byte  
s::String{13}  
n::Vector{Int16}(undef,10)  
x::Vector{Float32}(undef,20)  
end

f=open(“Data.dat”,“r”) # open file  
for i=1:100 # read 100 frames  
read!(f,Frame) # read one frame to have access to Frame.s, Frame.n, and Frame.x  
end  
close(f) # close file  
‘’’  
Of course, this snippet of code doesn’t work. But I hope I could express my intention.  
How must the code look like to work?

Best regards Zweta

---

<div class="post-metadata">

**Author:** ![aditi\_dey29](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aditi_dey29/32/45696_2.png) [@aditi\_dey29](https://discourse.julialang.org/u/aditi_dey29)\
**Post date:** [January 8, 2023, 10:20am UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/2 "2023-01-08T10:20:36Z")

</div>

Maybe you can try this:

```julia
struct Frame # frame consists of a string with 13 chars, 10 Int16, and 20 Float32=113Byte
    s::String
    n::Vector{Int16}
    x::Vector{Float32}
end

function readframe(f::IO)
    s = readstring(f, 13)
    n = read(f, Vector{Int16}, 10)
    x = read(f, Vector{Float32}, 20)
    return Frame(s, n, x)
end

f=open("Data.dat", "r") # open file
for i=1:100 # read 100 frames
    frame = readframe(f) # read one frame to have access to Frame.s, Frame.n, and Frame.x
end
close(f) # close file

```

This code will read the file “Data.dat” and for each of the 100 frames it will read a string of 13 characters, followed by 10 `Int16` values, and finally 20 `Float32` values. It will then store these values in a `Frame` struct and loop to the next frame.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [January 8, 2023, 10:39am UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/3 "2023-01-08T10:39:43Z")

</div>

> [@aditi\_dey29](#):
>
> ```julia
> function readframe(f::IO)
> s = readstring(f, 13)
> n = read(f, Vector{Int16}, 10)
> x = read(f, Vector{Float32}, 20)
> return Frame(s, n, x)
> end
> 
> ```

Where does `readstring` come from? Where do the `read(f, Vector{T}, n)` methods come from?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 8, 2023, 11:32am UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/4 "2023-01-08T11:32:58Z")

</div>

An example, not necessarily the best solution:

```julia
julia> struct Frame
           s::NTuple{13,Char}
           n::NTuple{10,Int16}
           x::NTuple{20,Float32}
       end

julia> Frame() = Frame(
           (rand(Char, 13)...,),
           (rand(Int16, 10)...,),
           (rand(Float32, 20)...,),
       )
Frame

julia> function write_frame(filename::String, frame::Frame)
           open(filename, "w") do io
               for field in fieldnames(Frame)
                   for x in getfield(frame, field)
                       write(io, x)
                   end
               end
           end
       end
write_frame (generic function with 1 method)

julia> function read_frame(filename::String)
           return open(filename, "r") do io
               s = ntuple(_ -> read(io, Char), 13)
               n = ntuple(_ -> read(io, Int16), 10)
               x = ntuple(_ -> read(io, Float32), 20)
               Frame(s, n, x)
           end
       end
read_frame (generic function with 1 method)

julia> a = Frame()
Frame(('\Ue2c3c', '\U6ce34', '𫁝', '\Ucde12', '\Ue9496', '\U7fb68', '\U12baf', '\U35114', '\U5b688', '𔑿', '𐤐', '\U6b4ef', '\U47830'), (-26877, -29246, -15065, -30020, -26097, -11079, -7182, -25570, 31616, -2110), (0.7977007f0, 0.3373096f0, 0.74674666f0, 0.5593646f0, 0.23270303f0, 0.72000647f0, 0.1837818f0, 0.67431325f0, 0.43312347f0, 0.3558365f0, 0.9129793f0, 0.62474954f0, 0.013182282f0, 0.68477046f0, 0.37474865f0, 0.6750861f0, 0.81095505f0, 0.556222f0, 0.10530347f0, 0.5351166f0))

julia> write_frame("file.dat", a)

julia> read_frame("file.dat") === a
true

```

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 8, 2023, 12:42pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/5 "2023-01-08T12:42:52Z")

</div>

Thank you very much for your quick answer. The problem is that the size of the Frame hasn’t the size of 113Byte but 160Byte instead. I was looking for a method to have it 113Byte. I’m sorry, I should have pointed this out.  
Best regards Zweta

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 8, 2023, 12:49pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/6 "2023-01-08T12:49:08Z")

</div>

Hello giordano,  
thank you for your quick response. The code works fine. I’m sorry for not having pointed out the fact that I need the structure to be 113Byte of size. So in this example I would read 113Byte of data and want them to be interpreted as 113Byte String + 10 numbers of Int16 + 20 numbers of Float32. That’s the difficulty. I haven’t found anything about this in the documentation.  
Best regards Zweta

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 8, 2023, 1:03pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/7 "2023-01-08T13:03:48Z")

</div>

> [@aditi\_dey29](#):
>
> ```julia
> function readframe(f::IO)
> s = readstring(f, 13)
> n = read(f, Vector{Int16}, 10)
> x = read(f, Vector{Float32}, 20)
> return Frame(s, n, x)
> end
> 
> ```

Right idea, but these aren’t valid methods. To make this approach work, you want something more like:

```julia
function readframe(f::IO)
    s = String(read(f, 13)) # read 13 bytes and interpret them as a String
    n = Vector{Int16}(undef, 10)
    x = Vector{Float32}(undef, 20)
    if sizeof(s) != 13 ||
       readbytes!(f, reinterpret(UInt8, n)) != sizeof(n) ||
       readbytes!(f, reinterpret(UInt8, x)) != sizeof(x)
        error("failed to read a complete frame")
    end
    return Frame(s, n, x)
end

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 8, 2023, 1:14pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/8 "2023-01-08T13:14:00Z")

</div>

> [@giordano](#):
>
> ```julia
> struct Frame
> s::NTuple{13,Char}
> n::NTuple{10,Int16}
> x::NTuple{20,Float32}
> end
> 
> ```

From the OP’s description, @Zweta_Fuze is reading strings consisting of 13 _ASCII_ characters (or 13 UTF-8 code units), and not 13 `Char`s`. So you need:

```julia
struct Frame
    s::NTuple{13,UInt8}
    n::NTuple{10,Int16}
    x::NTuple{20,Float32}
end

```

But you will still run into a problem with struct padding — the compiler pads `Frame` with 3 extra bytes so that `n` starts on a 32-bit word boundary.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 8, 2023, 1:18pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/9 "2023-01-08T13:18:18Z")

</div>

> [@stevengj](#):
>
> But you will still run into a problem with struct padding — the compiler pads `Frame` with 3 extra bytes so that `n` starts on a 32-bit word boundary.

Yes, I should have used `UInt8` instead of `Char`, but how’s padding a problem here?

```julia
julia> struct Frame
           s::NTuple{13,UInt8}
           n::NTuple{10,Int16}
           x::NTuple{20,Float32}
       end

julia> Frame() = Frame(
           (rand(UInt8, 13)...,),
           (rand(Int16, 10)...,),
           (rand(Float32, 20)...,),
       )
Frame

julia> function write_frame(filename::String, frame::Frame)
           open(filename, "w") do io
               for field in fieldnames(Frame)
                   for x in getfield(frame, field)
                       write(io, x)
                   end
               end
           end
       end
write_frame (generic function with 1 method)

julia> function read_frame(filename::String)
           return open(filename, "r") do io
               s = ntuple(_ -> read(io, UInt8), 13)
               n = ntuple(_ -> read(io, Int16), 10)
               x = ntuple(_ -> read(io, Float32), 20)
               Frame(s, n, x)
           end
       end
read_frame (generic function with 1 method)

julia> a = Frame()
Frame((0xb7, 0x6c, 0x28, 0x23, 0x0c, 0xb8, 0x20, 0xfb, 0x07, 0xb2, 0xaf, 0xff, 0xec), (-13167, -21111, -26729, -5595, -30379, -19431, 25308, -17026, -14928, 8709), (0.8693912f0, 0.520382f0, 0.95749295f0, 0.86897546f0, 0.4634325f0, 0.33964092f0, 0.55640644f0, 0.64960533f0, 0.7045306f0, 0.2612428f0, 0.66503394f0, 0.75041735f0, 0.5394455f0, 0.44677263f0, 0.41727632f0, 0.2482816f0, 0.50361246f0, 0.49845558f0, 0.97010577f0, 0.47037232f0))

julia> write_frame("file.dat", a)

julia> read_frame("file.dat") === a
true

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 8, 2023, 1:22pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/10 "2023-01-08T13:22:10Z")

</div>

> [@giordano](#):
>
> Yes, I should have used `UInt8` instead of `Char`, but how’s padding a problem here?
> 
> ```julia
> julia> struct Frame
> s::NTuple{13,UInt8}
> n::NTuple{10,Int16}
> x::NTuple{20,Float32}
> end
> 
> ```

It’s padded with 3 extra bytes, so each frame is not 113 bytes:

```julia
julia> sizeof(Frame)
116

```

_Update_ Sorry, I misunderstood you — I was thinking that you just wanted to do `read(io, Frame)`, which would fail with padding.

The padding is problem if you want to read the bytes manually (similar to the `readframe(f)` function above), but in that case I would probably store it as a `String` (or some related type) and not a tuple, to be more natural to work with in Julia once you have read it.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 8, 2023, 1:32pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/12 "2023-01-08T13:32:36Z")

</div>

> [@Zweta\_Fuze](#):
>
> ‘’'struct Frame # frame consists of a string with 13 chars, 10 Int16, and 20 Float32=113Byte

There used to be a nice package called [StrPack.jl](https://github.com/pao/StrPack.jl) for this sort of thing, but it stopped being maintained. But @Keno put together a replacement called [StructIO.jl](https://github.com/Keno/StructIO.jl), although it doesn’t seem to have gotten a lot of attention in the last couple of years. StructIO is a registered package and its tests still pass, however, so it would be worth trying.

Also, it could give a cleaner API if you use StaticArrays.jl and StaticStrings.jl instead of tuples for the `struct` fields.

For example:

```julia
using StructIO, StaticStrings, StaticArrays

@io struct Frame
    s::StaticString{13}
    n::SVector{10, Int16}
    x::SVector{20, Float32}
end align_packed

```

which gives an on-disk size of 113 bytes:

```julia
julia> packed_sizeof(Frame)
113

```

Then you should be able to read from a file `f` with simply:

```julia
f = open("data.dat", "r")
data = [unpack(f, Frame) for i = 1:100]

```

@Keno, it would be nice if StructIO.jl could be transferred to JuliaIO and given a little [TLC](https://en.wiktionary.org/wiki/tender_loving_care), especially more documentation.

---

<div class="post-metadata">

**Author:** ![aditi\_dey29](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aditi_dey29/32/45696_2.png) [@aditi\_dey29](https://discourse.julialang.org/u/aditi_dey29)\
**Post date:** [January 8, 2023, 2:00pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/14 "2023-01-08T14:00:59Z")

</div>

Yep right! Thanks 🙂

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 8, 2023, 6:22pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/15 "2023-01-08T18:22:46Z")

</div>

Wow, that’s just amazing. Thank you very much. It works exactly as I wanted.  
Just one little more question that might be easy for you.

How is it possible to fill the structure with data in Julia and how can I write this to a file?

That’s all I would need.

Best regards Zweta

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 8, 2023, 7:06pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/17 "2023-01-08T19:06:58Z")

</div>

> [@Zweta\_Fuze](#):
>
> How is it possible to fill the structure with data in Julia and how can I write this to a file?

With my StructIO example above, you can do e.g.:

```julia
d = Frame("foo", 1:10, rand(20)) # initialize with some data
open("bar.dat", "w") do f # open file for writing as f
    pack(f, d) # write d as packed binary struct to file
end

```

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 8, 2023, 7:32pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/18 "2023-01-08T19:32:16Z")

</div>

> [@stevengj](#):
>
> `Frame("foo", 1:10, rand(20))`

Thank you very much for your quick response. This completes all my open questions. I was hoping to find these examples in the Julia documentation but I didn’t find any. S o I thank you very much.  
Best regards Zweta

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 8, 2023, 7:34pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/19 "2023-01-08T19:34:25Z")

</div>

@everybody  
The answering of the above question is successfully completed. I thank everybody for the quick response. Thank you very much.  
Best regards Zweta

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 10, 2023, 7:35pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/20 "2023-01-10T19:35:08Z")

</div>

Thanks from the help of  
aditi\_dey29, GunnarFarneback, giordano, and stevengj  
I can summarize the following answer to the question above.  
The following text can be copied in the REPL and is supposed to run  
without error messages.  
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

```julia
# Example how to write and read a DataStructure to and from a file "bar.dat"

# Packages needed
using StructIO,StaticStrings,StaticArrays

# determine structure to be read as an example of a data format
@io struct Frame 
       s::StaticString{13}
       n::SVector{10,Int16}
       x::SVector{20,Float32}
    end align_packed

# show size of that structure       
println(packed_sizeof(Frame)) # sizeof Frame to be written

# generate some data as an example -> Fw
Fw=Frame("up to max 13",1:10,rand(20))

# write the contents of the frame to the file "bar.dat"
f=open("bar.dat","w")
pack(f,Fw)
close(f)

# show the written data
println(Fw)

# show the size of the file "bar.dat"
# this is supposed to be 13*8+10*2+20*4 = 113 byte
println(run(`ls -l bar.dat`)) # there we go

# read one frame of the file "bar.dat"
f=open("bar.dat","r")
Fr=unpack(f,Frame)
close(f)

# show the contents of the frame read
println(Fr) # Frame read

# compare the contents written and read which is supposed to be true as one can see
println(Fw==Fr)

```

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [January 10, 2023, 8:20pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/21 "2023-01-10T20:20:58Z")

</div>

Final tip: Use triple quotes (```) before and after multiline code section to format it properly in Discourse. Pressing edit on the last post above, is a good place to try it.

---

<div class="post-metadata">

**Author:** ![Zweta\_Fuze](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zweta_fuze/32/29615_2.png) [@Zweta\_Fuze](https://discourse.julialang.org/u/Zweta_Fuze)\
**Post date:** [January 10, 2023, 8:33pm UTC](https://discourse.julialang.org/t/how-do-you-read-structures-from-a-file/92646/22 "2023-01-10T20:33:11Z")

</div>

Well, thanks Dan. Tried it first but it didn’t work because I didn’t use the backquotes.