Write struct into binary file and read it back

New to Julia
1

I try to write a struct into a binary file and read it back.
My code is the following:

struct STRUCT
    x::Float64
    y::Float64
    z::Float64
end

isbitstype(STRUCT)

rp = STRUCT(2.0, 3.0, 4.0)

open("test.bin", "w") do io
    write(io, rp)
end

open("test.bin", "r") do io
    rp2 = read(io, STRUCT)
    @show rp2
end

I got the following error:

ERROR: MethodError: no method matching write(::IOStream, ::STRUCT)
This error has been manually thrown, explicitly, so the method may exist but be intentionally marked as unimplemented.

Closest candidates are:
  write(::IO, ::Any)
   @ Base io.jl:792
  write(::IO, ::Any, Any...)
   @ Base io.jl:793
  write(::IO, ::Union{Base.AnnotatedString, SubString{<:Base.AnnotatedString}})
   @ StyledStrings C:\Users\kiki\.julia\juliaup\julia-1.11.5+0.x64.w64.mingw32\share\julia\stdlib\v1.11\StyledStrings\src\io.jl:258
  ...

Stacktrace:
 [1] write(io::IOStream, x::STRUCT)
   @ Base .\io.jl:792
 [2] (::var"#15#16")(io::IOStream)
   @ Main w:\a\2d distribution\struct RAYPATH.jl:13
 [3] open(::var"#15#16", ::String, ::Vararg{String}; kwargs::@Kwargs{})
   @ Base .\io.jl:410
 [4] open(::Function, ::String, ::String)
   @ Base .\io.jl:407
 [5] top-level scope
   @ w:\a\2d distribution\struct RAYPATH.jl:12

What is the problem?
isbitstype(STRUCT) returns true.

2

You want to write an instance of type STRUCT to a file, but you have to implement the write method in order to do that, otherwise Julia cannot simply assume how a generic type should be saved.
The fact that isbitstype returns true means something different, not sure why you would expect that to be linked

3

Well, you can use reinterpret.

struct STRUCT
    x::Float64
    y::Float64
    z::Float64
end

isbitstype(STRUCT)

rp = STRUCT(2.0, 3.0, 4.0)

open("test.bin", "w") do io
    write(io, reinterpret(Int8, [rp, ]))
end

open("test.bin", "r") do io
    rp2 = reinterpret(STRUCT, read(io))
    @show rp2
end
1 Like
4

Or perhaps rp2 = only(reinterpret(STRUCT, read(io))) if you want to get a STRUCT out (i.e. rp instead of [rp]).

Edit:

Pedro’s approach would naturally work for any isbitstype, so there could have been a default implementation for write in this case.

1 Like
5

I am not sure if you are trying this for educational purposes, but there are many ways to serialize objects, for example JSON format is an option.

Julia has the Serialization package in its standard library. How about doing this:

julia> using Serialization

julia> struct STRUCT
           x::Float64
           y::Float64
           z::Float64
       end

julia> s = STRUCT(1.0, 2.0, 3.0)
STRUCT(1.0, 2.0, 3.0)

julia> serialize("output.dat", s)

and then

julia> object = deserialize("output.dat")
STRUCT(1.0, 2.0, 3.0)
5 Likes
6

If you use the Serialization package be aware that the file format may not be compatible across different machines or OS’es. From the docs:

In some cases, the word size (32- or 64-bit) of the reading and writing machines must match. In rarer cases the OS or architecture must also match, for example when using packages that contain platform-dependent code."

7

This can be an example, you can see that the file has a size of 24Bytes

julia> struct MyStruct
           x::Float64
           y::Float64
           z::Float64
       end

julia> function Base.write(io::IOStream, ms::MyStruct)
           write(io, ms.x)
           write(io, ms.y)
           write(io, ms.z)
       end

julia> function Base.read(io::IOStream, ::Type{MyStruct})
           x = read(io, Float64)
           y = read(io, Float64)
           z = read(io, Float64)
           return MyStruct(x, y, z)
       end
julia> open("test.bin", "w") do io
           write(io, MyStruct(2.0, 3.0, 4.0))
       end
8

shell> stat test.bin
  File: test.bin
  Size: 24    

julia> open("test.bin", "r") do io
           read(io, MyStruct)
       end
MyStruct(2.0, 3.0, 4.0)

Not sure I’d like that

It seems to give an overhead, don’t know why


julia> serialize("output.dat", MyStruct(1.0, 2.0, 3.0))

shell> stat output.dat
  File: output.dat
  Size: 56
8

Just because the serialized file carries a set of header information:

9

The header should be 8 bytes, but the overhead is 32 Bytes

10

It is not documented in a great detail but the file also includes the module name (in my case it’s Main) and the name of the structure.

Edit: I think the module name and the struct name have also headers (because of strings).

1 Like
11

I would use GitHub - JuliaIO/StructIO.jl: binary I/O methods generated from Julia structure definitions if you want to write the “raw bits” of the structure to disk (as opposed to some higher-level package like JLD.jl or Serialization). For example, this gives you control of field padding.

1 Like
12

My goal is to read a binary file generated by Zemax (optical simulation software). But because that did not worked, I tried to make a minimal example.
Having C/C++ background, it is strange, that it is so complicated to write/read N structs.

But with reinterpret or by overloading read/write I was able to overcome the problem.

1 Like
13

That’s exactly the use case for StructIO.jl

You need to specify the struct alignment/padding for an on-disk format, if you don’t want it to be compiler/ABI-dependent. That’s what StructIO.jl does.

(And then there is the question of endian conversion, though most things are little-endian these days. But Julia has functions like ntoh to do endian conversion.)

14

You may want to use CBinding.jl if you already have the c struct headers.