Reading byte-aligned struct from binary file

New to Julia
1

I need to read data from a binary file created by a C program using a byte-aligned struct. Below is a reduced version of the C program and my best effort at reading the binary file in Julia.

#include <stdio.h>
typedef struct {
  char fill;
  char sex;
  short days;
  char breed[2];
} PedInfo;
void main()
{
  PedInfo p;
  int i;
  FILE *f;
  f = fopen("test.bin", "w");
  for (i=1; i<=3; i++) {
    p.fill = ' ';
    p.sex = "MF"[i%2];
    p.days = i * 1111;
    p.breed[0] = "HJA"[i%3];
    p.breed[1] = "OEY"[i%3];
    fwrite(&p.sex, 1, 5, f); // misalignment created here!
  }
  fclose(f);
}

Here’s my ugly Julia program

using Printf
buf = zeros(UInt8, 5)
buf2 = zeros(UInt8, 2)
open("test.bin", "r") do f
    for i in 1:3
        read!(f, buf)
        sex = Char(buf[1])
        buf2[1:2] = buf[2:3]
        days = reinterpret(Int16, buf2)[1]
        breed = String(buf[4:5])
        @printf("%d %c %hd %s\n", i, sex, days, breed)
    end
end

with correct output

1 F 1111 JE
2 M 2222 AY
3 F 3333 HO

Is there a better/nicer way to read this binary file in Julia?

2

You could to something like

julia> struct Foo
         fill::Cchar
         sex::Cchar
         days::Int16
         breed::Tuple{Cchar, Cchar}
       end

julia> Base.show(io::IO, f::Foo) = print(io, Char(f.sex), ' ', f.days, ' ', Char(f.breed[1]), Char(f.breed[2]))

julia> buf = zeros(UInt8, 6);

julia> open("test.bin", "r") do io
         for i in 1:3
           read!(io, @view(buf[2:6]))
           println(i, ": ", reinterpret(Foo, buf)[1])
         end
       end
1: F 1111 JE
2: M 2222 AY
3: F 3333 HO

or even

julia> function readfile(file)
         open(file, "r") do io
           out = zeros(UInt8, 18) 
           for i in 0:2
             read!(io, @view(out[6i .+ (2:6)]))
           end
           reinterpret(Foo, out)
         end
       end
readfile (generic function with 1 method)

julia> readfile("test.bin")
3-element reinterpret(Foo, ::Vector{UInt8}):
 F 1111 JE
 M 2222 AY
 F 3333 HO
6 Likes
3

Thanks much! The first part of your answer contains all the elements I need (Struct with Cchar and
Tuple{Cchar, Cchar} and the @view) to move forward. In my case the file has 85 million records and many more fields in the struct, so I don’t want to slurp it all into memory as in your second example. However, that approach could be useful in other cases.

2 Likes