# Reading binary file into a Vector of custom struct
**URL:** https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097
**Category:** General Usage
**Tags:** question, binaryio
**Created:** [December 1, 2022, 2:32pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097 "2022-12-01T14:32:42Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@bensetterholm](https://discourse.julialang.org/u/bensetterholm)
#### Post date: [December 1, 2022, 2:32pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/1 "2022-12-01T14:32:42Z")
I am trying to find the best way to read a binary file directly into a vector of struct.
The binary file has a format so that it could be read in as a vector of objects with format like so (where the file may have any number of such objects appended one after another)
```julia
using StaticArrays
struct BinObj
head1::UInt32
head2::Float32
⋮
data::SMatrix{40,200,Float32,8000}
foot::UInt32
end
```
And I have been reading in as
```julia
reinterpret(BinObj, read("path/to/the/binary.file"))
```
However, since the data portion is such a large matrix, doing anything with any of the resultant SMatrix is very slow. I want to read the `data` directly into a vanilla Julia array, but how do I let the interpreter know how many bits to allocate to these arrays?
Secondly, the reinterpret call produces a reinterpret vector object, eg:
```julia
20-element reinterpret(BinObj, ::Vector{UInt8}):
```
How can I read directly into an vector of BinObj without the intermediate reinterpret object? Is this possible without doing a `copy` operation, which would allocate too much memory?
---
### Author:  [@contradict](https://discourse.julialang.org/u/contradict)
#### Post date: [December 1, 2022, 3:34pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/2 "2022-12-01T15:34:20Z")
How about this read [method](https://docs.julialang.org/en/v1/base/io-network/#Base.read)
```julia
io = open("path/to/the/binary.file")
read(io, BinObj)
```
---
### Author:  [@bensetterholm](https://discourse.julialang.org/u/bensetterholm)
#### Post date: [December 1, 2022, 3:36pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/3 "2022-12-01T15:36:45Z")
This yields
```julia
ERROR: The IO stream does not support reading objects of type BinObj
```
I presume because the binary file has multiple appended `BinObj`s and not just one?
---
### Author:  [@contradict](https://discourse.julialang.org/u/contradict)
#### Post date: [December 1, 2022, 3:43pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/4 "2022-12-01T15:43:06Z")
Oh, oops, I think those methods might only be for built-in types. I don’t know why `reinterpret` slows this down, is the problem that accessing `data` inside a `reinterpret` is much slower than accessing `data` from a directly constructed `BinObj`?
---
### Author:  [@bensetterholm](https://discourse.julialang.org/u/bensetterholm)
#### Post date: [December 1, 2022, 3:46pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/5 "2022-12-01T15:46:51Z")
That’s not really the huge issue, and I can live with reinterpret overhead if need be.
The big problem is that Static Arrays are very slow if they have more than ~100 elements, but I cannot think of any other way to make my struct as a bits type that reinterpret can read into (even though I know a priori how large my array should be). I really want the `data` element to be a vanilla Julia array.
---
### Author:  [@contradict](https://discourse.julialang.org/u/contradict)
#### Post date: [December 1, 2022, 4:10pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/6 "2022-12-01T16:10:30Z")
That makes sense. I think your best option is probably to write a function that performs multiple reads to construct a `BinObj`, then it can read `data` into an ordinary array and construct the final object from all the reads. Keep calling that function until EOF and appending to your final array of `BinObj`.
---
### Author:  [@contradict](https://discourse.julialang.org/u/contradict)
#### Post date: [December 1, 2022, 4:11pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/7 "2022-12-01T16:11:53Z")
Another option might be to read into one array, and then construct the BinObj from reinterpreted slices, but I’m not sure that won’t allocate more.
---
### Author:  [@Jordan\_Cluts](https://discourse.julialang.org/u/Jordan_Cluts)
#### Post date: [December 1, 2022, 4:12pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/8 "2022-12-01T16:12:33Z")
I have had a similar problem when trying to read a vector of Float32 values. That is obviously simpler than your compound struct but maybe the same technique would work?
I ended up using the `read!` function like so
```julia
n = div(filesize(filename),sizeof(BinObj))
read!(filename,Vector{BinObj}(undef,n))
```
This calculates from the file how many BinObj objects are in the file so it can allocate a Vector of them of the correct size. The `read!` call then stuffs the binary data into that Vector directly.
I’m not sure if this will actually work with a compound struct but worth a try.
---
### Author:  [@bensetterholm](https://discourse.julialang.org/u/bensetterholm)
#### Post date: [December 1, 2022, 4:19pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/9 "2022-12-01T16:19:21Z")
@Jordan_Cluts Thanks, this solves my secondary question about skipping the reinterpret and not allocating more than I need.
@contradict Thanks, I will give that approach a try after my morning meetings
---
### Author:  [@bensetterholm](https://discourse.julialang.org/u/bensetterholm)
#### Post date: [December 1, 2022, 5:20pm UTC](https://discourse.julialang.org/t/reading-binary-file-into-a-vector-of-custom-struct/91097/10 "2022-12-01T17:20:12Z")
@contradict and @Jordan_Cluts Thank you for your help!
This is the current way I have approached this problem. I’m sure it can be cleaned up some more, but for the time being this seems to work:
```julia
struct BinObj
head1::UInt32
head2::Float32
⋮
data::Matrix{Float32} # Has size (40, 200)
foot::UInt32
end # Has total size BINOBJSIZE
function _readIntoBinObj(iostream)
head1 = read(iostream, UInt32)
head2 = read(iostream, Float32)
⋮
data = read!(iostream, Matrix{Float32}(undef, 40, 200))
foot = read(iostream, UInt32)
return BinObj(head1, head2, ..., data, foot)
end
function getBinObjVector(filename)
n = filesize(filename) ÷ BINOBJSIZE
io = open(filename)
return [_readIntoBinObj(io) for i in 1:n]
end
```