Reading binary data from a file

[MrBean]

Dear all,
This is my first post and I am a baby user of Julia.

I would like to read data from a file saved in binary format that comes from the sampler of an analytical instrument.

The data is of a complex type and acquired in phase quadrature, that is, each sample element consists of real and imaginary parts (so that two points are needed to correctly describe each data).

The data format can be Int or float type depending on the sensor connected.

The number of totally acquired complex points is known.

What I would like to do as a first step is to load two separate vectors one containing real data, the other with imaginary data, and possibly to print the value of some of that points.

But nothing… I am not capable of it!

[stevengj]

See, for example:

• How to read binary file in Julia? - Stack Overflow
• How do I read a binary file back into an Array?
• Read binary data of arbitrary dims and type

All of these appeared in the first page of Google results for “read binary file Julia”, and they all point to the same function: read!.

(Of course, this assumes that your “binary data” just consists of raw arrays written directly to a file, as opposed to a more structured binary format like HDF5.jl.)

[stephancb]

Make sure that you understand the format of the file. It would be unusual if the data acquisition of the analytical instrument wrote the binary data directly from the start of the file without any further information or header. If you want to skip initial extra information, then seek(io, pos) does it. Int could be 1, 2, 4, or 8 bytes, in the worst case something like 12 bits + a sign bit directly from the ADC (we have like this in satellite telemetry). Float could be 2 bytes or 4 byte single precision or 8 byte double precision, or also from some non-IEEE 754 compatible hardware.

Julia has ComplexF16, ComplexF32 ,ComplexF64, and Complex{Intxx} types. Using those is a bit easier and slightly more performand than reading into separate real and imaginary arrays (if the file has always imaginary following the real part).

Edit: If the byte order is different from little endian, that your computer most likely has, then ntoh is useful.

[mike.ingold]

In terms of file formats, “binary data” can mean basically anything. Is the data saved in a particular standard or file extension?

[MrBean]

Thank you for your reply.
Of courses I had a look around before posting here, but unfortunately there is no examples useful for my needs.

[MrBean]

Thank you Stephancb,

Let start with Int32 format, I can extend the idea to Floats.

Can you post the code for reading N complex sampled and save real and imaginary parts in two vectors?

[stevengj]

@MrBean

Of courses I had a look around before posting here, but unfortunately there is no examples useful for my needs.

It’s hard to help you if you don’t make more of an effort — why don’t the posted links help? Have you tried using the read! function? Do you understand the specific format of your file?

Please read: make it easier to help you

Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best practices when asking for help. Following these points makes it easier for us to help you and more likely you’ll get a prompt, useful answer. Keywords are highlighted to make it easier to refer to specific points. Choose a descriptive title that captures the key part of your question, eg “plots with multiple axes” instead of …

[stephancb]

Type directly into the REPL or into an editor and save as e.g. read_complexi32.jl

function read_complexi32(filename, N)
    # Preallocate arrays:
    data_r = Vector{Int32}(undef, N)
    data_i = Vector{Int32}(undef, N)

    # Read the data into arrays
    open(filename, "r") do io
        for k in 1:N
            data_r[k] = read(io, Int32)
            data_i[k] = read(io, Int32)
        end
    end
    (real=data_r, imag=data_i)
end

If in a file

include("read_complexi32.jl")

My test in the REPL:

julia> N=1024
1024

julia> data_c = rand(Complex{Int32}, N);

julia> write("test_binary_int32", data_c)
8192

julia> z=read_complexi32("test_binary_int32", N);

julia> z.real==real(data_c)
true

julia> z.imag==imag(data_c)
true

There are of course variations how this can be coded. Remarks:

1. Make the reading a function, for reuse and better performance. The test is directly in the REPL

2. The function returns the two arrays as a “named tuple” with z.real the real array and z.imag the imaginary one.

3. Writing the test complex array into the file is much shorter and without explicit loop. Reading could be similarly concise, if you were persuaded to accept reading into a complex array instead of separate real and imaginary.

[MrBean]

Great,
as soon as I can access to the workstation I will try the code.

[MrBean]

Thanks Stephancb,
the code works very well!