Fast reading of multiple big-endian binary files

General Usage Performance
1

Hi all,

I have to read and preprocess a huge quantity of binary (big-endian) data from thousands of files. I managed to get everything to work properly, but my function is slow… However, as I’m a Julia beginner, I doubt my coding is optimal.

Here is the code snippet :

    function readBinaryFile(file::AbstractString)

        @info string("Reading binary file ", file)
        nInt32   = (numberOfChannels + headerSize)*samplesPerFile
        temp     = Vector{Int32}(undef, nInt32)

        open(file) do io
            read!(io, temp)
        end

        temp .= ntoh.(temp)
        temp = reshape(temp, (numberOfChannels+headerSize, samplesPerFile))
        data = convert(Array{Float64,2}, temp[headerSize+1:end,:])/2^12

        return transpose(data)

    end

Any way of doing it faster ?

Cheers,
Thomas

2

What happens to the speed if you use data = convert(Array{Float64,2}, @view temp[headerSize+1:end,:])*2^-12? That will save a copy, which should help a lot.

1 Like