# Fastest way to unpack bigendian Int24?

**URL:** <https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514>\
**Category:** General Usage\
**Created:** [March 5, 2019, 11:32pm UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514 "2019-03-05T23:32:54Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![joshua\_jones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshua_jones/32/7554_2.png) [@joshua\_jones](https://discourse.julialang.org/u/joshua_jones)\
**Post date:** [March 5, 2019, 11:32pm UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/1 "2019-03-05T23:32:54Z")

</div>

What’s the fastest way to unpack a vector of bigendian 24-bit signed integers stored in a binary file? Suppose the number of Int24s that I need to read is known.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 6, 2019, 1:34am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/2 "2019-03-06T01:34:33Z")

</div>

Reading a byte at time is probably pretty fast. It’s possible that reading 32-bits at a time would be faster but seems unlikely to be worth the additional complexity.

---

<div class="post-metadata">

**Author:** ![joshua\_jones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshua_jones/32/7554_2.png) [@joshua\_jones](https://discourse.julialang.org/u/joshua_jones)\
**Post date:** [March 6, 2019, 1:51am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/3 "2019-03-06T01:51:59Z")

</div>

My really clumsy solution is to define V = zeros(UInt8, 4) and loop as

```julia
for i=1:N
  V[4] = read(fid, UInt8)
  V[3] = read(fid, UInt8)
  V[2] = read(fid, UInt8)
   x[i] = reinterpret(Int32,V)[1] >> 8 
end

```

That seems to parse the data correctly but feels like a kludge.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 6, 2019, 2:16am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/4 "2019-03-06T02:16:59Z")

</div>

I would skip the `V` part:

```julia
for i = 1:N 
    x[i] = ((read(fid, UInt8) % UInt32) << 16) |
           ((read(fid, UInt8) % UInt32) << 8) | read(fid, UInt8)
end

```

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [March 6, 2019, 11:11am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/5 "2019-03-06T11:11:15Z")

</div>

I was under the impression that reading in an entire UInt8 vector at once and then byte swapping on in memory data was faster because reading has to go through filesystem checks for each call.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [March 6, 2019, 12:00pm UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/6 "2019-03-06T12:00:56Z")

</div>

When you benchmark it, do post the results.

---

<div class="post-metadata">

**Author:** ![Zach\_Christensen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zach_christensen/32/7220_2.png) [@Zach\_Christensen](https://discourse.julialang.org/u/Zach_Christensen)\
**Post date:** [March 6, 2019, 12:13pm UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/7 "2019-03-06T12:13:22Z")

</div>

I came across this behavior at the beginning of the year so I’ll try to dig up some code (I may have just been doing things wrong), but in the meantime [this issue](https://github.com/JuliaLang/julia/issues/30044) is where I got the logic about in memory being faster than calling repeated calls to `read`.

Edit: looking back it turned out what I was doing before was mostly slow for unrelated reasons. Now I use Traceur and am better informed. Sorry for the noise.

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [March 6, 2019, 9:04pm UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/8 "2019-03-06T21:04:04Z")

</div>

I think that issue refers to stdin/stdout specifically. Reading one byte at a time from a file should be fine, since it’s all buffered by the underlying C library. It does however incur a call to the C library for each character read, so there will be a little bit of overhead.

---

<div class="post-metadata">

**Author:** ![joshua\_jones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshua_jones/32/7554_2.png) [@joshua\_jones](https://discourse.julialang.org/u/joshua_jones)\
**Post date:** [March 7, 2019, 12:10am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/9 "2019-03-07T00:10:52Z")

</div>

OK, this is weird. The specification claims to use Int24s, which I’ve assumed are bigendian and signed since that’s what they do with Int16 and Int32. So Stefan’s solution or a variant with Ints should work … right?

The weird part: so far, only my solution yields the same values as their precompiled binary, which I know to be correct (e.g., I can visually inspect the seismograms).

So…is what I’m doing even consistent with a bigendian Int24? I think so but I don’t actually know that’s what they did. My solution was obtained via. guess-and-check; the format spec is a two-page PDF that can’t be Googled because the data format is officially named “win32”.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 7, 2019, 1:36am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/10 "2019-03-07T01:36:43Z")

</div>

I was converting to `UInt32` rather than `Int32`. Is that the difference? That does somewhat complicate things if they are using 24-bit two’s complement format. In that case I think this code might do the trick:

```julia
for i = 1:N
    y::UInt32 = 0
    y |= read(fid, UInt8) << 24
    y |= read(fid, UInt8) << 16
    y |= read(fid, UInt8) << 8
    x[i] = (y % Int32) >> 8
end

```

What this code is doing is reading the data into a `UInt32` in the high 24 bits (`y` is a typed local) and then reinterpreting that as an `Int32` and doing a signed shift.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 7, 2019, 2:05am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/11 "2019-03-07T02:05:05Z")

</div>

Sorry, I didn’t get that right (serves me right for not testing). This should be better:

```julia
for i = 1:1
    y = UInt32(read(fid, UInt8)) << 24
    y |= UInt32(read(fid, UInt8)) << 16
    y |= UInt32(read(fid, UInt8)) << 8
    x[i] = (y % Int32) >> 8
end

```

---

<div class="post-metadata">

**Author:** ![joshua\_jones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshua_jones/32/7554_2.png) [@joshua\_jones](https://discourse.julialang.org/u/joshua_jones)\
**Post date:** [March 7, 2019, 2:15am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/12 "2019-03-07T02:15:42Z")

</div>

> [@StefanKarpinski](#):
>
> x[i] = (y % Int32) \>\> 8

Yes! That worked! I’d need more benchmarking for a definitive answer but I think that yields a slight speed improvement.

For reference, this is the entirety of their Int format description: “…the sampled data are stored in the field #13 as the differences of sequential value (i.e. delta compression). Suitable data size of one sample in #13 is chosen and given in #10 among 0.5, 1 2, 3, and 4 octets (=bytes) to minimize data size.”

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [March 7, 2019, 3:49am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/13 "2019-03-07T03:49:23Z")

</div>

Ok, so it seems that they’re using a two’s complement signed integer representation. I kind of doubt that the bit twiddling here is going to be the difference—this seems like it would be I/O bound. It’s possible that using `mmap` might be faster.

---

<div class="post-metadata">

**Author:** ![joshua\_jones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joshua_jones/32/7554_2.png) [@joshua\_jones](https://discourse.julialang.org/u/joshua_jones)\
**Post date:** [March 7, 2019, 3:50am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/14 "2019-03-07T03:50:13Z")

</div>

Makes sense, that’s what they did with Int4…thanks again for all your help.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 7, 2019, 5:51am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/15 "2019-03-07T05:51:56Z")

</div>

I’m not sure if this helps, but if all the Int24’s are packed together, buffering on the julia side is really helpful for these kind of small reads:

```julia
# Read `N` big endian two's complement `Int24`s, buffered version
function read_int24(io, N)
    buf = read(io, 3*N)
    x = Vector{Int32}(undef, N)
    for i=1:length(x)
        @inbounds x[i] = reinterpret(Int32, (UInt32(buf[3*(i-1)+1]) << 24) |
                                            (UInt32(buf[3*(i-1)+2]) << 16) |
                                            (UInt32(buf[3*(i-1)+3]) << 8) ) >> 8
    end
    x
end

# Read `N` big endian two's complement `Int24`s, unbuffered version
function read_int24_unbuffered(io, N)
    x = Vector{Int32}(undef, N)
    for i=1:length(x)
        @inbounds x[i] = reinterpret(Int32, (UInt32(read(io, UInt8)) << 24) |
                                            (UInt32(read(io, UInt8)) << 16) |
                                            (UInt32(read(io, UInt8)) << 8) ) >> 8
    end
    x
end

# Create test data
function write_int24(io, x)
    buf = Vector{UInt8}(undef, 3*length(x))
    for i=1:length(x)
        y = reinterpret(UInt32, Int32(x[i]) << 8)
        buf[3*(i-1)+1] = (y >> 24) % UInt8
        buf[3*(i-1)+2] = (y >> 16) % UInt8
        buf[3*(i-1)+3] = (y >> 8) % UInt8
    end
    write(io, buf)
end

open("test.dat", "w") do io
    write_int24(io, -1000_000:1000_000)
end

```

The explicitly buffered version is 10x faster:

```julia
julia> io = open("test.dat", "r")
IOStream(<file test.dat>)

julia> @btime read_int24($io, 2_000_001) setup=(seek($io,0)) samples=1 evals=1;
  1.534 ms (4 allocations: 13.35 MiB)

julia> @btime read_int24_unbuffered($io, 2_000_001) setup=(seek($io,0)) samples=1 evals=1;
  15.000 ms (2 allocations: 7.63 MiB)

```

A lot of instruments have a packet-based format, where the packet header stores the size in bytes of the rest of the data packet. In this case it’s a good idea to read the whole packet into an `IOBuffer` and proceed from there. This way you recover a more reasonable performance and preserve the convenience of using the IO abstraction:

```julia
julia> @btime read_int24_unbuffered(IOBuffer(read(io, 3*2_000_001)), 2_000_001) setup=(seek($io,0)) samples=1 evals=1;
  3.953 ms (5 allocations: 13.35 MiB)

```

Edit: I should point out that benchmarks for this kind of thing will depend a lot on whether the file is in the operating system’s page cache. Whether this is true depends on your use case.

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [March 7, 2019, 7:34am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/16 "2019-03-07T07:34:12Z")

</div>

> [@c42f](#):
>
> buffering on the julia side is really helpful for these kind of small reads

Yes, that is indeed quite a big difference!

It’s probably relevant to differentiate between the buffering taking place in the underlying C library, which means that we don’t need to read one byte at a time from the file system, and the buffering that we can do within Julia, which means that we don’t need to call into the C library for each byte.

Your final statement matters here; I think the speedup we see by buffering within Julia mainly applies to _cached_ files (files already present in memory). In your example, a 6 MB file is processed in 1.5 ms, i.e. ~4 GB per second, which is a bit beyond what we can expect if reading from disk.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 8, 2019, 1:24am UTC](https://discourse.julialang.org/t/fastest-way-to-unpack-bigendian-int24/21514/17 "2019-03-08T01:24:36Z")

</div>

> [@bennedich](#):
>
> In your example, a 6 MB file is processed in 1.5 ms, i.e. ~4 GB per second, which is a bit beyond what we can expect if reading from disk.

Surprisingly it looks like some SSDs are nearly this fast. For example the [Intel 7600P](https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/professional-ssds/pro-7600p-series/pro-7600p-series-2-048tb-m-2-80mm-3d2.html) supposedly has a sequential read speed of ~3 GB/s. So doing it the slow way could be noticeable even for files which aren’t in memory.