I am trying to write a faster Parquet reader (faster than the one in Parquet.jl atm). One interesting problem that I encountered is this: How to efficiently unpack a set of bit-packed integers into Vector{T<:Unsigned} for some T? The data is stored in a bit-packed way explained here (see point 1). …

Do you actually need to unpack? It seems like one approach would be to define a type of 3 bit integer arrays.

is that even possible? How then do I sort it etc?

Feels like it only works if the data is a mulitple of 8 bits. The bitpacked values can be any number and not necessarily a multiple of 8.

Another problem is that the bitwidth is not a factor of 8 and may not be a multiple of 8 so one value can be stored across two bytes.

[image] xiaodai: Feels like it only works if the data is a mulitple of 8 bits. The bitpacked values can be any number and not necessarily a multiple of 8. Just break the data into chunks of the least-common multiple lcm(8, bitwidth) ÷ 8 bytes, and then use bit-shift/mask operations. For exam…

Thanks for the suggestion. I was thinking something along the same line, but the bitwidth can be as large as 32. So say my bitwidth is 17 so I can store it in a primitive type like this primitive type Foo lcm(17, 8) end and lcm(8, 17) happens to be 136. But there is no bitshift operation defined o…

[image] stevengj: and pull out 8 numbers from each 3-byte block. That would require defining a bitshift operation on 24bits or I need to Base.zext_int to a 32 bit integer before I can do the operations. But seems like I am out of luck for bitwidth where lcm(bitwidth, 8) > 128. One thing I ca…

I hardly know anything about this low level programming, but would this package be helpful? GitHub - RelationalAI-oss/Blobs.jl: Binary blobs with on-the-fly pointer patching

Thanks. I thinkit falls into the same category as previous suggestions in that it doesn’t to be able to deal with high bitwidth structures.

Efficient way to go from bit-packed value to `Vector{T<:Unsigned}`

New to Julia

rdeits May 25, 2020, 1:59am 4

What about something like this approach: Reinterpret packed data which is already in memory? - #2 by stevengj ?

Topic		Replies	Views
Simple bit manipulation question New to Julia	6	1406	March 1, 2018
Packing and unpacking an array of N-bit ints Performance	5	210	January 27, 2025
Convert integer to bits array New to Julia	12	17387	October 14, 2022
I have: Vector{UInt8}. I need: BitVector General Usage	6	3708	May 25, 2022
Convert UInt array to bitarray General Usage convert	4	1945	August 1, 2021

Efficient way to go from bit-packed value to `Vector{T<:Unsigned}`

Related topics