I need to use a number that has 2^12 discrete levels, i.e. a UInt12. Is there a package that does this (I tried BitIntegers but no joy)?
Can it be 2^12 discrete levels between [0, 1] instead? Then you can use GitHub - JuliaMath/FixedPointNumbers.jl: fixed point types for julia Else I would look through @JeffreySarnoff 's packages 
3 Likes
Julia’s primitive types are multiples of 8bits, so there is no native 12 bit UInt available.
@mschauer suggests FixedPointNumbers.jl and that is a strong, well-used package.
If you must have a UInt12 work-alike, it would be implemented using UInt16s, with the necessary testing and masking. What operations would you require? What is the purpose?
1 Like
Do you want a single UInt12 or do you want packed array storage for a whole slew of them? The latter is possible with a custom array type, but likely to be significantly slower than just using UInt16s even with the 25% space savings.
2 Likes
I see. This is just for the sake of messages to an external servo motor, so not worth too much trouble. I though though that since there are cameras that encode pixels in UInt12 (I think?) there would be something established already. But that’s fine, thanks for the awesome info!
In that case, it’s FixedPointNumbers that you want. Mapping to the interval [0, 1] has the advantage that each color will correspond to the same value(s), regardless of the number of bits used to store it. (And the same for servomotor angles, of course.)
3 Likes
@mkitti is working on a UInt12Array with functionality similar to BitArrays
2 Likes
Yes, @bjarthur is correct. I have been working on a package to do this. I’m working with a 12-bit camera. Essentially you have a UInt12Array that can be backed by a UInt24 Vector (via BitIntegers) or an UInt16 Vector. I’ve also developed SIMD based code to unpack 12-bit integer data into 16-bit integers.
Would that be helpful for you @yakir12 ?
2 Likes
That sounds cool. The need and use of 12-bits in cameras is what I thought would fuel this, so that makes total sense. I’ll gladly try and bench things for my needs once you’re done.
Thanks a lot!
Do you need to deal with arrays of UInt12s or an individual UInt12? I did some work on both cases, but mostly focused on arrays.
The main thing that was holding me back from pushing it to Github was feature creep. I think the package as-is might do too many things, so I was planning to break it up.
1 Like
The messages I would deal with are vectors and/or arrays.
Here’s the real solution, an actual package:
https://github.com/JaneliaSciComp/UInt12Arrays.jl
Let’s talk about the underlying data first and the packing. I’ve seen many ways to pack 12-bit integers together, but the most natural and common way is to pack them consecutively. Let’s say we have six bytes of data or 48 bits. This represents four 12-bit integers. In the six bytes below, I’ve numbered the the first six nibbles (4-bits each) using 1 through 6 and the last six nibbles using a through f. If we reinterpret them as 24-bit integers via BitIntegers.jl we can see the order holds.
julia> data = UInt8[0x21, 0x43, 0x65, 0xba, 0xdc, 0xfe]
6-element Array{UInt8,1}:
0x21
0x43
0x65
0xba
0xdc
0xfe
julia> using UInt12Arrays
julia> A24 = reinterpret(UInt24,data)
2-element reinterpret(UInt24, ::Array{UInt8,1}):
0x654321
0xfedcba
We want 12-bit integers though and not 24-bit integers. It is currently not possible to make non-byte sized bitstypes in Julia, so we cannot just use reinterpret as we did for 24-bit integers.
julia> using BitIntegers
julia> BitIntegers.@define_integers 12
ERROR: invalid number of bits in primitive type Int12
Stacktrace:
[1] top-level scope
@ ~\.julia\packages\BitIntegers\fcpdN\src\BitIntegers.jl:60
julia> reinterpret(UInt12,data)
ERROR: ArgumentError: cannot reinterpret `UInt8` `UInt12`, type `UInt12` is not a bits type
Stacktrace:
[1] (::Base.var"#throwbits#220")(::Type{UInt8}, ::Type{UInt12}, ::Type{UInt12}) at .\reinterpretarray.jl:16
[2] reinterpret(::Type{UInt12}, ::Array{UInt8,1}) at .\reinterpretarray.jl:33
[3] top-level scope at REPL[155]:1
What we essentially want to do is just break the 24-bit integer into 12-bit integer halfs:
julia> A24[1]
0x654321
julia> first(A24[1])
0x0321
julia> last(A24[1])
0x0654
julia> first(A24[2])
0x0cba
julia> last(A24[2])
0x0fed
This is what UInt12Array and UInt12Vector do. The default form uses UInt16 as an element type. However, you can also UInt12 as an element type.
julia> A16 = UInt12Vector(data)
4-element UInt12Array{UInt16,Array{UInt8,1},1}:
0x0321
0x0654
0x0cba
0x0fed
julia> A12 = UInt12Vector{UInt12}(data)
4-element UInt12Array{UInt12,Array{UInt8,1},1}:
0x321
0x654
0xcba
0xfed
UInt12 is actually just a boxed UInt16 in the current implementation, so you might as well just use UInt16 directly in most cases. The main advantage of using UInt12 is handling overflow correctly as well as display:
julia> A16[1] + 0xd00
0x1021
julia> A12[1] + 0xd00
0x021
However, even with 16-bit element types, assignment into the UInt12Array will properly discard the highest nibble:
julia> A16[1]
0x0321
julia> A16[1] += 0xd00; A16[1]
0x0021
Note that UInt12Array is basically just a 12-bit unsigned integer view of the original bytes. By changing the first 12-bit integer, we removed the 3 nibble from all the other views of the same underlying data:
julia> A24
2-element reinterpret(UInt24, ::Array{UInt8,1}):
0x654021
0xfedcba
julia> A12
4-element UInt12Array{UInt12,Array{UInt8,1},1}:
0x021
0x654
0xcba
0xfed
julia> A16
4-element UInt12Array{UInt16,Array{UInt8,1},1}:
0x0021
0x0654
0x0cba
0x0fed
julia> A16[1]
0x0021
julia> data[2] = 0x43
0x43
julia> A12
4-element UInt12Array{UInt12,Array{UInt8,1},1}:
0x321
0x654
0xcba
0xfed
Because UInt12Array is a view on the underlying data it is pretty inexpensive to create, but it may take a while to convert the entire array to a true UInt16 array that may be faster to work with for other applications:
julia> data = rand(UInt8, 1024*1024*1024*2+1)
2147483649-element Array{UInt8,1}:
0x84
0x7d
0xde
0x63
0x99
0x39
0xcb
⋮
0x8a
0x42
0x88
0x30
0x88
0x57
julia> @time A16 = UInt12Vector(data)
0.000013 seconds (6 allocations: 336 bytes)
1431655766-element UInt12Array{UInt16,Array{UInt8,1},1}:
0x0d84
0x0de7
0x0963
0x0399
0x01cb
0x0c2d
0x0795
⋮
0x0d52
0x0605
0x028a
0x0884
0x0830
0x0578
julia> @time copy(A16)
5.329485 seconds (2 allocations: 2.667 GiB, 3.17% gc time)
1431655766-element Array{UInt16,1}:
0x0d84
0x0de7
0x0963
0x0399
0x01cb
0x0c2d
0x0795
0x0e67
0x0dd7
0x05f4
0x0f4b
0x006b
0x0386
0x09f1
0x07bc
0x0b50
⋮
0x0538
0x02a2
0x0d40
0x092e
0x00ad
0x0412
0x0dbf
0x0496
0x079b
0x0d52
0x0605
0x028a
0x0884
0x0830
0x0578
For converting the entire array to a native UInt16 array, I overrode Base.convert and used SIMD.jl to accelerate conversion:
julia> @time convert(Array{UInt16}, A16)
1.029260 seconds (6 allocations: 2.667 GiB, 8.86% gc time)
1431655766-element Array{UInt16,1}:
0x0d84
0x0de7
0x0963
0x0399
Let me know if this works for you. One possible complication that I can see is if your 12-bit integers are packed differently.
6 Likes