# Bits and bytes manipulation library
**URL:** https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843
**Category:** General Usage
**Tags:** binaryio
**Created:** [August 31, 2024, 2:02pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843 "2024-08-31T14:02:53Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@sivakon](https://discourse.julialang.org/u/sivakon)
#### Post date: [August 31, 2024, 2:02pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/1 "2024-08-31T14:02:53Z")
Are there any available packages in Julia ecosystem for byte level data manipulation?
Similar packages are `scodec` from Scala - [GitHub - scodec/scodec: Scala combinator library for working with binary data](https://github.com/scodec/scodec)
Elixir has [Binaries, strings, and charlists — Elixir v1.17.2](https://hexdocs.pm/elixir/binaries-strings-and-charlists.html#bitstrings)
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [August 31, 2024, 9:02pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/2 "2024-08-31T21:02:17Z")
Just in case, would [this nice post](https://www.analytech-solutions.com/analytech-solutions/blog/binary-io.html) about using CBinding.jl be helpful?
---
### Author:  [@jar1](https://discourse.julialang.org/u/jar1)
#### Post date: [August 31, 2024, 9:16pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/3 "2024-08-31T21:16:16Z")
Can you give some example operations you’re looking for?
---
### Author:  [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)
#### Post date: [August 31, 2024, 9:40pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/4 "2024-08-31T21:40:03Z")
take a look at
> **[GitHub - oschulz/BitOperations.jl: Julia Bit Manipulation Functions](https://github.com/oschulz/BitOperations.jl)**
>
> Julia Bit Manipulation Functions
> **[GitHub - JeffreySarnoff/BitsFields.jl: Bitfields are made much easier to use](https://github.com/JeffreySarnoff/BitsFields.jl)**
>
> Bitfields are made much easier to use
---
### Author:  [@sivakon](https://discourse.julialang.org/u/sivakon)
#### Post date: [September 1, 2024, 5:32am UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/5 "2024-09-01T05:32:42Z")
This is helpful but depending on Clang, is big and unnecessary, I understand why they did it. I need a pure julia low level byte and bit level data manipulation without a lot of dependencies. I might write one if it’s not available 🙂
---
### Author:  [@sivakon](https://discourse.julialang.org/u/sivakon)
#### Post date: [September 1, 2024, 5:35am UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/6 "2024-09-01T05:35:58Z")
I basically want this library from [GitHub - scodec/scodec: Scala combinator library for working with binary data](https://github.com/scodec/scodec) in julia. It basically has functions for primitive types and combinators to compose them in arbitrary way to create a byte level parser. I use that basically to write network protocols, parse data formats like the post that @rafael.guerra mentioned.
Clojure also has this [Introduction · clj-commons/gloss Wiki · GitHub](https://github.com/clj-commons/gloss/wiki/Introduction)
---
### Author:  [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [September 1, 2024, 4:38pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/7 "2024-09-01T16:38:19Z")
Are you perhaps thinking of something like the [Seriliazation stdlib](https://docs.julialang.org/en/v1/stdlib/Serialization/) or something like serde in Rust? As far as I know, there isn’t really much for that kind of work at the moment, the package ecosystem is still more geared towards numerical work than general software engineering stuff.
I guess there’s also some parser combinator libraries, if that’s what you’re looking for. Parsers.jl comes to mind.
---
### Author:  [@sivakon](https://discourse.julialang.org/u/sivakon)
#### Post date: [September 1, 2024, 4:46pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/8 "2024-09-01T16:46:08Z")
serde has a lot of functionality, that’s a lofty goal for now. For example, I can say `UInt32 :: UInt32 :: UInt32 ` that should create a parser that can serialize and deserialize bytes 3 Uint32’s little endian for example.
On this note, is there Java’s `ByteBuffer` like abstraction in Julia?
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [September 2, 2024, 10:31am UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/9 "2024-09-02T10:31:24Z")
From just looking at scodec, it looks like you wouldn’t need a library for most functionality…
Julia’s immutable structs (with no mutable structs in them, or `isbitstype(T)`) are already doing mostly what scodec codecs are doing:
```julia
firscodec = Tuple{UInt8,UInt8,UInt16}
bytes = hex2bytes("102a03ff")
# ntoh to swap to endian of the current system
result = (ntoh.(reinterpret(firscodec, bytes)[1]))
Int(sum(result))
struct Point
x::Int
y::Int
z::Int
end
# Many ways to convert the result to a Point struct
# Might want some utility, or simply a good coverage of `convert(MyType, x)`
point = Point(result...)
# I guess one could have this convenience:
interpret_as(bytes, ::Type{T}, ::Type{Codec}) where {T, Codec} = convert.(T, reinterpret(Codec, bytes))
io = IOBuffer() # I guess similary to Java ByteBuffer
write(io, Ref(point))
seekstart(io)
point2 = Ref{Point}()
point2 = read!(io, point2)
@test point2[] == point
io = IOBuffer()
write(io, bswap(0x102a03ff))
bytes = take!(io)
ntoh.(reinterpret(firscodec, bytes)[1]) == result
```
You can likely make all of this a bit more elegant here and there, but those operations should work pretty well and should be close to C performance in julia.
---
### Author:  [@sivakon](https://discourse.julialang.org/u/sivakon)
#### Post date: [September 2, 2024, 11:22am UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/10 "2024-09-02T11:22:03Z")
thanks this is very helpful. I am just writing simple reusable components that you can combine to make a byte level parser.
My idea is create codec for every type (using macros and generated), and create a DSL that can compile a parser to a tuple. (I am still thinking about using heterogenous tuples vs structs)
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [September 2, 2024, 12:37pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/11 "2024-09-02T12:37:09Z")
> [@sivakon](#):
>
> using macros and generated
Just a tip from me after developing such code for quite a few years in Julia: I’d keep at as simple and function based as possible before doing anything more complicated.
I’ve regretted any macro and generated function, that I was able to avoid after understanding the actual problem I want to solve better 😉
---
### Author:  [@sivakon](https://discourse.julialang.org/u/sivakon)
#### Post date: [September 2, 2024, 1:05pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/12 "2024-09-02T13:05:54Z")
thanks for the advice, what’s the best way to emulate HLists (heterogenous lists), tuples are hard to metaprogram.
I am using tuples here, we can also use `Vector{Codec}`
This is what I got so far
```julia
abstract type Codec{T} end
struct IntCodec{T<:Integer} <: Codec{T}
data::T
end
struct FloatCodec{T<:AbstractFloat} <: Codec{T}
data::T
end
struct TupleCodec{T<:Tuple{Vararg{Codec}}} <: Codec{T}
data::T
end
# convert
convert(::Type{T}, codec::IntCodec{T}) where {T<:Integer} = codec.data
convert(::Type{T}, codec::FloatCodec{T}) where {T<:AbstractFloat} = codec.data
convert(::Type{T}, codec::TupleCodec{T}) where {T<:Tuple{Vararg{Codec}}} = codec.data
function Base.read(io::IO, ::Type{T}) where {T<:Tuple{Vararg{Codec}}}
codec_types = T.parameters
elements = map(codec_types) do C
read(io, C)
end
return TupleCodec(Tuple(elements))
end
function Base.read(io::IO, ::Type{IntCodec{T}}) where {T<:Integer}
value = read(io, T)
return IntCodec{T}(value)
end
# create a codec for every type
@generated function decode(io::IO, ::Type{C}) where {C<:Codec}
T = C.parameters[1]
codec_type = C.name.wrapper
if T <: Integer
type_name = "integers"
elseif T <: AbstractFloat
type_name = "floats"
elseif T <: Tuple{Vararg{Codec}}
type_name = "tuple"
else
error("Unsupported type for decode: $T")
end
quote
if eof(io) || (io.size - io.ptr + 1) < sizeof($T)
throw(ArgumentError("Not enough bytes for type $(sizeof($T))-byte $($type_name)"))
end
res = read(io, $T)
return res
end
end
```
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [September 2, 2024, 1:39pm UTC](https://discourse.julialang.org/t/bits-and-bytes-manipulation-library/118843/13 "2024-09-02T13:39:33Z")
There is also StructIO.jl for packed binary I/O of heterogeneous data structures.