Need a function similar to python struct.unpack

General Usage
1

Hello everybody.
I need to read binary file from a specific module (HydraHarp) which comes with python and matlab codes for decoding the file.

The python code uses extensively the struct.unapck functions. For example:

tyEmpty8 = struct.unpack(“>i”, bytes.fromhex(“FFFF0008”))[0]

For reference, this will yield the value: -65528

This is equivalent to the MatLab code:

tyEmpty8 = hex2dec(‘FFFF0008’);

I need a similar conversion in Julia (I am using 1.9.1 in a MacOs with ARM processor) and I have been unable to reproduce the value using several approaches (including PyCall). Any help would be very welcoed!

2

I’m confused. Is the hex in a string or is it binary in a file. Here’s the literal function starting from a string.

julia> fromhex(hex) = first(reinterpret(Int32, [parse(UInt32, "0x"*hex)]))
fromhex (generic function with 1 method)

julia> fromhex("FFFF0008")
-65528

I got a feeling that this is not what you want.

You probably want something that is more like this.

julia> write("test.bin", 0xFFFF0008)
4

julia> read("test.bin", Int32)
-65528

In the above case, the file “test.bin” contains 4 bytes in little endian order.

julia> read("test.bin")
4-element Vector{UInt8}:
 0x08
 0x00
 0xff
 0xff

The above bytes are in little endian order. 08 comes first, and FF comes last. It seems like you need this to be read in big endian order.

julia> write("test.bin", hton(0xFFFF0008))
4

julia> read("test.bin")
4-element Vector{UInt8}:
 0xff
 0xff
 0x00
 0x08

julia> ntoh(read("test.bin", Int32))
-65528

Now 0xFF comes first, and 0x08 comes last.

2 Likes
3 
reinterpret(Int32, hex2bytes("FFFF0008")) .|> hton

Output this:

1-element Vector{Int32}:
 -65528

This only works for unpacking single datatype.

4

Thanks a lot for your explanations! Actually I had both type of problems, some data are “hard wire” in the code (e.g, tyEmpty8 = struct.unpack(“>i”, bytes.fromhex(“FFFF0008”))[0]) and some data are read from the input file (e.g, tagInt = struct.unpack(“<q”, inputfile.read(8))[0]), I think your explanations cover both cases. Again, thanks a lot

5

Another relevant post on SO. Looks very similar to @mkitti’s answer.

Also this post on Julia Bloggers.

6

Using GitHub - analytech-solutions/CBinding.jl: Automatic C interfacing for Julia should be very easy for such things as you can define the structure in C code directly.

7

This seems a bit redundant, why not

fromhex(hex) = reinterpret(Int32, parse(UInt32, "0x"*hex))

?

8

You can avoid the string creation by manually specifying the base

fromhex(hex) = reinterpret(Int32, parse(UInt32, hex; base=16))
2 Likes
9

There’s also

fromhex(hex) = parse(UInt32, hex; base=16) % Int32

though I’m not 100% certain it’s exactly the same (on my phone here).

1 Like
10 
julia> hex2bytes("FFFF0008")
4-element Vector{UInt8}:
 0xff
 0xff
 0x00
 0x08

julia> hex2bytes("FFFF008") # <- valid number
ERROR: ArgumentError: length of iterable must be even

What would be the advantage of doing this in C?

11

I got curious enough about this that I started to implement the Python struct library in Julia.

(@v1.9) pkg> activate --temp
  Activating new project at `/tmp/jl_DFEY1E`

(jl_DFEY1E) pkg> add https://github.com/mkitti/PythonStructs.jl
    Updating git-repo `https://github.com/mkitti/PythonStructs.jl`
   Resolving package versions...
    Updating `/tmp/jl_DFEY1E/Project.toml`
  [ac2a11db] + PythonStructs v1.0.0-DEV `https://github.com/mkitti/PythonStructs.jl#main`
    Updating `/tmp/jl_DFEY1E/Manifest.toml`
  [ac2a11db] + PythonStructs v1.0.0-DEV `https://github.com/mkitti/PythonStructs.jl#main`

julia> using PythonStructs

julia> unpack(">i", hex2bytes("FFFF0008"))[begin]
-65528

julia> unpack("<i", hex2bytes("FFFF0008"))[begin]
134283263

julia> unpack(">I", hex2bytes("FFFF0008"))[begin]
0xffff0008

julia> hex2bytes("FFFF0008") |> unpack(">l") |> first
-65528

Also see the initial documentation.

3 Likes
12

Thanks a lot, this is really useful. The python library provides useful tools for binary data manipulation that are not obvious in Julia. I will go ahead and use your functions in my application

13

I have implemented the functions of your package in my code and they work just right. Thanks a lot.

juan jose gomez cadenas
jjgomezcadenas@gmail.com

1 Like