# Converting UInt8 Array to BigInt, and back

**URL:** <https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110>\
**Category:** New to Julia\
**Created:** [December 30, 2018, 8:19am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110 "2018-12-30T08:19:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [December 30, 2018, 8:19am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/1 "2018-12-30T08:19:04Z")

</div>

I am writing functions that would convert Array{UInt8,1} to BigInt and vice versa.

1. From BigInt to Array I use to do as follows, through an hexadecimal string.

```julia
function int2bytes(x::Integer)
    hex = string(x, base=16)
    if mod(length(hex), 2) != 0
        hex = string("0", hex)
    end
    return hex2bytes(hex)
end

```

@gdkrmr suggested the following faster implementation which I am now using.

```julia
function int2bytes(x::BigInt)
    result = Array{UInt8}(undef, x.size * sizeof(eltype(x.d)))
    unsafe_copyto!(convert(Ptr{eltype(x.d)}, pointer(result)), x.d, x.size)
    if ENDIAN_BOM == 0x04030201
        result = result[end:-1:1]
    end
    i = findfirst(x -> x != 0x00, result)
    result[i:end]
end

```

1. From Array to BigInt, I am also going through hexadecimal string at the moment.

```julia
function bytes2int(x::Array{UInt8,1})
    hex = bytes2hex(x)
    return parse(BigInt, hex, base=16)
end

```

Based on @gdkrmr suggestion I am now trying to implement this conversion without string but can’t make it work… I came up with the below code which works when broken down in REPL but crashes when running it in a function.

```julia
function bytes2int_new(x::Array{UInt8,1})
    l = length(x)
    if Sys.WORD_SIZE == 64
        T = UInt64
        xsize = cld(l, 8)
    elseif Sys.WORD_SIZE == 32
        T = UInt32
        xsize = cld(l, 4)
    end
    x = cat(fill(UInt8(0), xsize * 8 - l), x; dims=1)
    if ENDIAN_BOM == 0x04030201
        reverse!(x)
    end
    if xsize == 1
        return reinterpret(T,x)
    else
        result = big(0)
        unsafe_copyto!(result.d, convert(Ptr{T}, pointer(x)), xsize)
        result.size = xsize
        result.alloc = xsize + 1
        return result
    end
end

```

Clearly, there is something I do not understand about pointers and BigInt implementation…

First of all, is it at all possible to create a BigInt by assigning value to its fields?  
How to properly assign those value?  
What does the alloc field represent?

I may also be on a wrong path for this function, so any suggestions to avoid string representation is welcome for this function

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [December 30, 2018, 8:41am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/2 "2018-12-30T08:41:42Z")

</div>

If `x` is a vector of `UInt8`, then the preferred way to convert it to a vector of `BigInt` should be `BigInt.(x)` (I guess this ends up calling some C-function in GMP, probably after converting from `UInt8` to `Int`.)

Is that not fast enough?

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 30, 2018, 8:43am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/3 "2018-12-30T08:43:15Z")

</div>

I think the question is about converting an array of UInt8 to a single BigInt, not to an array of BigInts.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [December 30, 2018, 8:44am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/4 "2018-12-30T08:44:19Z")

</div>

Oh. I see. Carry on.

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [December 30, 2018, 9:05am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/5 "2018-12-30T09:05:54Z")

</div>

> [@roshii](#):
>
> first of all, is it at all possible to create a BigInt by assigning value to its fields?

Yes, but proper initialization must be taken care of.  
The problem in your implementation is likely that you are writing with `unsafe_copyto!` in memory regions which have not be allocated.  
To create a `BigInt` which can represent a number with `n` bits, you can use `a = Base.GMP.MPZ.realloc2(n)`. Then you don’t need to touch the `.alloc` field of `a`, and you can write `n` bits at `a.d` (or `cld(n, 64)` `UInt64` words if the architecture is 64 bits); you also need to manually update the `.size` field of `a`.

Note also that this is internal API and will break eventually.

---

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [December 30, 2018, 10:17am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/6 "2018-12-30T10:17:14Z")

</div>

This one is even faster:

```julia
function int2bytes3(x::BigInt)
    n_bytes_with_zeros = x.size * sizeof(Sys.WORD_SIZE)
    uint8_ptr = convert(Ptr{UInt8}, x.d)

    if ENDIAN_BOM == 0x04030201
        # the minimum should be 1, else the result array will be of
        # length 0
        n_bytes_without_zeros = 1
        for i in n_bytes_with_zeros:-1:1
            if unsafe_load(uint8_ptr, i) != 0x00
                n_bytes_without_zeros = i
                break
            end
        end

        result = Array{UInt8}(undef, n_bytes_without_zeros)

        for i in 1:n_bytes_without_zeros
            @inbounds result[n_bytes_without_zeros + 1 - i] = unsafe_load(uint8_ptr, i)
        end

        return result
    else
        error("not implemented... yet")
    end

end

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 30, 2018, 1:07pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/7 "2018-12-30T13:07:20Z")

</div>

> [@rfourquet](#):
>
> […] can use `a = Base.GMP.MPZ.realloc2(n)`

If this is calling the gmp library function I think it is calling, beware with the size, as `n` is limited by the library to INT\_MAX. This caught me off guard a while back trying to compute A(4,3) of a two argument version of the Ackermann function… So much for “arbitrary size” 😕

I have plans to create a (hopefully) pure Julia arbitrary size package, but it’s not even in it’s infancy…

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [December 30, 2018, 1:55pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/8 "2018-12-30T13:55:22Z")

</div>

Thanks, it is working as a charm, 3 order of magnitude faster than the original code 🙂  
I did remove the step where it was filling Array with leading zero, proven useless and resources hungry.

Final code for reference:

```julia
function bytes2big(x::Array{UInt8,1})
    l = length(x)
    if Sys.WORD_SIZE == 64
        T = UInt64
        xsize = cld(l, 8)
    elseif Sys.WORD_SIZE == 32
        T = UInt32
        xsize = cld(l, 4)
    end
    if ENDIAN_BOM == 0x04030201
        reverse!(x)
    end
    result = Base.GMP.MPZ.realloc2(xsize * Sys.WORD_SIZE)
    result.size = xsize
    unsafe_copyto!(result.d, convert(Ptr{T}, pointer(x)), xsize)
    return result
end

```

Noted for future compatibility “issues”, I’ll keep an eye on it

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [December 30, 2018, 1:58pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/9 "2018-12-30T13:58:23Z")

</div>

> [@Sukera](#):
>
> `n` is limited by the library to INT\_MAX

Good to know!

> [@Sukera](#):
>
> I have plans to create a (hopefully) pure Julia arbitrary size package, but it’s not even in it’s infancy…

That would come handy, especially if the GMP API beaks at some point 🙂

---

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [December 30, 2018, 2:08pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/10 "2018-12-30T14:08:33Z")

</div>

Are there in-place operations in `libgmp` and are they exposed to julia somewhere?

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [December 30, 2018, 2:23pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/11 "2018-12-30T14:23:12Z")

</div>

Yes, in the `Base.GMP.MPZ` module. Not all functions are exposed, you have to lookup that source file to know which ones are. In-place operations end with the `!` suffix as is usual in Julia.

@roshii Also, I didn’t think about that earlier, but you probably should use `Base.GMP.BITS_PER_LIMB` and `Base.GMP.Limb` to refer to the architecture-dependant number of bits per “limb” and “limb” type, it may not correspond to `Sys.WORD_SIZE` on some architectures (Windows?).

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [December 30, 2018, 5:11pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/12 "2018-12-30T17:11:04Z")

</div>

Lovely, thank again 🙂  
I assume the case for small endians would look as follows.

```julia
...
    else
        n_bytes_without_zeros = 1
        for i in 1:n_bytes_with_zeros
            if unsafe_load(uint8_ptr, i) != 0x00
                n_bytes_without_zeros = i
                break
            end
        end

        result = Array{UInt8}(undef, n_bytes_without_zeros)

        for i in 1:n_bytes_without_zeros
            @inbounds result[i] = unsafe_load(uint8_ptr, i)
        end
    end
    return result
end

```

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [December 30, 2018, 5:14pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/13 "2018-12-30T17:14:30Z")

</div>

Windows…  
Point taken, code looks even cleaner with it.

```julia
function bytes2big(x::Array{UInt8,1})
    xsize = cld(length(x), Base.GMP.BITS_PER_LIMB / 8)
    if ENDIAN_BOM == 0x04030201
        reverse!(x)
    end
    result = Base.GMP.MPZ.realloc2(xsize * Base.GMP.BITS_PER_LIMB)
    result.size = xsize
    unsafe_copyto!(result.d, convert(Ptr{Base.GMP.Limb}, pointer(x)), xsize)
    return result
end

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 30, 2018, 6:03pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/14 "2018-12-30T18:03:42Z")

</div>

If you decide to go with the `realloc2` approach, check the function in `Base` for error checking and thrown errors - libgmp will just signal abort if the new allocated size is too big, killing the calling process, so make sure that doesn’t happen! I don’t know off the top of my head if that particular function checks for problems before calling into libgmp.

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [January 2, 2019, 9:57am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/15 "2019-01-02T09:57:03Z")

</div>

I indeed notice some strange behavior:

```julia
julia> x = [rand(UInt8)]
1-element Array{UInt8,1}:
 0x7d

julia> for i in 1:10
       println(bytes2big(x))
       end
125
125
125
125
125
125
125
125
125
125

julia> for i in 1:10
       println(bytes2big(int2bytes(125)))
       end
139767316545661
139767292756093
18446744073709486205
139767289413757
139767316545661
139767312023677
139766825746557
125
139767316545661
139767292756093

```

I assume I’d need to free pointer reference at some point in time but I’m just struggling here… Would anyone have a clue on what is causing result to vary like that?  
How can I ensure a stable result?

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [January 2, 2019, 10:50am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/16 "2019-01-02T10:50:44Z")

</div>

Experiencing same randomness with the following:

```julia
julia> for i in 1:10
              println(bytes2big([0x02]))
              end
139654643580930
18446744073709486082
139654643580930
2
18446744073709486082
139654697844738
2
18446744073709486082
139654697844738
139654643580930

```

---

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [January 2, 2019, 1:28pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/17 "2019-01-02T13:28:30Z")

</div>

> [@Sukera](#):
>
> I have plans to create a (hopefully) pure Julia arbitrary size package, but it’s not even in it’s infancy…

Some time ago I started something like this, but it turned out to require more work than I was willing to put into it:

> **[GitHub - gdkrmr/UArrayInts.jl: use arrays of unsigned as unsigned](https://github.com/gdkrmr/UArrayInts.jl)**
>
> use arrays of unsigned as unsigned. Contribute to gdkrmr/UArrayInts.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Krastanov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krastanov/32/6817_2.png) [@Krastanov](https://discourse.julialang.org/u/Krastanov)\
**Post date:** [July 10, 2019, 8:25pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/18 "2019-07-10T20:25:30Z")

</div>

Were you able to fix the randomness issue? I observe the same, and I do not see any error messages that might explain what is happening.

---

<div class="post-metadata">

**Author:** ![Krastanov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krastanov/32/6817_2.png) [@Krastanov](https://discourse.julialang.org/u/Krastanov)\
**Post date:** [July 11, 2019, 2:14am UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/19 "2019-07-11T02:14:23Z")

</div>

I got something similar to the requirement discussed here to work (using `UInt64` instead of `UInt8`):

```julia
function ubig2ints(x::BigInt)
    if Base.GMP.Limb!=UInt64 || Sys.WORD_SIZE!=64 || ENDIAN_BOM!=0x04030201 || x.size<=0
        error("not implemented")
    end
    copy(unsafe_wrap(Array, x.d, x.size))
end

function ints2ubig(x::AbstractVector{UInt64})
    if Base.GMP.Limb!=UInt64 || Sys.WORD_SIZE!=64 || ENDIAN_BOM!=0x04030201
        error("not implemented")
    end
    r = Base.GMP.MPZ.realloc2(length(x)*64)
    r.size = length(x)
    unsafe_copyto!(r.d, pointer(x), length(x))
    r
end

```

---

<div class="post-metadata">

**Author:** ![roshii](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roshii/32/6211_2.png) [@roshii](https://discourse.julialang.org/u/roshii)\
**Post date:** [July 12, 2019, 12:17pm UTC](https://discourse.julialang.org/t/converting-uint8-array-to-bigint-and-back/19110/20 "2019-07-12T12:17:56Z")

</div>

We came up with an alternative implementation that doesn’t show this randomness issue.

```julia
function to_int(x::Vector{UInt8})
    if isempty(x)
        return 0
    end
    length(x) > div(Sys.WORD_SIZE, 8) ? T = BigInt : T = Int
    result = zero(T)
    if ENDIAN_BOM == 0x01020304
        reverse!(x)
    end
    for c in x
        result <<= 8
        result += c
    end
    return result
end

```

However the above appeared to be less performant (for 32 bytes messages) that the below, which I retained and implemented in [BitConverter.jl](https://gitlab.com/braneproject/BitConverter.jl)

```julia
function to_big(x::Vector{UInt8})
    hex = bytes2hex(x)
    return parse(BigInt, hex, base=16)
end

```