# Why did I get IO error when broadcasting higher precision (bits \>= 98) ArbFloat variables to workers?

**URL:** <https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025>\
**Category:** General Usage\
**Tags:** question, parallel, distributed\
**Created:** [December 20, 2024, 12:42pm UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025 "2024-12-20T12:42:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Edsnowden233](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/edsnowden233/32/214302_2.png) [@Edsnowden233](https://discourse.julialang.org/u/Edsnowden233)\
**Post date:** [December 20, 2024, 12:42pm UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/1 "2024-12-20T12:42:35Z")

</div>

I used the following code to test the data allocation on distributed workers. I found that broadcasting an ArbFloat variable to workers with precision equal to or higher than 98 bits will cause the IO error.  
**Code:**

> using ArbNumerics, Distributed  
> rmprocs(workers())  
> addprocs(1);
> 
> @everywhere dn = 90  
> @everywhere using ArbNumerics  
> @everywhere setworkingprecision(ArbFloat,bits=dn)
> 
> a = ArbFloat(pi)
> 
> @everywhere function check\_data(x)  
> println("Worker $(myid()): Data check: ", typeof(x), " with value: ", x)  
> end
> 
> for i = 1:10  
> @everywhere setworkingprecision(ArbFloat,bits=dn+$i)  
> a = ArbFloat(pi)  
> try  
> @everywhere check\_data(a) catch e println("The error happened at bits = (dn+i) ",e)  
> end  
> end

“error”：

> Worker 1: Data check: ArbFloat{91} with value: 3.141592653589793238462643383279503  
> From worker 3: Worker 3: Data check: ArbFloat{91} with value: 3.141592653589793238462643383279503  
> Worker 1: Data check: ArbFloat{92} with value: 3.141592653589793238462643383279503  
> From worker 3: Worker 3: Data check: ArbFloat{92} with value: 3.141592653589793238462643383279503  
> Worker 1: Data check: ArbFloat{93} with value: 3.1415926535897932384626433832795029  
> From worker 3: Worker 3: Data check: ArbFloat{93} with value: 3.1415926535897932384626433832795029  
> Worker 1: Data check: ArbFloat{94} with value: 3.1415926535897932384626433832795029  
> From worker 3: Worker 3: Data check: ArbFloat{94} with value: 3.1415926535897932384626433832795029  
> Worker 1: Data check: ArbFloat{95} with value: 3.14159265358979323846264338327950288  
> From worker 3: Worker 3: Data check: ArbFloat{95} with value: 3.14159265358979323846264338327950288  
> Worker 1: Data check: ArbFloat{96} with value: 3.14159265358979323846264338327950288  
> From worker 3: Worker 3: Data check: ArbFloat{96} with value: 3.14159265358979323846264338327950288  
> Worker 1: Data check: ArbFloat{97} with value: 3.141592653589793238462643383279502884  
> From worker 3: Worker 3: Data check: ArbFloat{97} with value: 3.141592653589793238462643383279502884  
> Worker 1: Data check: ArbFloat{98} with value: 3.141592653589793238462643383279502884  
> From worker 3: Worker 3: Data check: ArbFloat{98} with value:  
> Worker 3 terminated.
> 
> The error happened at bits = 98
> 
> Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)  
> Stacktrace:  
> [1] wait\_readnb(x::Sockets.TCPSocket, nb::Int64)  
> @ Base .\stream.jl:410  
> [2] (::Base.var"#wait\_locked#832")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)  
> @ Base .\stream.jl:981  
> [3] unsafe\_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)  
> @ Base .\stream.jl:987  
> [4] unsafe\_read  
> @ .\io.jl:891 [inlined]  
> [5] unsafe\_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)  
> @ Base .\io.jl:890  
> [6] read!  
> @ .\io.jl:895 [inlined]  
> [7] deserialize\_hdr\_raw  
> @ C:\Users\dell.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Distributed\src\messages.jl:167 [inlined]  
> [8] message\_handler\_loop(r\_stream::Sockets.TCPSocket, w\_stream::Sockets.TCPSocket, incoming::Bool)  
> @ Distributed C:\Users\dell.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Distributed\src\process\_messages.jl:172  
> [9] process\_tcp\_streams(r\_stream::Sockets.TCPSocket, w\_stream::Sockets.TCPSocket, incoming::Bool)  
> @ Distributed C:\Users\dell.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Distributed\src\process\_messages.jl:133  
> [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()  
> @ Distributed C:\Users\dell.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Distributed\src\process\_messages.jl:121  
> CompositeException(Any[ProcessExitedException(3)])  
> Worker 1: Data check: ArbFloat{99} with value: 3.1415926535897932384626433832795028842  
> Worker 1: Data check: ArbFloat{100} with value: 3.1415926535897932384626433832795028842

---

<div class="post-metadata">

**Author:** ![Edsnowden233](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/edsnowden233/32/214302_2.png) [@Edsnowden233](https://discourse.julialang.org/u/Edsnowden233)\
**Post date:** [December 21, 2024, 7:42am UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/2 "2024-12-21T07:42:04Z")

</div>

```julia
using ArbNumerics, Distributed
addprocs(2);
@everywhere dn = 90
@everywhere using ArbNumerics
@everywhere setworkingprecision(ArbFloat,bits=dn)
a = ArbFloat(pi)
@everywhere function check_data(x)
    println("Worker $(myid()): Data check: ", typeof(x), " with value: ", x)
end

@everywhere check_data($b)
for i = 1:10
    @everywhere setworkingprecision(ArbFloat,bits=dn+$i)
    a = ArbFloat(pi)
    try
        @everywhere check_data($a)
    catch e
        println("The error happened at bits = $(dn+i)! ",e)
    end
end

```

---

<div class="post-metadata">

**Author:** ![Edsnowden233](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/edsnowden233/32/214302_2.png) [@Edsnowden233](https://discourse.julialang.org/u/Edsnowden233)\
**Post date:** [December 21, 2024, 7:44am UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/3 "2024-12-21T07:44:26Z")

</div>

If I use BigFloat, no error will appear. The origin of the problem seems to be the ArbNumerics package.

---

<div class="post-metadata">

**Author:** ![eldee](https://avatars.discourse-cdn.com/v4/letter/e/b5a626/32.png) [@eldee](https://discourse.julialang.org/u/eldee)\
**Post date:** [December 23, 2024, 2:39pm UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/4 "2024-12-23T14:39:10Z")

</div>

Hi, and welcome to the Julia community!

The Julia machinery concerning serialisation does not seem to be properly implemented in ArbNumerics.jl (which relies on `ccall`s). For example,

```julia
using Distributed, ArbNumerics

setworkingprecision(10_000)

filename = tempname()
Distributed.serialize(filename, ArbFloat(pi))
println("ArbFloat: ", filesize(filename)) # 89

Distributed.serialize(filename, BigFloat(pi, precision=10_000))
println("BigFloat: ", filesize(filename)) # 1324

```

So we store the approximately 10 000 bits required to represent `ArbFloat(pi)` using only 89 bytes. Obviously something goes wrong here. In contrast, `BigFloat` uses a reasonable number of bytes (a bit more than 1250).

(`BigFloat` also relies on `ccall`s, but seeing it is a core part of Julia, I imagine proper care has indeed been taken to make it fit into the rest of the framework.)

* * *

I’m not sure how `ArbFloat` works internally, but you can see that the `dump` output changes drastically at 98 bits and no longer represents the value itself:

```julia-repl
julia> for nbits = 96:99
           setworkingprecision(ArbFloat, bits=nbits)
           dump(ArbFloat(pi))
       end
ArbFloat{96}
  exp: Int64 2
  size: UInt64 0x0000000000000004
  d1: UInt64 0xc4c6628b80dc1cd1
  d2: UInt64 0xc90fdaa22168c234
ArbFloat{97}
  exp: Int64 2
  size: UInt64 0x0000000000000004
  d1: UInt64 0xc4c6628b80dc1cd1
  d2: UInt64 0xc90fdaa22168c234
ArbFloat{98}
  exp: Int64 2
  size: UInt64 0x0000000000000006
  d1: UInt64 0x0000000000000003
  d2: UInt64 0x000001bce4e04ba0
ArbFloat{99}
  exp: Int64 2
  size: UInt64 0x0000000000000006
  d1: UInt64 0x0000000000000003
  d2: UInt64 0x000001bce4e04b80

julia> (big(0xc4c6628b80dc1cd1) + big(0xc90fdaa22168c234) << 64) / big(2)^(128 - 2) # presumable interpretation of the ArbFloat{97} bits
3.141592653589793238462643383279502884195286358297445035858198682759360371345934

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 23, 2024, 3:01pm UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/5 "2024-12-23T15:01:48Z")

</div>

> [@eldee](#):
>
> The Julia machinery concerning serialisation does not seem to be properly implemented in [ArbNumerics.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/ArbNumerics) (which relies on `ccall`s).

Yes, the problem seems to be that ArbNumerics needs to overload `Serialization.serialize` for `ArbFloat` (similar to e.g. [how `BigInt` is serialized](https://github.com/JuliaLang/julia/blob/dde5028a11662fca831dd676505264566aec84ab/stdlib/Serialization/src/Serialization.jl#L346-L349)).

Right now, it is using the default `serialize` method, which just writes the contents of the Julia `struct`. Since this probably contains a C pointer, that won’t work.

I filed an issue for further discussion of this: [implement Serialization serialize/deserialize · Issue #77 · JeffreySarnoff/ArbNumerics.jl · GitHub](https://github.com/JeffreySarnoff/ArbNumerics.jl/issues/77)

---

<div class="post-metadata">

**Author:** ![Edsnowden233](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/edsnowden233/32/214302_2.png) [@Edsnowden233](https://discourse.julialang.org/u/Edsnowden233)\
**Post date:** [December 24, 2024, 7:37am UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/6 "2024-12-24T07:37:14Z")

</div>

Thank you! Might get the feedback from the dev after xmas 👍

---

<div class="post-metadata">

**Author:** ![Edsnowden233](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/edsnowden233/32/214302_2.png) [@Edsnowden233](https://discourse.julialang.org/u/Edsnowden233)\
**Post date:** [December 24, 2024, 7:43am UTC](https://discourse.julialang.org/t/why-did-i-get-io-error-when-broadcasting-higher-precision-bits-98-arbfloat-variables-to-workers/124025/7 "2024-12-24T07:43:49Z")

</div>

Jeff has responded in the channel of #distributed on Julia slack. He is on vacation and will not be back until Mid January