# Non-blocking network IO

**URL:** <https://discourse.julialang.org/t/non-blocking-network-io/123344>\
**Category:** General Usage\
**Created:** [December 2, 2024, 7:43am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344 "2024-12-02T07:43:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 2, 2024, 7:43am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/1 "2024-12-02T07:43:35Z")

</div>

I’m going to write a node that maintains network connections with other nodes. When the node is told to shut down, it has to finish sending and receiving messages, then shut down. This has to take a few seconds at most. (Julia 1.11.1)

First I create a listening socket:

```julia
julia> svr=listen(n) # I'm keeping the port number secret for now
Sockets.TCPServer(RawFD(25) active)

```

In another terminal, I connect to the port using netcat, then accept the connection in Julia:

```julia
julia> c2=accept(svr)
TCPSocket(RawFD(26) open, 0 bytes waiting)

```

First problem: how do I tell if there is a connection waiting on the listening socket? `man accept` says it’s possible

> In order to be notified of incoming connections on a socket, you can use select(2), poll(2), or epoll(7).  
> but how do I do this in Julia?

I sent the socket a few kilobytes. (First I tried eight bytes, `aoeusnth`, then thought that’s a lot smaller than the MTU, so I tried about 5 kB.) Then I did `bytesavailable(c2)` and got 0. I ran `netstat -t` and saw that there were no bytes waiting in the receive or transmit queue. I did `read(c2,4)` and got the first four bytes of the file. Then I did `bytesavailable(c2)` and got 4910. The connection is on localhost, so the bytes should be available immediately, shouldn’t they?

Second problem: where are these bytes hiding, and how can I tell if there are some bytes hiding there? [How to read from socket in non-blocking mode](https://discourse.julialang.org/t/how-to-read-from-socket-in-non-blocking-mode/4710) is relevant, but it’s from 2017-07, before Julia 1.0. If I knew that some bytes were hiding there, I could call `read(c2,1)` and `bytesavailable` would tell me how many more bytes there were.

In case it matters, the program will AFAIK run only on some sort of Unix.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 2, 2024, 9:54am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/2 "2024-12-02T09:54:33Z")

</div>

Julia doesn’t currently have non-blocking I/O. As the [documentation for `bytesavailable`](https://docs.julialang.org/en/v1/base/io-network/#Base.bytesavailable) says:

> Return the number of bytes available for reading before a read from this stream or buffer will block.

In effect this means that even if the underlying abstraction has data buffered, there still needs to be some minimally blocking call that updates the Julia-exposed datastructure before `bytesavailable` can show this on the Julia side.

Libuv (which julia uses under the hood here) in theory has recently merged io\_uring support, but this is not exposed in this API (which is quite unfortunate from my POV).

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 2, 2024, 10:42am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/4 "2024-12-02T10:42:09Z")

</div>

Is there a way to call libuv to find out if the buffer has data?

---

<div class="post-metadata">

**Author:** ![Pedro](https://avatars.discourse-cdn.com/v4/letter/p/919ad9/32.png) [@Pedro](https://discourse.julialang.org/u/Pedro)\
**Post date:** [December 2, 2024, 10:52am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/5 "2024-12-02T10:52:21Z")

</div>

I am currently writing Julia code with network IO and unfortunately the support for it is limited in comparison with C API. ☹

If you want to find out what you can call in Julia from libuv, you should probably start by looking here: [julia/src/jl\_uv.c at master · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/blob/master/src/jl_uv.c).

> Libuv (which julia uses under the hood here) in theory has recently merged io\_uring support, but this is not exposed in this API (which is quite unfortunate from my POV).

Be aware that Julia maintains and uses own [fork of Libuv](https://github.com/JuliaLang/libuv).

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 3, 2024, 3:25am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/6 "2024-12-03T03:25:42Z")

</div>

I don’t know how to make use of this. If there’s a line

```julia
JL_DLLEXPORT int jl_fs_rename(const char *src_path, const char *dst_path)

```

does that mean that I can call `fs_rename` or `jl_fs_rename` from Julia? I also don’t know libuv.

I typed `using Sockets`, and by hitting tab found out that there is `accept_nonblock`, but it has no docstring. Where’s the source? That would probably also help me figure out how to use jl\_uv.c.

---

<div class="post-metadata">

**Author:** ![Pedro](https://avatars.discourse-cdn.com/v4/letter/p/919ad9/32.png) [@Pedro](https://discourse.julialang.org/u/Pedro)\
**Post date:** [December 3, 2024, 11:38am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/7 "2024-12-03T11:38:14Z")

</div>

If you want call C function, then you should study more about [calling C functions](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/) from Julia.

For example, to call `jl_fs_rename`, you can use `@ccall` macro:

```julia
@ccall jl_fs_rename("test"::Cstring, "test2"::Cstring)::Cint

```

`Sockets.accept_nonblock` is a private symbol, so it’s not advised to use it, unless you know what you doing. As docstring warns

> The following bindings may be internal; they may change or be removed in future versions: Sockets.accept\_nonblock.

REPL help, shows the path to the source code, so you can check the source code of this function to learn more about.

---

<div class="post-metadata">

**Author:** ![caleb-allen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/caleb-allen/32/14054_2.png) [@caleb-allen](https://discourse.julialang.org/u/caleb-allen)\
**Post date:** [December 3, 2024, 1:07pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/8 "2024-12-03T13:07:53Z")

</div>

> [@Sukera](#):
>
> Libuv (which julia uses under the hood here) in theory has recently merged io\_uring support, but this is not exposed in this API (which is quite unfortunate from my POV).

It is very unfortunate, there was a great talk last JuliaCon about the traps of running an HTTP service, many of which were related to IO scheduling and would be alleviated (directly or indirectly) with io\_uring.

There was [interest](https://discourse.julialang.org/t/io-uring-support/48666) in using io\_uring prior to libuv gaining the feature, I imagine the barrier is a matter of manpower (I recall reading about libuv being an especially hairy dependency to upgrade?)

_Edit:_ [Here’s the JuliaCon talk](https://youtu.be/veZ7xB62olI)

---

<div class="post-metadata">

**Author:** ![Pedro](https://avatars.discourse-cdn.com/v4/letter/p/919ad9/32.png) [@Pedro](https://discourse.julialang.org/u/Pedro)\
**Post date:** [December 3, 2024, 2:17pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/9 "2024-12-03T14:17:59Z")

</div>

That’s a really good talk. At least after watching it I probably can answer my own question I have recently posted. 🙂

> [@Does stdlib network sockets use only first thread?](https://discourse.julialang.org/t/does-stdlib-network-sockets-use-only-first-thread/123027/1):
>
> I am writing a multi-threaded applications with a GUI (CImGui.jl) and a network sockets (Base.Sockets) to handle low-latency data stream over local network via UDP. I have a few independent threads to handle data processing and networking. First UDPSocket (source of data) and recv calls in one thread, and second UDPSocket (sink for processed data) and send calls in another thread. Data exchange between threads is done via Channel. After various test why the output network stream lags, I found …

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 3, 2024, 3:09pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/10 "2024-12-03T15:09:17Z")

</div>

> [@caleb-allen](#):
>
> It is very unfortunate, there was a great talk last JuliaCon about the traps of running an HTTP service, many of which were related to IO scheduling and would be alleviated (directly or indirectly) with io\_uring

The thing is, even if julias libuv fork would use io\_uring under the hood, we’d still have to “hide” that behind the current (blocking) API. If you want to make proper use of asynchronicity, you _have_ to expose the asynchronicity of the underlying API though, which would make for a very different API. We could immediately return from a `read` with a `Future`, for example, so that the I/O is done asynchronously to our code doing something different. If we only have a blocking API exposed, we have to manually create tasks that handle the blocking part again, which IMO defeats the purpose of using io\_uring under the hood in the first place.

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 4, 2024, 5:02am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/11 "2024-12-04T05:02:49Z")

</div>

I tried `accept_nonblock` and this is what happened:

```julia
julia> c1=Sockets.accept_nonblock(svr)
ERROR: IOError: accept: resource temporarily unavailable (EAGAIN)
Stacktrace:
 [1] uv_error
   @ ./libuv.jl:106 [inlined]
 [2] accept_nonblock(server::Sockets.TCPServer)
   @ Sockets /usr/local/lib/julia-1.11.1/share/julia/stdlib/v1.11/Sockets/src/Sockets.jl:679
 [3] top-level scope
   @ REPL[98]:1
 [4] top-level scope
   @ none:1

julia> c1
ERROR: UndefVarError: `c1` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ :0
 [2] top-level scope
   @ none:1

```

Then I connected to the port with netcat and tried again.

```julia
julia> c1=Sockets.accept_nonblock(svr)
TCPSocket(RawFD(26) open, 0 bytes waiting)

```

This is good enough for me; I can `try` to accept the socket, `catch` the error if no one’s connected, and pass the connection to a thread if someone connected.

There is no `read` function in Sockets.jl. Where is the `read` function that reads from a socket? `?read` doesn’t tell me where it is because there’s a docstring.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [December 4, 2024, 10:52am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/12 "2024-12-04T10:52:50Z")

</div>

Why not define a non-blocking Julia API first, which uses a Julia task, and if and when the libuv fork is updated, use the non-blocking API from libuv? Or are the two solutions different in behaviour?

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 4, 2024, 11:47am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/13 "2024-12-04T11:47:37Z")

</div>

What do you mean by “uses a Julia task”? Any task must never wait indefinitely for something, so that the main thread can tell it to stop and know that it will stop within a limited time.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [December 4, 2024, 6:28pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/14 "2024-12-04T18:28:19Z")

</div>

Sorry, I was referring to the statement

> [@Sukera](#):
>
> If we only have a blocking API exposed, we have to manually create tasks that handle the blocking part

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 4, 2024, 7:06pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/15 "2024-12-04T19:06:16Z")

</div>

> [@phma](#):
>
> There is no `read` function in [Sockets.jl](https://juliahub.com/ui/Packages/General/Sockets). Where is the `read` function that reads from a socket? `?read` doesn’t tell me where it is because there’s a docstring.

It’s just the regular `read` function:

```julia
julia> TCPSocket <: IO
true

```

It doesn’t have its own docstring, but you can still get to the method with `edit(read, (TCPSocket,))`. You’ll see that this ends up taking the IO lock again, so you won’t get around having to wait if something else wants to use IO at the same time.

> [@simsurace](#):
>
> Why not define a non-blocking Julia API first, which uses a Julia task, and if and when the libuv fork is updated, use the non-blocking API from libuv? Or are the two solutions different in behaviour?

One problem is that Julia-level IO takes an IO lock whenever an operation interacts with libuv, so that concurrency on the julia side doesn’t mess up the internal state. If you want to move existing code using “regular” `read`/`write`s to non-blocking use of libuv, you may inadvertently introduce correctness bugs in programs that implicitly relied on those locks for correctness. That’s part of why I’m saying an async API would/should look different from just `read`/`write`.

Another problem is that you likely won’t be able to control rescheduling of your `Task` objects in the way that you want them to be scheduled, because you don’t have control over when the Julia scheduler reschedules a specific task. It’s a somewhat hidden implementation detail.

This is subtle, but basically means that you either have to weave locks per-`Task` through every operation of your async API again (which you don’t want, since that means artificial overhead when those `Task`s are woken up and notice that they aren’t allowed to progress yet) or forego the julia-native `Task`s and roll your own. In the latter case you need to be careful to never `yield` to the Julia scheduler on your main julia `Task` that acts as your scheduler, or you might not be rescheduled in time again when your “hidden” IO is poking you to wake up your other tasks.

I’ve had these exact scenarios happen when I experimented with this a while ago while using Julia `Task`s - if you only have one thread, you can’t ever `yield` from your scheduler! But then your Julia `Task`s don’t ever get scheduled by the Julia scheduler either. And if you do `yield`, you can easily deadlock/introduce overhead because there’s no guarantee that your scheduler is scheduled again when you want it to be. It’s a Catch-22.

Other than these (and probably more) difficulties though, yes, that’s absolutely in the realm of possibility, and I’d love to see it! 🙂

* * *

Finally, a bit of a cleanup/correction: if you want to get _technically correct_, you could link to the manual and mention that Julia does have Asynchronous I/O:

> **[Networking and Streams · The Julia Language](https://docs.julialang.org/en/v1/manual/networking-and-streams/#Asynchronous-I/O)**
>
> Documentation for The Julia Language.

A quick look at the text though reveals that what is meant here is that calling the regular, blocking `read`/`write` inside an `@async` block is totally possible. This of course ends up blocking the async `Task` that’s spawned by the `@async`! IMO, that’s a far cry from an async-native I/O API, since it sidesteps all the issues with async I/O and leaves them up to the user to figure out themselves.

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 4, 2024, 7:46pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/16 "2024-12-04T19:46:23Z")

</div>

> [@Sukera](#):
>
> edit(read, (TCPSocket,))

That brings up stream.jl (in nano, I then opened it in kwrite). The function contains `bytes = take!(stream.buffer)`, so I’m searching the file for `.buffer` to figure out how something gets into the buffer.

As to the IO lock, if no one holds the lock for more than a second, it’s not a problem. Each socket is read and written by only one task.

---

<div class="post-metadata">

**Author:** ![simsurace](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simsurace/32/30216_2.png) [@simsurace](https://discourse.julialang.org/u/simsurace)\
**Post date:** [December 4, 2024, 8:23pm UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/17 "2024-12-04T20:23:04Z")

</div>

Thanks for the clarifications. My relatively naive approach up to now has been to spawn a task to do IO and let it put the result in some channel. Maybe the tediousness that comes with this is what you are referring to when you say that all is on the user to figure out for themselves. I guess one could write some abstractions to make things like these more convenient, and they may behave almost the same as true async IO, but I think I‘m out of my depth to understand the subtleties after this.

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 5, 2024, 7:01am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/18 "2024-12-05T07:01:33Z")

</div>

There seems to be a misunderstanding. What am I trying to do?

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 5, 2024, 7:30am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/19 "2024-12-05T07:30:10Z")

</div>

@simsurace is not replying to you - note the small arrow & profile picture in the upper right hand corner of their message, indicating that it’s a reply to my earlier wall of text 🙂 You can also click that arrow to expand the originally replied-to message.

> [@phma](#):
>
> As to the IO lock, if no one holds the lock for more than a second, it’s not a problem. Each socket is read and written by only one task.

I have not further looked into the exact details of that lock, but my understanding is that it’s for the entirety of libuv across all tasks. So if something were to write tens of gigabytes of data into a socket with a single `write` call, my impression would be that the lock would be held for the entirety of that call, which may take quite a while. The lock is shared across tasks if I read the source correctly, so it doesn’t matter whether your tasks only access a single socket or not.

---

<div class="post-metadata">

**Author:** ![phma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phma/32/6576_2.png) [@phma](https://discourse.julialang.org/u/phma)\
**Post date:** [December 5, 2024, 7:42am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/20 "2024-12-05T07:42:22Z")

</div>

The most written in one `write` call is a few kilobytes. I haven’t decided if one message could ever be more than a gigabyte, but I’m leaning against it. If not, the longest message will be about 80 MB.

Is it possible for a `write` call to block indefinitely? I’m thinking it might happen if the network connection went down in the middle of writing a big message, and the outgoing buffer filled up, in which case it’d wait for the OS to figure out that the TCP connection was cut off.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 5, 2024, 10:00am UTC](https://discourse.julialang.org/t/non-blocking-network-io/123344/21 "2024-12-05T10:00:07Z")

</div>

> [@phma](#):
>
> I haven’t decided if one message could ever be more than a gigabyte, but I’m leaning against it. If not, the longest message will be about 80 MB.

To my understanding this lock is shared across _all_ tasks, not just yours. So if there’s another library performing that long-winded call, it would theoretically block your code too.

To be clear, I haven’t tested this extensively, this is just from my understanding of what the code seems to be doing. I can try to run that experiment later today, it’s fairly straightforward. Just a netcat server listening, piping what it receives to `/dev/null`, launching a julia task that writes a few gigabyte large array in one call to a TCPSocket and attempt to send another networked message on another background task at the same time. If there’s a delay for the background message, there’s interference between tasks due to one write taking an abnormally long time.

[Next page](https://discourse.julialang.org/t/non-blocking-network-io/123344.md?page=2)