# Inconsistent results based on number of processes

**URL:** <https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191>\
**Category:** Internals & Design\
**Created:** [August 2, 2017, 7:18pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191 "2017-08-02T19:18:49Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![appascoe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/appascoe/32/1537_2.png) [@appascoe](https://discourse.julialang.org/u/appascoe)\
**Post date:** [August 2, 2017, 7:18pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/1 "2017-08-02T19:18:49Z")

</div>

I stumbled across this inconsistency, which I find disconcerting, during a process of upgrading from 0.4.7 to 0.6. (Also, I discovered that the manual’s example of an echo server does not properly handle multiple connections, but perhaps that’s another topic, and is likely related to this example.)

Create a file, `test.jl`:

```julia
@sync begin
    ref = @spawn 1 + 1

    function whatever(ref)
        sleep(5)
        write(STDOUT, string(fetch(ref), "\n"))
    end

    @async whatever(ref)

    ref = @spawn 2 + 2
end

```

Then:

```julia
> julia test.jl
4
> julia -p 2 test.jl
2

```

What’s the reason for the discrepancy? Is this behavior expected?

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [August 2, 2017, 8:59pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/2 "2017-08-02T20:59:30Z")

</div>

The order of the two actions:

- `write(STDOUT, string(fetch(ref), "\n"))`
- `ref = @spawn 2 + 2`

is unspecified in terms of the concurrency logic of Julia according to the code. When there are two Julia processes available, one order is chosen, and when there is one the reverse.

If for example the `ref = @spawn 2+2` was after the `@sync`’s `end` then it was guaranteed to run after the `write` and the script would always output 2.

---

<div class="post-metadata">

**Author:** ![appascoe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/appascoe/32/1537_2.png) [@appascoe](https://discourse.julialang.org/u/appascoe)\
**Post date:** [August 3, 2017, 1:00am UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/3 "2017-08-03T01:00:35Z")

</div>

So I added the `@sync` so I’d get the stdout on the shell. I think the real issue here is when the expression `whatever(ref)` gets resolved. Let me point out the issue I have with the manual for the TCP server (the one using the nested `@async` at the bottom of [this section](https://docs.julialang.org/en/stable/manual/networking-and-streams/#A-simple-TCP-example-1)). I don’t believe it does what it claims to do by allowing multiple connections. Here:

```julia
server = listen(2000)
while true
    sock = accept(server)
    @async begin
        x = readline(sock)
        write(sock, x)
    end
end

```

Then, in a shell just do `> julia test.jl`. Pardon my python, but in IPython, I can then do:

```julia
In [1]: import socket

In [2]: s = socket.socket()

In [3]: s.connect(('', 2000))

In [4]: s.send('check check\n')
Out[4]: 12

In [5]: s.recv(1024)
Out[5]: 'check check'

```

Works great. _However_, there’s an issue with multiple connections.

```julia
In [1]: import socket

In [2]: s = socket.socket()

In [3]: t = socket.socket()

In [4]: s.connect(('', 2000))

In [5]: t.connect(('', 2000))

In [6]: s.send('check 1\n')
Out[6]: 8

In [7]: s.recv(1024)

```

And this hangs, waiting for the response. This is because the first `@async` task is on the queue, blocking on reading from the `s` socket. But after opening the `t` socket, `sock` gets redefined and instead the first task writes to the `t` socket, which is not the appropriate client. Simple enough to confirm:

```julia
In [1]: import socket

In [2]: s = socket.socket()

In [3]: t = socket.socket()

In [4]: s.connect(('', 2000))

In [5]: t.connect(('', 2000))

In [6]: s.send('check 1\n')
Out[6]: 8

In [7]: t.recv(1024)
Out[7]: 'check 1'

```

These weren’t issues in 0.4.7 (I suspect they were introduced in 0.5 with some of the changes to function closures, but that’s speculation on my part). Perhaps this is expected, or even desired, behavior, but at the very least, the manual should be much clearer about the perils here.

Maybe I missed it in the language spec, but it would be nice to have tight control over when an expression is evaluated/resolved/etc.

---

<div class="post-metadata">

**Author:** ![appascoe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/appascoe/32/1537_2.png) [@appascoe](https://discourse.julialang.org/u/appascoe)\
**Post date:** [August 3, 2017, 1:00am UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/4 "2017-08-03T01:00:38Z")

</div>

This seems to work appropriately, but is a bit convoluted:

```julia
server = listen(2000)
while true
    sock = accept(server)
    ex = quote
        x = readline($sock)
        write($sock, x)
    end
    @async eval(ex)
end

```

---

<div class="post-metadata">

**Author:** ![Michael\_Eastwood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/michael_eastwood/32/669_2.png) [@Michael\_Eastwood](https://discourse.julialang.org/u/Michael_Eastwood)\
**Post date:** [August 3, 2017, 3:54am UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/5 "2017-08-03T03:54:19Z")

</div>

This seems to work as well and is a little less convoluted. I don’t know why it works though…

```julia
server = listen(2000)
read_and_write(sock) = write(sock, readline(sock))
while true
    sock = accept(server)
    @async read_and_write(sock)
end

```

---

<div class="post-metadata">

**Author:** ![appascoe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/appascoe/32/1537_2.png) [@appascoe](https://discourse.julialang.org/u/appascoe)\
**Post date:** [August 3, 2017, 5:18pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/6 "2017-08-03T17:18:53Z")

</div>

Yeah, it’s not clear to me what makes that fundamentally different from the `begin ... end` block.

But you’ve expressed my overall concern: asynchronous tasks and parallelism can already be difficult to reason about, and adding an additional layer of uncertainty around when symbols get resolved makes it even harder. Generally speaking, it’s not a good situation where you write code that works (or seems to work…) and don’t know why it does what it does.

At the very least, these issues should be discussed heavily in the docs. It’s been a bumpy ride migrating a production system from 0.4.7 to 0.6, and that’s largely in part because the release notes for 0.5 and 0.6 didn’t, in my opinion, adequately touch on the consequences of the changes.

---

<div class="post-metadata">

**Author:** ![Michael\_Eastwood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/michael_eastwood/32/669_2.png) [@Michael\_Eastwood](https://discourse.julialang.org/u/Michael_Eastwood)\
**Post date:** [August 3, 2017, 5:45pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/7 "2017-08-03T17:45:55Z")

</div>

The original example seems to have stopped working sometime between v0.4.7 and v0.5.0 (just tested on an old installation). So the closure changes might indeed be responsible. I think you should open a github issue about the concerns raised in this thread. The docs could certainly use improvement and at the very least that example should be fixed.

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [August 4, 2017, 10:49am UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/8 "2017-08-04T10:49:22Z")

</div>

A `begin` block doesn’t introduce a new scope and therefore doesn’t capture `sock`. You could use a closure or a `let` block and the code works fine:

```julia
server = listen(2001)
while true
    sock = accept(server)
    @async let sock = sock
        println(STDOUT, readline(sock))
    end
    # or:
    # @async (() -> println(STDOUT, readline(sock)))()
end

```

---

<div class="post-metadata">

**Author:** ![Michael\_Eastwood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/michael_eastwood/32/669_2.png) [@Michael\_Eastwood](https://discourse.julialang.org/u/Michael_Eastwood)\
**Post date:** [August 4, 2017, 5:12pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/9 "2017-08-04T17:12:47Z")

</div>

But isn’t `@async` already wrapping its argument in a closure?

[https://github.com/JuliaLang/julia/blob/v0.6.0/base/task.jl#L334-L337](https://github.com/JuliaLang/julia/blob/v0.6.0/base/task.jl#L334-L337)

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [August 4, 2017, 5:28pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/10 "2017-08-04T17:28:26Z")

</div>

It is. No idea why that isn’t enough.

Also

```julia
@async let sock = sock
    x = readline(sock)
    println(sock, x)
end

```

works, while

```julia
@async begin
    x = readline(sock)
    println(sock, x)
end

```

doesn’t. `println(sock, readline(sock))` works without being wrapped though…

---

<div class="post-metadata">

**Author:** ![appascoe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/appascoe/32/1537_2.png) [@appascoe](https://discourse.julialang.org/u/appascoe)\
**Post date:** [August 4, 2017, 5:43pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/11 "2017-08-04T17:43:30Z")

</div>

Regardless of `@async` wrapping…

```julia
@async let sock = sock
    x = readline(sock)
    println(sock, x)
end

```

This isn’t clear to me when the `let` block actually gets executed. It seems like this may be the same as the previous race condition, where if the task sits on the queue, it’s not clear when `sock` is resolved. Semantically, wouldn’t this be clearer? (Haven’t tested this yet.)

```julia
let sock = sock
    @async begin
        x = readline(sock)
        println(sock, x)
    end
end

```

---

<div class="post-metadata">

**Author:** ![appascoe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/appascoe/32/1537_2.png) [@appascoe](https://discourse.julialang.org/u/appascoe)\
**Post date:** [August 4, 2017, 5:59pm UTC](https://discourse.julialang.org/t/inconsistent-results-based-on-number-of-processes/5191/12 "2017-08-04T17:59:03Z")

</div>

At the risk of bifurcating the discussion, I created a GitHub issue: [https://github.com/JuliaLang/julia/issues/23130](https://github.com/JuliaLang/julia/issues/23130)