I thought I understood "pass-by-sharing" in Julia until I found this

I guess this really highlights your questions in the other thread about variables being label/boxes - it isn’t totally clear sometimes - not in the language generally, but in the transfer of code from different scopes from global/REPL~ scope to functions.

But when I move this snippet to a local scope, behaviour is the same:

let
  anon = Array{Any}(undef, 2)
  for i = 1:2
    anon[i] = ()-> println(i)
    i += 1
  end

  anon[1]()
  anon[2]()
end
.....
2
3

Where is syntax like anon[1].i described? A function has fields?

Hmm yeah. So the for loop scope is always captured as a boxed variable.

But if you do it with map instead:

julia> let
         anon = map(1:2) do i
             ()-> println(i)
             i += 1
         end
         anon[1]()
         anon[2]()
       end
1
2

Oops I missed the key line- it’s the same with map too:

julia> let
        anon = map(1:2) do i
            f = ()-> println(i)
            i += 1
            f
        end
        anon[1]()
        anon[2]()
       end
2
3

Someone more familiar with internals would have to answer here.

But anonymous functions are really functors underneath. I think the fields are an implementation detail, not something you’re mean to mess with so they aren’t really discussed.

1 Like

This is explained in detail in the docs for let blocks. With examples. Search for the part starting with

Here is an example where the behavior of let is needed:

Please

  1. Do read the manual. It is written to be read and a lot of work went into it to explain details like this.

  2. Try to give up all the metaphors from the C/C++ world (“pass by sharing”) etc. They will just distract you from understanding Julia.

1 Like

This metaphor does not exist in C/C++ world, it is taken straight from Julia manual (https://docs.julialang.org/en/v1/manual/functions/#Argument-Passing-Behavior):

Julia function arguments follow a convention sometimes called “pass-by-sharing”, which means that values are not copied when they are passed to functions. Function arguments themselves act as new variable bindings (new locations that can refer to values), but the values they refer to are identical to the passed values.

I read the manual trying to find more precise definition of these metaphors, but I failed so far.

1 Like

Your example has nothing to do with argument passing; it is about scope (see the link).

FWIW, I think that the whole call-by-X discussion is just confusing as the Xs are not clearly defined — if anything, Julia is call-by-value. But, again, this is a red herring here.

1 Like

Parameter passing has to be defined somewhere, at least in the minds of Julia implementers. I’m looking for this definition in order to avoid frequent surprises.

It is well defined, but the problem is that the terminology isn’t.

But it is much, much simpler than you think. Basically if I call f(a), then within

function f(b)
    ...
end

you have a === b to start with. That’s all there is to it, really.

1 Like

That would lead me to expect:

let
  anon = Array{Any}(undef, 2)
  for i = 1:2
    anon[i] = ()-> println(i)
    i += 1
  end

  anon[1]()
  anon[2]()
end
.....
3
3

but I need to user outer for this to happen:

let
  anon = Array{Any}(undef, 2)
  i=0
  for outer i = 1:2
    anon[i] = ()-> println(i)
    i += 1
  end

  anon[1]()
  anon[2]()
end
.....
3
3

The i += 1 is lowered into i = i + 1, which is an assignment, and creates an i in the local scope.

2 Likes

This is true, but we have to be honest that in the context of the example it’s still confusing, even with a few reads of the docs.

Knowing that the i + 1 boxed variable is what will be return by the anonymous function, instead of the value of i at the line the function was created on isn’t totally obvious.

And there is an actual practical switch between pass-by-reference and pass-by-value behavior happening in the background when constructing the anonymous function struct, depending on the scope of i - so it’s not a completely unrelated problem.

Anyway it’s probably difficult for a newcomer, even with the docs. It’s still confusing to me after registering a bunch of packages. Scoping rules are confusing, Julia has quite a history of demonstrating this.

3 Likes

Possibly. Personally, I don’t expect all aspects of programming to be “totally obvious”. Most interesting things in life aren’t.

Understanding scoping in any nontrivial language requires working through a few examples. Fortunately, the manual has quite a few of these now, but I am sure that more pertinent ones are always welcome as PRs.

The history of Julia’s scoping rules is indeed a bit meandering, but that is not really required for understanding how they work now. I don’t think that Julia’s scoping rules are especially confusing, but YMMV.

3 Likes

The i inside a closure is in the same scope as the i outside it so updates to i is visible inside the closure. The potentially confusing thing is that a closure can outlive the lexical scope where it was defined.

2 Likes

Is this what you wanted?

anon = Array{Any}(undef, 2)
for i = 1:2
  local k = i
  anon[i] = ()-> println(k)
end

It gives

julia> anon[1]()
1

julia> anon[2]()
2

julia>     

In the OP case, as other have already pointed out, I think the root of the problem lies in a very common misunderstanding about the scope of for loops.

A for block does not define a single scope, but a succession of scopes one for each iteration.

The distinction between a single scope and a succession of scopes is often little relevant (I believe this is the reason for it being a common misunderstanding). However, together with closures it become relevant and crashes a mental model in which all iterations shared the same inner scope.

I think it can be illustrated by:

julia> function f()
           for i = 1:5
               if @isdefined x
                   println("x found: $x")
               else
                   println("x not found")
               end
               x = i
           end
           return
       end
f (generic function with 1 method)

julia> f()
x not found
x not found
x not found
x not found
x not found
13 Likes

While this may be a subtle and common misunderstanding, I don’t think this is the root problem in this case. At least for me it wasn’t.

Both of the following would not meet OP expectation:

Succession of scopes one for each iteration:

anon = Array{Any}(undef, 2)
for i = 1:2
    anon[i] = ()-> println(i)
    i += 1
end
anon[1]()
anon[2]()

Single scope across all iterations:

anon = Array{Any}(undef, 2)
let
    local i
    for outer i = 1:2
        anon[i] = ()-> println(i)
        i += 1
    end
end
anon[1]()
anon[2]()

Rather, I think the key insight to help understanding in this case is:

anon = Array{Any}(undef, 2)
for i = 1:2
    let i = i
        anon[i] = ()-> println(i)
    end
    i += 1
end
anon[1]()
anon[2]()

Is my solution identical to yours?

I wasn’t trying to solve a problem with closures. I was searching for something related to for loop scope and I run into this particular example, which confused me.

This is a part of a bigger problem I have searching through Julia documentation. It would really help to have a BNF (or another formal notation) syntax of Julia published somewhere, otherwise searching for keywords like outer is futile.

3 Likes

The search facility is steadily getting better, I think in this particular case the problem is really the documentation for outer. See

2 Likes

Yes, both maybe would not match what OP expected, even elucidating this misconception. There are others things at play, but mostly, they are related to how closures work, and after understanding the scope problem, I think they become minor details.

This gives

2
3

and I suppose you mean the expected would be

1
2

What happens here is that the first closure captures the binding i in the first scope, and the second closure captures the binding i in the second scope. Moreover, both closures are callled after the loop has ended. The i of the first scope, at the ends of its loop/scope, has value 2, and the i of the second scope, at the ends of its loop/scope, has value 3. These both distinct is keep existing because they are referred by the closure, and they print the last value they had. If you did:

anon = Array{Any}(undef, 2)
for i = 1:2
    # the nothing below is just to avoid extra printing in the REPL
    anon[i] = () -> (println(i); i += 1; nothing)
    i += 1
end
anon[1]()
anon[1]()
anon[1]()
anon[1]()
anon[1]()

anon[2]()
anon[2]()

You would get

2
3
4
5
6
3
4

If you want something that really is surprising, you should check that closures are able to capture not-yet-defined variables of the scope in which they are defined.

anon = Array{Any}(undef, 2)
for i = 1:2
    anon[i] = () -> println(x)
    x = i * 2
end
anon[1]()
anon[2]()

gives

2
4

But at least:

anon = Array{Any}(undef, 2)
for i = 1:2
    anon[i] = () -> println(x)
    anon[i]()
    x = i * 2
end

Gives

ERROR: UndefVarError: x not defined

In the second example you give, I do not really agree the behaviour is surprising:

anon = Array{Any}(undef, 2)
let
    local i
    for outer i = 1:2
        anon[i] = ()-> println(i)
        i += 1
    end
end
anon[1]()
anon[2]()

Giving

3
3

Makes complete sense to me. i is not a binding of the inner scope, it is a binding of the outer scope (as the keyword indicates) so there is only one of it, and it is shared among all iterations. The for loop do not declare a new binding i each iteration but just attributes the correct value to the i already available. Note that the for do not just increase i, otherwise the second iteration could end up being skipped, it does attribute to i the right value each iteration. In the example below, anon[2] would not be defined if it did not work this way:

anon = Array{Any}(undef, 2)
let
    local i
    for outer i = 1:2
        anon[i] = ()-> println(i)
        i = 20
    end
end

anon

Gives

2-element Array{Any,1}:
 var"#3#4"(Core.Box(20))
 var"#3#4"(Core.Box(20))

This kinda kills my argument there are no boxes in Julia :sweat_smile: :laughing:.

Finally, as I said before, all closures are capturing and sharing the same binding, what can be checked by:

anon = Array{Any}(undef, 2)
let
    local i
    for outer i = 1:2
        # the nothing below is just to avoid extra printing in the REPL
        anon[i] = ()-> (println(i); i *= 2; nothing)
    end
end
anon[1]()
anon[2]()
anon[1]()
anon[2]()
anon[1]()
anon[2]()

Which gives:

2
4
8
16
32
64
5 Likes