I thought I understood "pass-by-sharing" in Julia until I found this

Here is a snippet from “Julia 1.0 Programming Complete Reference Guide”:

anon = Array{Any}(undef, 2)
for i = 1:2
  anon[i] = ()-> println(i)
  i += 1
end

Can someone explain or point me to explanation why

julia> anon[1]()
2

not

julia> anon[1]()
1

as I expected. The manual reads: Function arguments themselves act as new variable bindings (new locations that can refer to values), but the values they refer to are identical to the passed values. Argument i is bound to value 1 when anon[1]() is defined, so the passed value is 1. What does location that can refer to values mean? Is there a more formal definition of these concepts somewhere?

4 Likes

And:

julia> anon[2]()
3

The reason seems to be that i is a boxed variable:

julia> anon[1].i
Core.Box(2)

If you use a let block you should get the what you’re going for:

julia> for i = 1:2
         let i = i 
             anon[i] = ()-> println(i)
         end
         i += 1
       end

julia> anon[1]()
1

This anonymous function is a struct of bitstypes, not boxed variables. I guess you need to be careful with anonymous functions in global scope Edit: in scopes where the variables are changed, as they seem similar but are not really the same thing as anonymous functions in for loop/let block scope where the variable is not changed.

4 Likes

I guess this really highlights your questions in the other thread about variables being label/boxes - it isn’t totally clear sometimes - not in the language generally, but in the transfer of code from different scopes from global/REPL~ scope to functions.

But when I move this snippet to a local scope, behaviour is the same:

let
  anon = Array{Any}(undef, 2)
  for i = 1:2
    anon[i] = ()-> println(i)
    i += 1
  end

  anon[1]()
  anon[2]()
end
.....
2
3

Where is syntax like anon[1].i described? A function has fields?

Hmm yeah. So the for loop scope is always captured as a boxed variable.

But if you do it with map instead:

julia> let
         anon = map(1:2) do i
             ()-> println(i)
             i += 1
         end
         anon[1]()
         anon[2]()
       end
1
2

Oops I missed the key line- it’s the same with map too:

julia> let
        anon = map(1:2) do i
            f = ()-> println(i)
            i += 1
            f
        end
        anon[1]()
        anon[2]()
       end
2
3

Someone more familiar with internals would have to answer here.

But anonymous functions are really functors underneath. I think the fields are an implementation detail, not something you’re mean to mess with so they aren’t really discussed.

1 Like

This is explained in detail in the docs for let blocks. With examples. Search for the part starting with

Here is an example where the behavior of let is needed:

Please

  1. Do read the manual. It is written to be read and a lot of work went into it to explain details like this.

  2. Try to give up all the metaphors from the C/C++ world (“pass by sharing”) etc. They will just distract you from understanding Julia.

1 Like

This metaphor does not exist in C/C++ world, it is taken straight from Julia manual (https://docs.julialang.org/en/v1/manual/functions/#Argument-Passing-Behavior):

Julia function arguments follow a convention sometimes called “pass-by-sharing”, which means that values are not copied when they are passed to functions. Function arguments themselves act as new variable bindings (new locations that can refer to values), but the values they refer to are identical to the passed values.

I read the manual trying to find more precise definition of these metaphors, but I failed so far.

1 Like

Your example has nothing to do with argument passing; it is about scope (see the link).

FWIW, I think that the whole call-by-X discussion is just confusing as the Xs are not clearly defined — if anything, Julia is call-by-value. But, again, this is a red herring here.

1 Like

Parameter passing has to be defined somewhere, at least in the minds of Julia implementers. I’m looking for this definition in order to avoid frequent surprises.

It is well defined, but the problem is that the terminology isn’t.

But it is much, much simpler than you think. Basically if I call f(a), then within

function f(b)
    ...
end

you have a === b to start with. That’s all there is to it, really.

1 Like

That would lead me to expect:

let
  anon = Array{Any}(undef, 2)
  for i = 1:2
    anon[i] = ()-> println(i)
    i += 1
  end

  anon[1]()
  anon[2]()
end
.....
3
3

but I need to user outer for this to happen:

let
  anon = Array{Any}(undef, 2)
  i=0
  for outer i = 1:2
    anon[i] = ()-> println(i)
    i += 1
  end

  anon[1]()
  anon[2]()
end
.....
3
3

The i += 1 is lowered into i = i + 1, which is an assignment, and creates an i in the local scope.

2 Likes

This is true, but we have to be honest that in the context of the example it’s still confusing, even with a few reads of the docs.

Knowing that the i + 1 boxed variable is what will be return by the anonymous function, instead of the value of i at the line the function was created on isn’t totally obvious.

And there is an actual practical switch between pass-by-reference and pass-by-value behavior happening in the background when constructing the anonymous function struct, depending on the scope of i - so it’s not a completely unrelated problem.

Anyway it’s probably difficult for a newcomer, even with the docs. It’s still confusing to me after registering a bunch of packages. Scoping rules are confusing, Julia has quite a history of demonstrating this.

3 Likes

Possibly. Personally, I don’t expect all aspects of programming to be “totally obvious”. Most interesting things in life aren’t.

Understanding scoping in any nontrivial language requires working through a few examples. Fortunately, the manual has quite a few of these now, but I am sure that more pertinent ones are always welcome as PRs.

The history of Julia’s scoping rules is indeed a bit meandering, but that is not really required for understanding how they work now. I don’t think that Julia’s scoping rules are especially confusing, but YMMV.

3 Likes

The i inside a closure is in the same scope as the i outside it so updates to i is visible inside the closure. The potentially confusing thing is that a closure can outlive the lexical scope where it was defined.

2 Likes

Is this what you wanted?

anon = Array{Any}(undef, 2)
for i = 1:2
  local k = i
  anon[i] = ()-> println(k)
end

It gives

julia> anon[1]()
1

julia> anon[2]()
2

julia>     

In the OP case, as other have already pointed out, I think the root of the problem lies in a very common misunderstanding about the scope of for loops.

A for block does not define a single scope, but a succession of scopes one for each iteration.

The distinction between a single scope and a succession of scopes is often little relevant (I believe this is the reason for it being a common misunderstanding). However, together with closures it become relevant and crashes a mental model in which all iterations shared the same inner scope.

I think it can be illustrated by:

julia> function f()
           for i = 1:5
               if @isdefined x
                   println("x found: $x")
               else
                   println("x not found")
               end
               x = i
           end
           return
       end
f (generic function with 1 method)

julia> f()
x not found
x not found
x not found
x not found
x not found
14 Likes

While this may be a subtle and common misunderstanding, I don’t think this is the root problem in this case. At least for me it wasn’t.

Both of the following would not meet OP expectation:

Succession of scopes one for each iteration:

anon = Array{Any}(undef, 2)
for i = 1:2
    anon[i] = ()-> println(i)
    i += 1
end
anon[1]()
anon[2]()

Single scope across all iterations:

anon = Array{Any}(undef, 2)
let
    local i
    for outer i = 1:2
        anon[i] = ()-> println(i)
        i += 1
    end
end
anon[1]()
anon[2]()

Rather, I think the key insight to help understanding in this case is:

anon = Array{Any}(undef, 2)
for i = 1:2
    let i = i
        anon[i] = ()-> println(i)
    end
    i += 1
end
anon[1]()
anon[2]()

Is my solution identical to yours?

I wasn’t trying to solve a problem with closures. I was searching for something related to for loop scope and I run into this particular example, which confused me.

This is a part of a bigger problem I have searching through Julia documentation. It would really help to have a BNF (or another formal notation) syntax of Julia published somewhere, otherwise searching for keywords like outer is futile.

3 Likes