Is it defined behavior to modify `iter` while `for`-looping over it?

[WalterMadelim]

@eldee

depending on how you define difference

If the local i used in the second pass associates to the same object to the i used in the first pass, then they are the same. (my understanding about julia is advanced again☺️)

julia> function test()
           v = Function[]
           for i = S(4)
               push!(v, () -> println("I'm ", i))
               map(x -> x(), v)
           end
       end
test (generic function with 1 method)

julia> test()
I'm [1]
I'm [1, 2]
I'm [1, 2]
I'm [1, 2, 3]
I'm [1, 2, 3]
I'm [1, 2, 3]
I'm [1, 2, 3, 4]
I'm [1, 2, 3, 4]
I'm [1, 2, 3, 4]
I'm [1, 2, 3, 4]

julia> function test()
           v = Function[]
           for i = 1:4
               push!(v, () -> println("I'm ", i))
               map(x -> x(), v)
           end
       end
test (generic function with 1 method)

julia> test()
I'm 1
I'm 1
I'm 2
I'm 1
I'm 2
I'm 3
I'm 1
I'm 2
I'm 3
I'm 4

[WalterMadelim]

Mutate a variable is mainly for performance (I guess).

Reusing a name (re-bind it to an updated object) is also useful. Although sometimes lead to unexpected results (e.g. capturing), I have to admit that the existing design has its rationale.

[WalterMadelim]

@foobar_lv2

julia> function foo(fun, iterable)
           for item in iter
              fun(item)
           end
           nothing
       end

julia> function bar(fun, iterable)
           next =  iterate(iterable)
           while next !== nothing
               item, state = next
               fun(item)
               next = iterate(iterable, state)
           end
           nothing
       end

For all remaining questions, you simply need to look up the implementations of the respective iterate functions.

@eldee Gave me a hint. I’m sorry but I’m more confused now.

I searched some implementations in the julia github but I fail to find an easy one, e.g. what is the implementation of iterate([2, 3, 4], 2) or iterate(1:3, 2)? (Forgive my poor searching ability)

From your transformation foo to bar here, I notice that item as a local variable within the while loop, lives until the while is quitted. Then how to explain the code in my #21 post? why is the older anonymous function not updated?

[eldee]

@WalterMadelim

I searched some implementations in the julia github but I fail to find an easy one, e.g. what is the implementation of iterate([2, 3, 4], 2) or iterate(1:3, 2)? (Forgive my poor searching ability)

You can use @which to find which method is called and where it is defined:

julia> @which iterate([2, 3, 4], 2)
iterate(A::Array, i)
     @ Base array.jl:902

Via @less you can view the source code in your terminal, or via @edit in an editor.

julia> @less iterate([2, 3, 4], 2)  # q to exit
iterate(A::Array, i=1) = (@inline; (i - 1)%UInt < length(A)%UInt ? (@inbounds A[i], i + 1) : nothing)

## Indexing: getindex ##

"""
    getindex(collection, key...)

Retrieve the value(s) stored at the given key or index within a collection. The syntax
`a[i,j,...]` is converted by the compiler to `getindex(a, i, j, ...)`.

See also [`get`](@ref), [`keys`](@ref), [`eachindex`](@ref).

# Examples
```jldoctest
julia> A = Dict("a" => 1, "b" => 2)
Dict{String, Int64} with 2 entries:
  "b" => 2
  "a" => 1

julia> getindex(A, "a")
1
(...)
-- More (33%) --

(The Discourse highlighting is a bit too eager here (in my opinion).) The first line of the output is what you’re looking for:

iterate(A::Array, i=1) = (@inline; (i - 1)%UInt < length(A)%UInt ? (@inbounds A[i], i + 1) : nothing)

There are also more advanced options like Cthulhu.jl or Debugger.jl.

[WalterMadelim]

I cannot figure out why the output isn’t “I’m 1, // 2, 2, // 3, 3, 3” as the item updates from 1 to 3.

julia> function test()
           iter = 1:3
           v = []
           next = iterate(iter)
           while next !== nothing
               item, state = next
               f = function()
                   println("I'm ", item)
               end
               push!(v, f)
               @info "item = $item"
               map(x -> x(), v)
               next = iterate(iter, state)
               end
       end;

julia> test()
[ Info: item = 1
I'm 1
[ Info: item = 2
I'm 1
I'm 2
[ Info: item = 3
I'm 1
I'm 2
I'm 3

I also don’t understand why in this situation there is no Core.Box, whereas if I shift f above for one line then Core.Box occurs.

(@info is merely for pretty printing, it is not relevant here and should be changed to println)

It’s very brain burning.

---

Why isn’t

        item, state = next
        f = function()
            println("I'm ", item)
        end

and

        f = function()
            println("I'm ", item)
        end
        item, state = next

THE SAME THING? As I wrote them inside a function. I thought they were the same.

[GunnarFarneback]

The appearance of Core.box is likely a manifestation of the infamous issue 15276.

[eldee]

@WalterMadelim

I cannot figure out why the output isn’t “I’m 1, // 2, 2, // 3, 3, 3” as the item updates from 1 to 3.

Regardless of whether item gets boxed, the value in the closure f should be consistent with the one from the current variable item. When in the second iteration you have item, state = next you create a new variable* called item (with value 2), but it is not the same as the old one (still with value 1). Within v you do still have access to the old one, via the (old) f, i.e. v[1].

This is the same behaviour as in the simpler

julia> function test_simple()
           iter = 1:3
           v = []  # stores the items
           next = iterate(iter)
           while next !== nothing
               item, state = next
               push!(v, item)
               next = iterate(iter, state)
           end
           v
       end;

julia> test_simple()
3-element Vector{Any}:
 1
 2
 3

---

In my example above the i were also different variables, but they all referred to the same Vector. In the loop this Vector was at various stages of construction, hence the different prints. But when you query all different is at the same time, as in collect you get the same Vector content.

---

Edit: * Due to scoping rules in loops. If you would manually unroll the loop as in

function test_unrolled()
    iter = 1:2
    v = []
    next = iterate(iter)
    item, state = next
    f = function()
        println("1: I'm ", item)
    end
    push!(v, f)
    @info "item = $item"
    map(f -> f(), v)
    next = iterate(iter, state)
    item, state = next
    f = function()
       println("2: I'm ", item)
    end
    push!(v, f)
    @info "item = $item"
    map(f -> f(), v)
    next = iterate(iter, state)
    item, state = next
    f = function()
        println("3: I'm ", item)
    end
    push!(v, f)
    @info "item = $item"
    map(f -> f(), v)
    next = iterate(iter, state)
end

you get different results

julia>  test_unrolled()
[ Info: item = 1
1: I'm 1
[ Info: item = 2
1: I'm 2
2: I'm 2
[ Info: item = 3
1: I'm 3
2: I'm 3
3: I'm 3

[eldee]

@eldee

you create a new variable called item (with value 2), but it is not the same as the old one (still with value 1).

While I still think this is correct, I’m now confused why something like item = -1 does not constitute a new variable as far as the closure is concerned:

julia> function test2()
           iter = 1:3
           v = []
           next = iterate(iter)
           while next !== nothing
               item, state = next
               push!(v, () -> println("I'm ", item)
               item = -1
               map(f -> f(), v)
               next = iterate(iter, state)
           end
       end;

julia> test2()
I'm -1
I'm -1
I'm -1
I'm -1
I'm -1
I'm -1

This still happens if you change the type of item, e.g. item = rand().
(This is what I would expect from my experience and is sort of the point of boxing in closures, but seems inconsistent with my explanation.)

---

Edit: I guess the explanation in the documentation about assignment in soft scope implies the different item within one iteration are the same variable, but the item across iterations are different variables. The first part sounds a bit contrary to a variable being a name associated to a value, though. Edit2 (after a night’s rest): This is not contrary at all: a variable is a name (not the currently bound value), so if you rebind it, it remains the same variable. This makes complete sense.

[sgaure]

Automatic boxing of variables sucks bigly. The reason is that the decision to box is taken by some heuristic early in the compile pipeline, during lowering, before types are known. That’s why boxed variables have unknown type, i.e. Core.Box is not parametrized by type, unlike Ref. This is also the reason why seemingly innocent changes to the code may affect boxing.

[WalterMadelim]

@eldee

I’m now confused

I somehow figured out this brain burning situation, thanks to odow in an other topic.

function test()
    v = []
    for k = 1:3 # 3 passes here
        # associate the pass-local name `item` to `-9`
        item = -9
        f = function()
            println("I'm ", item)
        end # `f` is following the pass-local _name_ `item`
        # re-bind the pass-local name `item` to `k` 
        item = k
        push!(v, f)
        # print the status (note that the latest status of `item` is `k`)
        map(x -> x(), v)
        # re-bind the pass-local name `item` to `-1`,
        # therefore `f` is updated accordingly
        item = -1
        # `item` is going to lose connection with us,
        # therefore the pass-local `f` will not be updated in the future
    end
end

test()

[WalterMadelim]

@eldee

@WalterMadelim

I have learned that i is local and different among passes.

I don’t think this is necessarily true

I think you’ve instilled some some horrible knowledge in my brain that I can ill digest considering my current ability. But whatever , possibly I would never encounter that design in my practical code.

the i in for i = iter is merely a single-pass-local name, so they are essentially different names. (Despite the fact that they can all bind to the same object somewhere else).

---

I guess probably the real meaning in your #20 post is this process

julia> function process()
           v = Vector{Int}[]
           a = [1, 2, 3]
           let item = circshift!(a, 1) # item in the first pass
               push!(v, item)
           end
           println(v)
           let item = circshift!(a, 1) # item in the second pass
               push!(v, item)
           end
           println(v)
           let item = circshift!(a, 1) # item in the second pass
               push!(v, item)
           end
           println(v)
       end

julia> process()
[[3, 1, 2]]
[[2, 3, 1], [2, 3, 1]]
[[1, 2, 3], [1, 2, 3], [1, 2, 3]]

We notice that the elements of v are correlated.
By start contrast, the following is NOT correlated.

julia> function process()
           v = Vector{Int}[]
           a = [1, 2, 3]
           a = let item = circshift(a, 1) # item in the first pass
               push!(v, item)
               item
           end
           println(v)
           a = let item = circshift(a, 1) # item in the second pass
               push!(v, item)
               item
           end
           println(v)
           a = let item = circshift(a, 1) # item in the second pass
               push!(v, item)
               item
           end
           println(v)
       end
process (generic function with 1 method)

julia> process()
[[3, 1, 2]]
[[3, 1, 2], [2, 3, 1]]
[[3, 1, 2], [2, 3, 1], [1, 2, 3]]

[WalterMadelim]

Even equivalent change simple as this

function test()
    () -> a
    a
end;
@code_warntype test() # No ::Core.Box

function test() 
    () -> a
    local a
end;
@code_warntype test() # Has ::Core.Box

Are they equivalent? I’m not sure, regardless.

Different: 1. scope of a—the first is nondeterministic, the second is deterministic. 2. the first returns a, the second returns nothing.

Anyway, stop focusing on these…

[WalterMadelim]

@sgaure

That’s fine. As long as you don’t mutate snap (with something like shot!(snap, timestamp)), but just reassign it to a new object, there is no problem.

I re-thought about this problem.

In my original code, there is a main task that updates snap (via snap = k_th_NEW_OBJECT) iteratively, while at the same time distribute snap to other many subproblem tasks via
@spawn subproblem_duty(j, snap).

So this is a one-thread write, multiple other threads read async program.

The disquieting fact is that while I think snap = k_th_NEW_OBJECT is just a re-labeling action which can be finished instantly. (So I didn’t even use any locks.) The real underlying code julia execute is actually a mutation
Core.setfield!(snap, :contents, k_th_NEW_OBJECT).

So my concept about re-labeling turns out to be an illusion.

Nevertheless, the upside is: when I did a test to see if the other reading threads can receive false data (which is theoretically possible), I do not see any false data. In other words, my worry is not occurring in practice.

import Random.seed! as seed!
N::Int = 999
function gen(cnt)
    if cnt % 2 == 1
        seed!(1)
        a = rand(Int128, N, N);
        b = rand(Int128, N, N);
        c = rand(Int128, N, N);
        d = rand(Int128, N, N);
        e = rand(Int128, N, N);
        f = rand(Int128, N, N);
        return (; a, b, c, d, e, f)
    else
        seed!(7)
        a = rand(Int128, N, N);
        b = rand(Int128, N, N);
        c = rand(Int128, N, N);
        d = rand(Int128, N, N);
        e = rand(Int128, N, N);
        f = rand(Int128, N, N);
        return (; a, b, c, d, e, f)
    end
end;
function check(s)
    if s == gen(0)
        println(0)
    elseif s == gen(1)
        println(1)
    else # I cannot see this in practice
        println("\tEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE")
    end
end;

function test(m)
    for c = 1:10000000
        for j = 1:14
            Threads.@spawn check(m)
        end
        println("c=$c")
        m = gen(c) #  Core.setfield!(m, :contents, %51)
    end
end

test(gen(0))