Referencing local variable before assignment results in unexpected behavior

I don’t think this is accurate. I can at least suggest one alternative lowering (which is more or less how FLoops.jl works). Also, current do is not completely non-disruptive since it changes how return/continue/break/@goto works/not work; although you probably meant non-disruptive in terms of scoping (since that’s what we are discussing). But let me shoehorn a “fix” for this, too, anyway. Consider:

s = 0
foreach(v) do x
    s += x
    s > 0 && return s  # this is now `return` in outer scope
end

We can lower this to

# Defined in `Core`:
struct Return{T} <: Terminate
    value::T
end

struct C{X}
    x::X
end

function (c::C)(x)
    s = c.s
    s += x
    s > 0 && return Return(s)
    return C(s)
end

s = 0
c = foreach(C(s), v)
if c isa Return
    return c.value
end
s = c.s

To support this, function that is intended to be used with the do-block must follow the calling convention c = c(args...); c isa Terminate && return c; i.e., foreach is

function foreach(c, xs)
    for x in xs
        c = c(x)
        c isa Terminate && return c
    end
    return c
end

Likewise, we can define Break <: Terminate, Continue <: Terminate and Goto <: Terminate (how foreach should handle Break and Continue is a bit of a discussion). Undef variables can also be handled using a special sentinel type.

The definition of open is unchanged:

function open(c, name)  # ignoring options
    f = open(name)
    try
        return c(f)
    finally
        close(f)
    end
end

Actually, it may be better to lower f(args.....) do to (say) __do__(f, args..) so that we won’t accidentally invoke incompatible functions.

Also, relying on the users to correctly invoke c = c(x); c isa Terminate && return c may not be the best API. But we can easily put a syntax sugar on top of the functional API to make it hard to mis-define the API. This is pretty much how FGenerators.jl defines foldl using @yield.

A possible syntax sugar (not important)

For example, we can have a macro @do such that

@do foreach(xs) begin
    for x in xs
        @yield x
    end
end

lowers to

foreach(f, xs) = __do__(foreach, Stateless(f), xs)  # "legacy" API
function __do__(::typeof(foreach), c, xs)
    for x in xs
        c = c(x)
        c isa Terminate && return c
    end
    return c
end

where Stateless is an appropriate callable wrapper:

struct Stateless{F}
    f::F
end

function (c::Stateless)(args...)
    c.f(args...)
    return c
end
1 Like

Your proposed lowering only really works if the closure is called right away in the enclosing scope though. It doesn’t really address the case of how to handle something like:

julia> function f()
           x = 0
           return () -> x += 1, () -> x -= 1
       end
f (generic function with 1 method)

julia> p, m = f()
(var"#9#11"(Core.Box(0)), var"#10#12"(Core.Box(0)))

julia> p()
1

julia> p()
2

julia> m()
1
1 Like

Scala does something like this (and they are moving away from it, because it turned out to be a bad idea, cf Deprecated: Nonlocal Returns). Please don’t.

Higher order functions may pass their argument off to other higher order functions. In other words, you need to setjump / longjump. A nonlocal return would unwind the stack until we hit the return stored in the closure instance (because your function could be reentrant). But then, lots of intermediate functions might have badly written exception handlers that intercept the “NonLocalReturn(stackAddr::Ptr{Nothing})” exception, even though their stackpointer is not identical to the one stored in the exception. Julia ecosystem exception handling is just not robust enough to use as routine control flow.

Also, what do you do when the closure outlives the stackframe that allocated it?

foreach will never ever behave the same as for. You can attempt to minimize the differences (e.g. the way we currently implement closures), but you cannot erase them while supporting return / continue / break / goto.

So what? Users mostly don’t and shouldn’t have to think about stack frames. You should be able to reason about what code does without knowing about stack frames or registers or any of that stuff. You may need to have a decent model of it in your head to reason about performance but never about behavior. Not only are stack frames an implementation detail that should not be forced on the user, they’re not even a fixed property of the implementation — it’s entirely possible for the compiler to inline the closure and completely eliminate the additional stack frame. Should we then change the behavior of the language because the implementation changed? Or should we force there to be a stack frame there because we’ve made the stack frame a semantically significant part of the language? Allowing this kind of implementation detail to bleed into the semantics makes for an abstraction that’s harder to use, and tends to prevent future optimizations: the implementation of a good abstraction can be changed without affecting behavior, but leaky abstractions lock you into specific implementations.

I’ve also considered something like this but it seems like a dangerous path to go down and the fact that Scala has come to regret it suggests that it’s not great. Ruby also does weird stuff to try to eliminate this difference and it seems too complex and confusing to be worth it. Yes, this is a point where the for-foreach equivalence breaks down, but the cures all seem to be worse than the disease.

10 Likes

That’s a good point. I should’ve emphasised that the hypothetical lowering works only for the do block and only if we restrict its semantics. The closure/inner function can have more unrestricted semantics.

I think restricting semantics help readability (observation: Python’s with) and the optimizer (no Box’ing by construction; see also Rust’s FnOnce).

This is not true. CPS composes well. You just need to stick with the calling convention. If you are curious, see FGenerators.@yieldfrom.

Can’t we learn an alternative lesson? That is to say, the semantics of “less disruptive do” has to be reasonably restricted to make it less dangerous. As a limit of simplicity, Python’s with seems to be doing a good job (the lowering I discussed + restriction for calling the block only once will provide an equivalent API). Expanding the semantics slightly to support foreach-like construct may still work (as it’s essentially a (dual of) generator). But I do agree it’s non-trivial to design this API.

Anyway, my main point was not return support. Even without it, I think my lowering can function as a counterexample of

1 Like

I kinda disagree with this, strongly. I mean, users of course should not need to care too much about inlining / tail-calls / which stackframes get materialized. But the notion of “function call” / “return” / “callstack” are pretty fundamental to how most people think about code, and trying to shield users from these concepts or muddying the waters between them looks super misguided to me. It is not just an “implementation detail” to know where in the callstack you are (“semantic callstack”, i.e. what you get with debug info that knows provenance of inlined instructions, not raw stack – that would indeed be an implementation detail with respect to current compiler optimization choices).

If one considers the concept of “function” as fundamental, then for x in v ... and foreach(v) do x; ... are fundamentally different, even if they might compile to the same machine code in some cases. (but hopefully our debug info retains the difference when a stacktrace is taken)

If one took that standpoint (concept of function / callstack are fundamental), then it becomes a question of what side of the trade-off we like more: Explicit Ref makes code much clearer, but imposes a modest amount of boilerplate, and might seem weird to some people.

Yes, there exist calling conventions / language restrictions where this might compose well. General purpose julia calling convention and general purpose julia functions do not adhere to the restrictions you need to make this work.

Yep, we have very different philosophies on exposing aspects of the implementation to users. Which is, of course, totally fine — opinions on these things are allowed to differ.

Explicit Ref makes code much clearer, but imposes a modest amount of boilerplate, and might seem weird to some people.

The thing is people can already write the explicit Ref version, we just don’t force them to do it that way. So the status quo is that people can modify bindings from closures but it might have a performance penalty which may go away in the future and which can be worked around by using a typed Ref. The alternative you’re proposing is to completely disallow modifying bindings from closures and force people to always write the Ref version because we can’t, at the moment, always optimize it well. The status quo is strictly more flexible and nothing is preventing you from only writing code that uses Ref to modify outer locals. Moreover, if the worry is that people will do this without noticing that they’re doing it, it would be easy to write a linter check (and add it to, say, VS Code) to find places where people modify a closed over binding and warn them about the potential performance issue.

7 Likes

Just want to add that local functions can be nice to use as a self-documenting code feature, and being able to just stick things inside local functions and have it execute the same without rewriting them is convenient.

What’s really nice about Julia, as a user, is the ability to do lots of things that are easy to write and read but not necessarily high performance. But then when you want to optimize a section, transforming the code to improve performance is rather easy.

Being forced to rewrite everything just because you want to switch between a loop and a foreach()/do sounds like kind of a headache. Same as the headache that used to exist when testing code on the REPL and in a module.

function process1(x)
    x *= -1
    x += 2
    x *= 5
end

# Should produce the same result (and currently does) as

function process2(x)
    function step1()
        x *= -1
    end
    function step2()
        x += 2
    end
    function step3()
        x *= 5
    end

    step1()
    step2()
    step3()
end
4 Likes

This is a side issue for the purposes of the main discussion, but I don’t think that the iterator interface requires a functional implementation.

While most iterator implementations have a functional iterate, keeping (part of the) state in the iterator is allowed and conforming. Eg iterators that wrap IO are implicitly stateful, eg Base.EachLine.

1 Like

Or, your know, Base.Iterators.Stateful that is an wrapper type that makes any iterator stateful.

Fwiw, changing the binding in a closure is a relatively dangerous thing to do, so it would not be so bad and could actually prevent some bugs to require such uses to declare variables specifically for that purpose (eg requiring them to be marked specifically as local).

The performance bug is a very nasty one that appears unlikely to ever go away completely, and just its nature of always possibly lurking in the corner prevents users (certainly me at least) from using closures (higher order functions, comprehensions and @threads) anywhere near performance critical code, simply out of (possibly misplaced) fear.

3 Likes

I am not sure why you think this. Conceptually, it is not different from changing a value of a field in a mutable struct.

1 Like

Dangerous might be a bit strong, but it’s unusual (at least for me), might differ from other languages and can be the source of hard-to-understand bugs.

Sorry, I still don’t understand. Almost every feature of a language differs from some other language.

That said, as explaned by @StefanKarpinski above, Julia’s closures are really the plain vanilla closures you find in almost every language that is part of or derives from the Lisp family. Really no secret sauce here.

Sure, this applies to mutable state in general. Best to avoid, but sometimes you do need it. Same issue as struct vs mutable struct.

Sure, but I would imagine most people using Julia do not come from a functional programming background and have never heard of lisp.

It is not the same as mutable structures because it interferes with scoping issues (is assigning to a variable creating a new variable or is it assigning to a previously-defined variable?), and scoping issues are notoriously confusing, as the OP shows.

2 Likes

This is interesting, because just recently I’ve used this feature and I was so thankful, that it exist.

The task was to parse contents of some zip file, which on its own was a mess of nested subdirectories and some of those had files which I needed. So I did something like this

function parsezip(f, zipfile)
# some logic which opens file, go through the nested file tree 
# and finally catch necessary file
...
  x = unpack_contents(file)
  f(x)
# some other logic which process errors if they happened and finally gracefully close the file
end

Now, in my actual processing part I did something like this

function process(zipfile)
  v = [] # surely it wasn't `Any[]`, but it's not important here
  parsezip(zipfile) do x
    # some logic to convert contents to DataFrame and postprocessing
    ....
    data = .... # finally I get my data
    push!(v, data)
  end
  return v
end

And that’s it! Logic of traversing zipfile and data content parsing was separated, I get what I need and it was totally aligned with my mental model of what is happening. Probably one can do it better (write iterators or forcing to pass parameters or something else), but the fact that it worked as it is was really great.

4 Likes

I would not expect people who aren’t used to functional programming to define inner functions at all though? You can’t do that in C/C++, for example. What non-functional programming language are people coming from that they expect to be able to define inner functions but those inner functions aren’t closures?

4 Likes

Isn’t that what the OP is about, that you can’t do that in python?

@Skoffer I completely agree, it’s an awesome feature and I use it a lot too (for accumulating results and for tracking state in an iterative algorithm). But, on your example, you might want to have/introduce for instance a data variable in the outer function, and that would result in unexpected bugs. The v variable is “special” and it would make sense (and actually help code readability) to mark it as such. Typically this is the sort of things you want to put in comments anyway.

So, if I understand correctly, the pros are 1) clarity by explicitness 2) avoids bugs 3) remove a major source of performance unpredictability. The downsides are 1) it’s breaking 2) it uses up special-purpose syntax 3) it makes it very slightly harder to implement this kind of function (not that bad: you write the function, it raises an error, you add the keyword) 4) it makes foreach and for behave differently. Tradeoffs, always tradeoffs!

1 Like

Please recall that you were talking about

so, presumably, the scoping issues are already out of the way since you know what you are changing, specifically.

That is neither here or there, you were talking about how Julia’s implementation

and it doesn’t, really.

Again, I might be missing something, but isn’t the OP precisely about how julia differs from python?

I don’t know if scoping is the right word, but there’s no question that this issue is more confusing than simply mutation, because there are several rules that govern what variables are used / created, and so unless you know the rules very well there’s potential for confusion: for instance, for loops do not reuse the variable in the outer scope.

Also I’m not sure if you intend it or not but your posts are quite personal and aggressive, there’s no need for that.

1 Like