Keyword behaves differently using `;` or `,`

Is this intended? It had @ccoffrin and I quite confused (not the subtle difference between , and ;):

julia> sum(0.0 for i in []; init = 0.0)
0.0

julia> sum(0.0 for i in [], init = 0.0)
ERROR: ArgumentError: reducing over an empty collection is not allowed
Stacktrace:
  [1] _empty_reduce_error()
    @ Base ./reduce.jl:299
  [2] mapreduce_empty(f::Function, op::Base.BottomRF{typeof(Base.add_sum)}, T::Type)
    @ Base ./reduce.jl:342
  [3] reduce_empty(op::Base.MappingRF{var"#36#37", Base.BottomRF{typeof(Base.add_sum)}}, #unused#::Type{Tuple{Any, Float64}})
    @ Base ./reduce.jl:329
  [4] reduce_empty_iter
    @ ./reduce.jl:355 [inlined]
  [5] reduce_empty_iter
    @ ./reduce.jl:354 [inlined]
  [6] foldl_impl
    @ ./reduce.jl:49 [inlined]
  [7] mapfoldl_impl
    @ ./reduce.jl:44 [inlined]
  [8] #mapfoldl#214
    @ ./reduce.jl:160 [inlined]
  [9] mapfoldl
    @ ./reduce.jl:160 [inlined]
 [10] #mapreduce#218
    @ ./reduce.jl:287 [inlined]
 [11] mapreduce
    @ ./reduce.jl:287 [inlined]
 [12] #sum#221
    @ ./reduce.jl:501 [inlined]
 [13] sum
    @ ./reduce.jl:501 [inlined]
 [14] #sum#222
    @ ./reduce.jl:528 [inlined]
 [15] sum(a::Base.Generator{Base.Iterators.ProductIterator{Tuple{Vector{Any}, Float64}}, var"#36#37"})
    @ Base ./reduce.jl:528
 [16] top-level scope
    @ REPL[36]:1

I guess the answer is “yes”, but it is sure confusing:

julia> f() = sum(0.0 for i in [], init = 0.0)
f (generic function with 1 method)

julia> g() = sum(0.0 for i in []; init = 0.0)
g (generic function with 1 method)

julia> @code_lowered f()
CodeInfo(
1 ─      #9 = %new(Main.:(var"#9#10"))
│   %2 = #9
│   %3 = Base.vect()
│   %4 = Base.product(%3, 0.0)
│   %5 = Base.Generator(%2, %4)
│   %6 = Main.sum(%5)
└──      return %6
)

julia> @code_lowered g()
CodeInfo(
1 ─       #11 = %new(Main.:(var"#11#12"))
│   %2  = #11
│   %3  = Base.vect()
│   %4  = Base.Generator(%2, %3)
│   %5  = (:init,)
│   %6  = Core.apply_type(Core.NamedTuple, %5)
│   %7  = Core.tuple(0.0)
│   %8  = (%6)(%7)
│   %9  = Core.kwfunc(Main.sum)
│   %10 = (%9)(%8, Main.sum, %4)
└──       return %10

These both work so it’s about generator parsing I think:

julia> sum((0.0 for i in []); init = 0.0)
0.0

julia> sum((0.0 for i in []), init = 0.0)
0.0
2 Likes

Yeah, the one with the comma is equivalent to

sum(0.0 for i in [], init in 0.0)

Perhaps if numbers were not iterable then the error message would be better.

5 Likes

Ah actually the keyword in the second example is parsed as a third arg of the generator expression, which seems to be ignored, so the whole thing acts as if no keyword had been set. That’s why this works as well sum(0.0 for i in [], init = 0.0; init = 0.0), so the generator doesn’t care about this syntax appendage. I would think this is a parsing bug then.

julia> Meta.@dump sum(0.0 for i in [], init = 0.0)
Expr
  head: Symbol call
  args: Array{Any}((2,))
    1: Symbol sum
    2: Expr
      head: Symbol generator
      args: Array{Any}((3,))
        1: Float64 0.0
        2: Expr
          head: Symbol =
          args: Array{Any}((2,))
            1: Symbol i
            2: Expr
              head: Symbol vect
              args: Array{Any}((0,))
        3: Expr
          head: Symbol =
          args: Array{Any}((2,))
            1: Symbol init
            2: Float64 0.0

I’m squarely in the field “always use semicolon for keyword arguments and forget about those nasty edge cases where a comma would be a problem”.

21 Likes

We do get a better error message if the first iterator in the generator expression is not empty:

julia> struct A end

julia> sum((i, j) for i in [1, 2], j = A())
ERROR: MethodError: no method matching iterate(::A)

But if the first iterator is empty, then we still get the original error message:

julia> sum((i, j) for i in [], j = A())
ERROR: ArgumentError: reducing over an empty collection is not allowed

Python:

In [2]: sum(1 for _ in [], start=0)
  File "<ipython-input-2-646ca842f927>", line 1
    sum(1 for _ in [], start=0)
        ^
SyntaxError: Generator expression must be parenthesized

In [3]: sum((1 for _ in []), start=0)
Out[3]: 0
3 Likes

I once honestly thought it’s a bug in sum, and created an issue: Sum with init doesn't work for empty iterable · Issue #39669 · JuliaLang/julia · GitHub.
Turns out this is just due to the lack of ; or () indeed.

I am curious about these nasty edge cases.

I told you, I forgot what they are because I don’t run into them anymore :slightly_smiling_face:

But for example the syntax f(; a) for f(; a=a) works only if you explicitly use the semicolon. Or also slurping all the keyword arguments, f(x; kwargs...), works only with semicolon and not the colon. There are probably other things that wouldn’t work but I just don’t remember them. I’m pretty sure I already ran into the issue reported at the top of this thread, before stopping using the comma. My suggestion is: always use the semicolon to start the list of keyword arguments. By “always” I mean literally always, also when there are no positional arguments.

2 Likes

But apparently only in that case, because this works without parentheses:

>>> sum(1 for i in [1,2,3])
3

But I’m not sure why Python requires parentheses when keyword arguments are involved, because the language doesn’t seem to allow multidimensional generators like Julia does:

>>> (1 for i in [1,2,3], j in [1,2])
  File "<stdin>", line 1
    (1 for i in [1,2,3], j in [1,2])
                       ^
SyntaxError: invalid syntax

so it seems like there is no ambiguity to resolve by requiring parentheses.

The keyword is optional so it is disambiguating between

sum((x for x in xs), y)
sum(x for x in (xs, y))
1 Like

Thanks. I am also in the habit of always using the colon, albeit for the more superficial reason that doing so forces me to remind myself of how I defined my functions and keep them consistent.