# ANN: Underscores.jl: Placeholder syntax for closures

**URL:** <https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482>\
**Category:** Package Announcements\
**Tags:** data, syntax\
**Created:** [March 25, 2020, 10:47am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482 "2020-03-25T10:47:13Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 26, 2020, 9:16pm UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/21 "2020-03-26T21:16:11Z")

</div>

I think this is interesting (and I do think Observables could do with nicer syntax!) though this seems like a special enough case that Observables should just define its own special purpose macro.

For Observables you need to distinguish between variables which are `Observable` vs those which are not (for the purposes of creating the closure). This is a kind of “escaping” so it might be natural to use `$x` syntax. For example:

```julia
x = 1
a, b = Observable(1), Observable(2)
c = @Observable (a+b)*a + $x

```

Alternatively, if you think most variables will be captured (rather than form arguments to the closure) you could just reuse the existing syntax for observable getindex, but automatically lift it out into a closure:

```julia
c = @Observable (a[] + b[])*a[] + x

```

A third option would be to just lift the names of the things being observed out of a closure argument list to also take them from the current scope. This is general and the syntax is very evocative:

```julia
c = @Observable (a,b) -> (a + b)*a + x

```

You could even allow a combination of these forms, for example the first for short one-liners, and the third for more general cases

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 26, 2020, 10:12pm UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/22 "2020-03-26T22:12:26Z")

</div>

Does it make sense to lower

```julia
@_ mapreduce(f, _, xs) do x, y
    x + y
end

```

to

```julia
mapreduce(f, (x, y) -> x + y, xs)

```

and

```julia
@_ reduce(op, Filter(_), xs) do x
    x > 0
end

```

to

```julia
reduce(op, Filter(x -> x > 0), xs)

```

?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 12:20am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/23 "2020-03-27T00:20:07Z")

</div>

Oh right, `do` syntax for arbitrary slots; I hadn’t considered that, that’s a _really_ cool idea!

Now that you’ve mentioned it, I really want to do something like this though I fear we can’t use `_` because it changes the meaning of `_` in a weird way. It also disallows using `_` for it’s “normal” meaning which could be really handy for higher order functions which accept multiple closures.

So I think we need a different marker. An ugly but straightforward possibility is just to use `_do`:

```julia
@_ mapreduce(2^_, _do, xs) do x, y
    x + y
end

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 1:45am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/24 "2020-03-27T01:45:30Z")

</div>

> [@haberdashPI](#):
>
> Just my two cents: can’t say I love `_` meaning only the first argument. This may just be that I am most familiar with the syntax from Scala, where each `_` is replaced with a distinct argument.

It’s interesting that there’s several precedents here which are quite different. In yet another case, Elixir uses `&`, [Kernel.SpecialForms — Elixir v1.15.7](https://hexdocs.pm/elixir/Kernel.SpecialForms.html#&/1-anonymous-functions), to get things like `&(&1 + &2)`. Amusingly / surprisingly this syntax is available in Julia (though it may eventually be used for `Ref`?). In some ways I find `&1` more visually pleasing than `_1` but I do think the “outer” placement of the `@_` is quite nice.

Mathematica uses `#` or `#1` for our `_` and `_1`, but it’s arguably a bit of a different case if the vast majority of Mathematica users use it via the standard UI where `#` is highlighted and in a different font [Functional Operations—Wolfram Language Documentation](https://reference.wolfram.com/language/tutorial/FunctionalOperations.html#17469)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 27, 2020, 2:00am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/25 "2020-03-27T02:00:35Z")

</div>

I was thinking “two step” lowering which (conceptually) transforms above pieces of code to

```julia
@_ ans = mapreduce(f, _, xs)
ans() do x, y
    x + y
end

@_ ans = reduce(op, Filter(_), xs)
ans() do x
    x > 0
end

```

and then to

```julia
ans = _1 -> mapreduce(f, _1, xs)
ans() do x, y
    x + y
end

ans = _1 -> reduce(op, Filter(_1), xs)
ans() do x
    x > 0
end

```

The last step already works so I guess the fishy part is the first one? That is to say, inserting `ans =` alters the boundary of the implicit anonymous function?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 2:21am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/26 "2020-03-27T02:21:43Z")

</div>

> [@tkf](#):
>
> The last step already works so I guess the fishy part is the first one? That is to say, inserting `ans =` alters the boundary of the implicit anonymous function?

Exactly. Though `__` is actually consistent in this two-step scheme. So maybe it would be fine to have

```julia
@_ mapreduce(2^_, __, xs) do x, y
    x + y
end

```

I think this might be correct! Is there anything else this could mean?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 27, 2020, 2:30am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/27 "2020-03-27T02:30:57Z")

</div>

Nice! So, it’d mean that `do` block acts like `|>` and delimits the range of “whole expression?”

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 3:35am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/28 "2020-03-27T03:35:31Z")

</div>

> [@tkf](#):
>
> So, it’d mean that `do` block acts like `|>` and delimits the range of “whole expression?”

Yes that’s a good way to state it! The way `Expr(:do)` is parsed this is also very easy and natural to implement with the `Expr(:call)` nesting inside the `Expr(:do)` in the AST.

> Is there anything else this could mean?

Thinking about it, there’s a choice, but the other alternative is relatively useless:

```julia
@_ map(__, xs) do x
    x+y
end

```

Could be the current meaning:

```julia
__1 -> (map(__1, xs) do x
    x+y
end)

```

but this is only really helpful inside piping or composition, IMO and one wouldn’t use a `do` block in conjunction with that. (`__` can also be used to create an anonymous function but that’s not the core use case for `@_` as I see it.)

Compared this to the more useful meaning:

```julia
f = (__1 -> map(__1, xs))
f() do x
    x+y
end

```

This second one seems fine, though there’s one oddity which is the need to insert an extra `Expr(:call)` (the `()` in `f()`). But I think that’s ok. Also `__2` can’t mean something sensible in this case, but that’s probably not a big deal, I’ll just make it an error.

I think in practice I’ll lower this slightly differently to avoid creating another nested closure:

```julia
let __1 = (x,)->begin # The closure from `Expr(:do)`
        x+y
    end
    map(__1, xs)
end

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 4:22am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/29 "2020-03-27T04:22:38Z")

</div>

Here’s a WIP implementation of do block support: [Allow \_\_ to stand for the argument which accepts a do block by c42f · Pull Request #4 · c42f/Underscores.jl · GitHub](https://github.com/c42f/Underscores.jl/pull/4)

I know you’ve thought a lot about `mapreduce` and related functions. Do you have a compelling example of how this would be used somewhere which I could put in the docs?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 27, 2020, 4:22am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/30 "2020-03-27T04:22:48Z")

</div>

> [@c42f](#):
>
> Could be the current meaning:
> 
> ```julia
> __1 -> (map(__1, xs) do x
> x+y
> end)
> 
> ```
> 
> but this is only really helpful inside piping or composition, IMO and one wouldn’t use a `do` block in conjunction with that.

Well… I do sometimes 😅. But can’t it be done with single `_`? Something like

```julia
@_ itr |> 
   filter(_) do x
       x > 0
   end |>
   map(_) do x
       x^2
   end

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 5:39am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/31 "2020-03-27T05:39:13Z")

</div>

> [@tkf](#):
>
> Well… I do sometimes 😅. But can’t it be done with single `_` ? Something like
> 
> ```julia
> @_ itr |> 
> filter(_) do x
> x > 0
> end |>
> map(_) do x
> x^2
> end
> 
> ```

Haha, I take it back 🙂

But no, that example doesn’t seem like it will work very nicely if you also want `mapreduce(_+1, __, xs) do ... end` to work.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 27, 2020, 5:55am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/32 "2020-03-27T05:55:37Z")

</div>

So what I’ve written was equivalent to

```julia
itr |> 
filter(identity) do x
    x > 0
end |>
map(identity) do x
    x^2
end

```

?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 6:01am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/33 "2020-03-27T06:01:57Z")

</div>

Yes. The rule is that the boundary for `_` is inside the “outer” call, where piping (and `do` in the PR) doesn’t count for the purposes of determining “outerness”

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 6:08am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/34 "2020-03-27T06:08:24Z")

</div>

So if you want this to work, we’d need a separate marker for `do` so you can have `__` mean the current thing and have this work

```julia
@_ itr |> 
   filter(__) do x
       x > 0
   end |>
   map(__) do x
       x^2
   end

```

then, unrelatedly have a `_do` marker for

```julia
@_ mapreduce(_+1, _do, xs) do x,y
    x + y
end

```

which we could compose to get

```julia
@_ itr |> 
   filter(__) do x
       x > 0
   end |>
   mapreduce(_+1, _do, __) do x
       x + y
   end

```

It’s getting complicated 😬

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 27, 2020, 6:26am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/35 "2020-03-27T06:26:49Z")

</div>

Actually I think not being able to use `do` with `|>` is OK 🙂 I was just wondering what the rule is. I was probably mixing it with some other proposals.

So how do you apply the rules to lower `@_ map("X $(repeat(_,_))", ["a","b","c"], [1,2,3])` in the documentation to `map((a, b) -> "X $(repeat(a, b))", ["a","b","c"], [1,2,3])` and not `map("X $((a, b) -> repeat(a, b))", ["a","b","c"], [1,2,3])` or `map("X $(repeat(identity, identity))", ["a","b","c"], [1,2,3])`?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 6:34am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/36 "2020-03-27T06:34:42Z")

</div>

> [@tkf](#):
>
> `@_ map("X $(repeat(_,_))", ["a","b","c"], [1,2,3])`

This is no longer a valid example :-/ Read the dev docs rather than stable docs — people further up the thread have (mostly) persuaded me that `_` always means `_1` and that’s what’s on master.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 6:37am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/37 "2020-03-27T06:37:04Z")

</div>

> [@tkf](#):
>
> Actually I think not being able to use `do` with `|>` is OK

Hmm. The need to add an extra `Expr(:call)` implicitly in the more specialized lowering of `do` does make me worry that the most convenient interpretation of `__` may not be entirely consistent.

---

<div class="post-metadata">

**Author:** ![haberdashPI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/haberdashpi/32/26337_2.png) [@haberdashPI](https://discourse.julialang.org/u/haberdashPI)\
**Post date:** [March 27, 2020, 2:12pm UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/38 "2020-03-27T14:12:01Z")

</div>

After thinking about it for longer, I actually am disagreeing with myself now. In particular, if `_` indicates a different argument upon each use, then one might expect ` __` to have this same property; it seems clear you wouldn’t want that to happen to `__ `. On that front does it make sense to have ` __1`, `__ 2`, etc…???

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 27, 2020, 9:01pm UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/39 "2020-03-27T21:01:49Z")

</div>

> [@haberdashPI](#):
>
> On that front does it make sense to have ` __1` , `__ 2` , etc

Yes, this was actually implemented from the start 🙂 If you look at the implementation you’ll see it’s the exact same code which lowers `_` and `__` (and associated numbered versions) — it’s just some regexes which differ and the part of the expression that they’re applied to. So it’s all quite uniform and things like `@___₁ ^ __₁` work.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 28, 2020, 1:10am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/40 "2020-03-28T01:10:10Z")

</div>

@c42f Thanks, this is a cool package! I just [commented on](https://github.com/JuliaLang/julia/pull/24990#issuecomment-605364343) the lengthy discussion of PR #24990, so I don’t want to duplicate everything I said there. But basically, I like this approach, except for the double underscores. I feel like the double underscore syntax adds complexity to the syntax in order to solve a problem not directly related to syntax for closures. Namely, the base pipe simply isn’t powerful enough. Syntax like `data |> d -> map(x -> x.f, d)` demonstrates that we need better pipes.

It would be great if we had

```julia
x |> f(a, b) # translated to f(x, a, b)

```

and

```julia
x |>> f(a, b) # translated to f(a, b, x)

```

Then your example would be written as

```julia
@_ t |>>
    filter(_.age > 27) |>>
    group(_.sex) |>>
    map(length)

```

Would it be possible to add a new macro (or update the `@_` macro) to support the more advanced pipes, and thus support the syntax shown in the pipeline above?

(I’m guessing that a symbol other than `|>>` would have to be found, since `|>>` doesn’t parse as a binary operator.)

[Previous page](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482.md?page=1)

[Next page](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482.md?page=3)