# ANN: Underscores.jl: Placeholder syntax for closures

**URL:** <https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482>\
**Category:** Package Announcements\
**Tags:** data, syntax\
**Created:** [March 25, 2020, 10:47am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482 "2020-03-25T10:47:13Z")\
**Posts on this page:** 10\
**Page:** 3

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 28, 2020, 2:55am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/41 "2020-03-28T02:55:24Z")

</div>

Thanks. My current preference to solve this problem would be to have `filter(x->x.age > 27)` return a lazy `Filter` object. (I also replied on the issue to elaborate on this.)

The ` __` syntax is a concession to the fact that all the `Base` functions are eager by default, and that this is generally the convention in the Julia ecosystem as a whole. Having syntax for this adds complexity, but I feel it’s necessary complexity. For example, how would you add `sort` to your data pipeline? With `__ ` it’s really easy:

```julia
@_ people |>
   filter(_.age > 27, __) |>
   sort(__, by=_.height)

```

Without `__` we would effectively be saying “oh no you generally can’t use functions in your data pipeline unless they’re designed for use in a data pipeline”.

(By the way, thinking about `sort` was the use case which convinced me that `__` was actually a necessity for composability with arbitrary library functions.)

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 28, 2020, 3:51am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/42 "2020-03-28T03:51:10Z")

</div>

Actually, I think my proposal for pipes along with the single underscore usage of your macro works just fine with `sort`:

```julia
@_ people |>>
    filter(_.age > 27) |>
    sort(by=_.height)

```

**Explanation**

If we assume that the `|>` and `|>>` operations are re-written by the parser, the following would happen. First, your macro expands the underscores to this:

```julia
people |>>
    filter(x -> x.age > 27) |>
    sort(by = x -> x.height)

```

and then the parser re-writes the pipe operations to the following:

```julia
sort(filter(x -> x.age > 27, people), by = x -> x.height))

```

In other words, the _newly defined_ `|>` operator pipes the result of `filter` into the first argument of `sort`, and the _newly defined_ `|>>` operator pipes `people` into the last argument of `filter`.

EDIT: Basically our two different notations produce exactly the same result in this case (though the intermediate code is not identical). I just happen to think that my notation is more concise and is more understandable. 🙂

EDIT #2: Obviously my approach falls down if you want to pipe into a middle argument of a function, rather than the first or the last, but my hunch is that situation is uncommon and can often be mitigated.

EDIT #3: Removed the `@_` from the post-expansion code snippet. (Copy-paste error.)

EDIT #4: The above sequence of events might not work, depending on the order in which macro expansion and other code transformations occur. But I think it should work if the underscore expansion and the pipe translation are either both implemented via a macro or both implemented in the parser.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 29, 2020, 12:09am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/43 "2020-03-29T00:09:33Z")

</div>

Oh I see; so you’re saying that `|>` should behave differently than it does. In theory that’s a possibility for Julia 2.0. It’s also something that could be done if we were willing to rewrite the meaning of `|>` in the macro to be different from Base julia.

However I think @tkf was correct to say (over at [https://github.com/JuliaLang/julia/pull/24990#issuecomment-605388522](https://github.com/JuliaLang/julia/pull/24990#issuecomment-605388522)) that having special syntax+lowering for piping doesn’t compose well.

Though I might state it slightly differently — that it’s a _non-orthogonal design_ where `|>` is truly just another way to call functions and there’s no independent meaning for the things between the pipes.

Without an independent meaning for the things between pipes, how can one work with them as first class entities and reuse them in other contexts? For example I think it’s neat that the following works:

```julia
f = @_ filter(_.age > 27, __)
s = @_ sort(__, by = _.height)

pipeline = s ∘ f

people |> pipeline

```

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 29, 2020, 12:39am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/44 "2020-03-29T00:39:15Z")

</div>

I also think `__` is more flexible in the sense that `ys |> Iterators.product(xs,__, zs)` works (although maybe this example is a bit too contrived).

But I think what I agree with @CameronBieganek is that `_`-based anonymous function is useful when combined with “argument threading” (not sure if that’s the right term). So maybe it’s useful to make `@_` work with [Lazy.jl](https://github.com/MikeInnes/Lazy.jl) or just directly implement `@_`-like behavior in `@>`/`@>>`?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 29, 2020, 1:05am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/45 "2020-03-29T01:05:52Z")

</div>

It’s interesting that `__` combined with `|>` provides explicit argument threading like `Lazy.@as`, but with an opinionated choice of the threaded argument name.

I feel like explicitly having the `|>`'s is an improvement over the implicit threading in `Lazy` because it provides a natural way to do line continuation, and an important visual clue for what’s going on.

It so happens that `@_` already goes together with `Lazy.@>` in a neat way:

```julia
julia> foo(x, i) = (println("foo(_,$i)"); x)
foo (generic function with 1 method)

julia> @_ @> 10 foo(_, 1) foo(_, 2)
foo(_,1)
foo(_,2)
10

```

I think this is party by happy coincidence, but the design of Julia’s AST also makes this kind of consistency appear naturally.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [March 29, 2020, 1:21am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/46 "2020-03-29T01:21:44Z")

</div>

Ah, yes. Since `@_` already has `__`, supporting threaded macro does not adds much to what it can do. I also agree that infix `|>` is much easier to read than the spaces in Lazy.jl macros.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 29, 2020, 2:37am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/47 "2020-03-29T02:37:32Z")

</div>

I’m starting to come around on the double underscore, although it would be nice if it was a symbol that’s easier to distinguish from a single underscore.

@tkf, I think `ys |> Iterators.product(xs, __, zs)` is a reasonable example. Eventually (or regularly), someone is going to want to pipe into a middle argument, and then they would be disappointed when they found out that `|>` and `|>>` won’t do the trick.

> [@c42f](#):
>
> Though I might state it slightly differently — that it’s a _non-orthogonal design_ where `|>` is truly just another way to call functions and there’s no independent meaning for the things between the pipes.

I can see your point here. Currently in base Julia, and with your `@_` macro, anything to the right of a pipe must evaluate to a unary function. However, with the updated `|>` and `|>>` pipes, if you try to evaluate an expression to the right of a pipe on its own, you’ll end up calling a method of your function with one less argument than the method invoked by the pipe. That doesn’t necessarily make the syntax wrong, but it does seem conceptually less satisfying.

On the other hand, I’m not sure your example is too convincing. If I wanted to pull out part of a pipe and create a named function for later re-use, then I would probably just use normal function syntax:

```julia
f(x) = @_ filter(_.age > 27, x)
s(x) = @_ sort(x, by = _.height)
pipeline = s ∘ f

```

Or I could still use the updated-syntax pipes if I wanted to:

```julia
f(x) = x |>> filter(y -> y.age > 27)
s(x) = x |> sort(by = y -> y.height)

```

A minor quibble: It’s cool that `@_` and `@>` work together, but `@_ @> 10 foo(_, 1) foo(_, 2)` is a bit more complicated than just using `@>` by itself:

```julia
julia> foo(x, i) = (println("foo(_,$i)"); x)
foo (generic function with 1 method)

julia> @> 10 foo(1) foo(2)
foo(_,1)
foo(_,2)
10

```

Back to the double underscore. I do appreciate the way it marks which argument is being piped into, and that you can use it to pipe into any argument. But since this is a macro, for visual clarity would you consider using `~` instead of the double underscore? Here’s an example to look at:

```julia
table = [
    (age=45, height=180),
    (age=32, height=170),
    (age=55, height=175)
]

@_ table |>
    filter(_.age < 50, ~) |>
    sort(~, by=_.height) |>
    first |>
    ~.age

```

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 29, 2020, 3:06am UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/48 "2020-03-29T03:06:31Z")

</div>

> [@CameronBieganek](#):
>
> If I wanted to pull out part of a pipe and create a named function for later re-use, then I would probably just use normal function syntax

Yes. To be clear, I don’t think `__` is super clear syntax; in this case using a normal function would be a lot better. Actually the main point of `Underscores.jl` is to take seriously the observation that we generally don’t need a better way to write functions. What we need is a better way to write functions that will be immediately passed to functions. Hence the behavior of `_`.

So I don’t want to argue that `__` is desirable but rather that _something like it is necessary_ for piping in the current way things work.

`~` is an interesting option though being an operator it has some parsing surprises which I think will make it a bit unusable in general:

```julia
julia> :(~ == a)
ERROR: syntax: "==" is not a unary operator
Stacktrace:
 [1] top-level scope at REPL[41]:0

julia> :(__ == a)
:(__ == a)

```

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [March 30, 2020, 12:03pm UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/49 "2020-03-30T12:03:59Z")

</div>

I’ve wanted to react to 90% of the posts in this thread with the :mind-blown: emoji

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [April 6, 2020, 1:22pm UTC](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482/50 "2020-04-06T13:22:28Z")

</div>

Taking into account feedback here about having multiple `_`'s refer to the same argument of a single argument function, I’ve implemented this and released version 2. Also in the new release is the support for (optionally) using numeric unicode subscripts in placehoders allowing things like `_₂/_₁`.

For now I’ve punted on deciding about the precise behavior of `do` syntax; instead I’ve just disabled it for now so it can be added back in a compatible way once a decision is made.

[Previous page](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482.md?page=2)