ANN: Underscores.jl: Placeholder syntax for closures

Package Announcements
,
41

Thanks. My current preference to solve this problem would be to have filter(x->x.age > 27) return a lazy Filter object. (I also replied on the issue to elaborate on this.)

The __ syntax is a concession to the fact that all the Base functions are eager by default, and that this is generally the convention in the Julia ecosystem as a whole. Having syntax for this adds complexity, but I feel it’s necessary complexity. For example, how would you add sort to your data pipeline? With __ it’s really easy:

@_ people |>
   filter(_.age > 27, __) |>
   sort(__, by=_.height)

Without __ we would effectively be saying “oh no you generally can’t use functions in your data pipeline unless they’re designed for use in a data pipeline”.

(By the way, thinking about sort was the use case which convinced me that __ was actually a necessity for composability with arbitrary library functions.)

2 Likes
42

Actually, I think my proposal for pipes along with the single underscore usage of your macro works just fine with sort:

@_ people |>>
    filter(_.age > 27) |>
    sort(by=_.height)

Explanation

If we assume that the |> and |>> operations are re-written by the parser, the following would happen. First, your macro expands the underscores to this:

people |>>
    filter(x -> x.age > 27) |>
    sort(by = x -> x.height)

and then the parser re-writes the pipe operations to the following:

sort(filter(x -> x.age > 27, people), by = x -> x.height))

In other words, the newly defined |> operator pipes the result of filter into the first argument of sort, and the newly defined |>> operator pipes people into the last argument of filter.

EDIT: Basically our two different notations produce exactly the same result in this case (though the intermediate code is not identical). I just happen to think that my notation is more concise and is more understandable. :slight_smile:

EDIT #2: Obviously my approach falls down if you want to pipe into a middle argument of a function, rather than the first or the last, but my hunch is that situation is uncommon and can often be mitigated.

EDIT #3: Removed the @_ from the post-expansion code snippet. (Copy-paste error.)

EDIT #4: The above sequence of events might not work, depending on the order in which macro expansion and other code transformations occur. But I think it should work if the underscore expansion and the pipe translation are either both implemented via a macro or both implemented in the parser.

43

Oh I see; so you’re saying that |> should behave differently than it does. In theory that’s a possibility for Julia 2.0. It’s also something that could be done if we were willing to rewrite the meaning of |> in the macro to be different from Base julia.

However I think @tkf was correct to say (over at https://github.com/JuliaLang/julia/pull/24990#issuecomment-605388522) that having special syntax+lowering for piping doesn’t compose well.

Though I might state it slightly differently — that it’s a non-orthogonal design where |> is truly just another way to call functions and there’s no independent meaning for the things between the pipes.

Without an independent meaning for the things between pipes, how can one work with them as first class entities and reuse them in other contexts? For example I think it’s neat that the following works:

f = @_ filter(_.age > 27, __)
s = @_ sort(__, by = _.height)

pipeline = s ∘ f

people |> pipeline
1 Like
44

I also think __ is more flexible in the sense that ys |> Iterators.product(xs, __, zs) works (although maybe this example is a bit too contrived).

But I think what I agree with @CameronBieganek is that _-based anonymous function is useful when combined with “argument threading” (not sure if that’s the right term). So maybe it’s useful to make @_ work with Lazy.jl or just directly implement @_-like behavior in @>/@>>?

1 Like
45

It’s interesting that __ combined with |> provides explicit argument threading like Lazy.@as, but with an opinionated choice of the threaded argument name.

I feel like explicitly having the |>'s is an improvement over the implicit threading in Lazy because it provides a natural way to do line continuation, and an important visual clue for what’s going on.

It so happens that @_ already goes together with Lazy.@> in a neat way:

julia> foo(x, i) = (println("foo(_,$i)"); x)
foo (generic function with 1 method)

julia> @_ @> 10 foo(_, 1) foo(_, 2)
foo(_,1)
foo(_,2)
10

I think this is party by happy coincidence, but the design of Julia’s AST also makes this kind of consistency appear naturally.

3 Likes
46

Ah, yes. Since @_ already has __, supporting threaded macro does not adds much to what it can do. I also agree that infix |> is much easier to read than the spaces in Lazy.jl macros.

1 Like
47

I’m starting to come around on the double underscore, although it would be nice if it was a symbol that’s easier to distinguish from a single underscore.

@tkf, I think ys |> Iterators.product(xs, __, zs) is a reasonable example. Eventually (or regularly), someone is going to want to pipe into a middle argument, and then they would be disappointed when they found out that |> and |>> won’t do the trick.

I can see your point here. Currently in base Julia, and with your @_ macro, anything to the right of a pipe must evaluate to a unary function. However, with the updated |> and |>> pipes, if you try to evaluate an expression to the right of a pipe on its own, you’ll end up calling a method of your function with one less argument than the method invoked by the pipe. That doesn’t necessarily make the syntax wrong, but it does seem conceptually less satisfying.

On the other hand, I’m not sure your example is too convincing. If I wanted to pull out part of a pipe and create a named function for later re-use, then I would probably just use normal function syntax:

f(x) = @_ filter(_.age > 27, x)
s(x) = @_ sort(x, by = _.height)
pipeline = s ∘ f

Or I could still use the updated-syntax pipes if I wanted to:

f(x) = x |>> filter(y -> y.age > 27)
s(x) = x |> sort(by = y -> y.height)

A minor quibble: It’s cool that @_ and @> work together, but @_ @> 10 foo(_, 1) foo(_, 2) is a bit more complicated than just using @> by itself:

julia> foo(x, i) = (println("foo(_,$i)"); x)
foo (generic function with 1 method)

julia> @> 10 foo(1) foo(2)
foo(_,1)
foo(_,2)
10

Back to the double underscore. I do appreciate the way it marks which argument is being piped into, and that you can use it to pipe into any argument. But since this is a macro, for visual clarity would you consider using ~ instead of the double underscore? Here’s an example to look at:

table = [
    (age=45, height=180),
    (age=32, height=170),
    (age=55, height=175)
]

@_ table |>
    filter(_.age < 50, ~) |>
    sort(~, by=_.height) |>
    first |>
    ~.age
1 Like
48

Yes. To be clear, I don’t think __ is super clear syntax; in this case using a normal function would be a lot better. Actually the main point of Underscores.jl is to take seriously the observation that we generally don’t need a better way to write functions. What we need is a better way to write functions that will be immediately passed to functions. Hence the behavior of _.

So I don’t want to argue that __ is desirable but rather that something like it is necessary for piping in the current way things work.

~ is an interesting option though being an operator it has some parsing surprises which I think will make it a bit unusable in general:

julia> :(~ == a)
ERROR: syntax: "==" is not a unary operator
Stacktrace:
 [1] top-level scope at REPL[41]:0

julia> :(__ == a)
:(__ == a)
1 Like
49

I’ve wanted to react to 90% of the posts in this thread with the :mind-blown: emoji

1 Like
50

Taking into account feedback here about having multiple _'s refer to the same argument of a single argument function, I’ve implemented this and released version 2. Also in the new release is the support for (optionally) using numeric unicode subscripts in placehoders allowing things like _₂/_₁.

For now I’ve punted on deciding about the precise behavior of do syntax; instead I’ve just disabled it for now so it can be added back in a compatible way once a decision is made.

12 Likes