Fixing the Piping/Chaining Issue

Internals & Design
, , , ,
1

Objective

To satisfy Julia’s piping/chaining/currying/partial application problem with an elegant and functional approach worthy and idiomatic of Base Julia.

If you want, skip the background reading material straight down to the proposal.

Background

Piping, chaining, threading—whatever people call it, it’s a very common thought pattern to take an object and pull it through a sequence of one or more transformations. In many contexts it’s most natural to think of the object first, and then to think through the sequence of methods that act on it (or on transformed versions of it). As a consequence, many languages implement some mechanism for chaining methods to reduce the programmer’s mental load. Likewise, the English language reserves a keyword, “it,” to allow an object to be held in memory while performing a sequence of operations on it.

Julia implements the pipe operator |> for exactly this purpose, but unfortunately its inability to specify more than one argument makes it lame. As a result, about a dozen macro packages have been written (see appendix), attempting to solve the problem in slightly different ways. These macros use non-standard and non-standardizable syntax (i.e., syntax which cannot be considered idiomatic Julia and cannot exist outside of a macro call), meaning there’s little chance of seeing proper adoption into the Base language. Hence, the solution remains fragmented.

Perplexingly, many of those macros take a concept from an OOP language, Scala.

Until we solve this, we will continue to see requests and complaints (see appendix), as such a common thought pattern which is solved satisfactorily in many other languages is instead “supported” in Julia by a collection of competing macros with non-standardizable syntax. As much as Julians might wish the problem to go away, it simply doesn’t and will likely remain a lightning rod indefinitely.

What some other languages do

OOP Languages

OOP languages (such as C++, Java, and Python) employ dot-syntax to call member methods, offering a high degree of convenience when chaining object transformations using syntax like myObject.meth1(arg1a, arg1b).meth2(arg2a, arg2b). The reduction in mental load is so great—because it matches a natural thought pattern certainly, but also in good part because of the dot-tab-autocomplete this method call syntax enables—that it’s tempting to define functions as class members even when they needn’t be, just to reap the benefits of convenience that member methods enjoy. The package ObjectOriented.jl implements this.

Languages such as D and Nim implement UFCS (Universal Function Call Syntax), which appears exactly like dot-syntax but applies global functions and simply passes the dotted object as a first argument into the function:

myObject.meth1(arg1).meth2(arg2) == meth2(meth1(myObject, arg1), arg2)

Clojure

Clojure has two macros, -> and ->>, for “threading” objects through a chain of function calls. The thread-first macro, ->, takes an object and threads it through the function calls as a first argument. For example:

(-> obj (func1 arg1) (func2 arg2) (func3 arg3))

;; is equivalent to

(func3 (func2 (func1 obj arg1) arg2) arg3)

meanwhile, the thread-last macro, ->>, threads the object as a last argument through the function calls. For example:

(->> (range 10) (filter odd?) (map #(* % %)) (reduce +))

;; is equivalent to

(reduce + (map #(* % %) (filter odd? (range 10))))

The package Lazy.jl implements these threading macros as @> and @>>.

Scala

Scala, an OOP language which has dot-syntax for method calls, additionally employs “placeholder syntax” which uses the _ character as a placeholder for the argument of an auto-generated anonymous function. For whatever reason, this seems to be the approach many Julians have set their sights on; a bunch of Julia packages have been written to mimic this behavior (albeit, in slightly different ways): DataPipes.jl, Pipe.jl, Chain.jl, Underscores.jl, and Hose.jl, and several requests written about it: #24990, #46916. A strange justification for this approach is in circulation: a claim that in a multi-dispatch language no argument position should be treated as more privileged than any other.

Proposal

As outlined in this brainstorming session, what is proposed is a pair of partial application types Base.FixFirst and Base.FixLast, and a corresponding pair of infix operators /> (“frontfix”, or “fix”) and \> (“backfix”) respectively for syntactical sugar defined something like this:


struct FixFirst{F,X} f::F; x::X end
struct FixLast{F,X} f::F; x::X end
(fixer::FixFirst)(args...; kwargs...) = fixer.f(fixer.x, args...; kwargs...)
(fixer::FixLast)(args...; kwargs...) = fixer.f(args..., fixer.x; kwargs...)

/>(x, f) = Base.FixFirst(f, x) # frontfix
\>(x, f) = Base.FixLast(f, x) # backfix

The types Base.FixFirst and Base.FixLast are simply the logical extensions of the existing fix types Base.Fix1 and Base.Fix2, when extended to functions of other than just two arguments.

The fix and backfix operators /> and \> have right-associativity, bind more tightly than function calls, and have the same operator precedence as each other.

These properties allow these operators to satisfy the desire to chain operations on an object threaded as a first or last argument through a sequence of methods, as so:


# Frontfix (Base.FixFirst)
my_object /> meth1(args1...) /> meth2(args2...) /> meth3(args3...) ==
    meth3(meth2(meth1(my_object, args1...), args2...), args3...)

# Backfix (Base.FixLast)
my_object \> meth1(args1...) \> meth2(args2...) \> meth3(args3...) ==
    meth3(args3..., meth2(args2..., meth1(args1..., my_object)))

Notice that fix /> satisfies the role that dot-notation serves in OOP and UFCS languages. As we will see, it is more powerful. Inspiration has also been drawn from Clojure’s threading macros in suggesting backfix \>. However, because fix /> and backfix \> are infix operators, they are more convenient and useful than Clojure’s macros too. Also, as Base.FixFirst and Base.FixLast are logical extensions of Base.Fix1 and Base.Fix2, but with syntax sugar in the form of /> and \>, they should satisfy the desire to generalize and bring elevated status to the fix operations.

Examples

Let’s see how these operators can be put to use with some examples. Later, you can play with a demo macro to see how it works (see bottom).

Frontfix Chaining: XML Tree Navigation


document/>root()/>firstelement()/>setnodecontent!("Hello, world!")

# same as

setnodecontent!(firstelement(root(document)), "Hello, world!")

Frontfix Chaining: Starting a Spark Session


SparkSession.builder/>appName("Main")/>master("local")/>getOrCreate()

Frontfix Chaining: Sequence of Transformations


"Hello, world!"/>replace("o"=>"e")/>split(",")/>join(":")/>uppercase() ==
    uppercase(join(split(replace("Hello, world!", "o"=>"e"), ","), ":")) ==
    "HELLE: WERLD!"

Backfix chaining


[1, 2, 3]\>filter(isodd)\>map(x->x^2)\>sum()\>sqrt() ==
    sqrt(sum(map(x->x^2, filter(isodd, [1, 2, 3])))) ==
    3.1622776601683795

Combining Frontfix, Backfix, and Broadcasting


"1 2, 3; hehe4" \> eachmatch(r"(\d+)") \> first.() \> parse.(Int) /> join(", ") ==
    join(parse.(Int, first.(eachmatch(r"(\d+)", "1 2, 3; hehe4"))), ", ") ==
    "1, 2, 3, 4"

Replacing Base.Fix1 and Base.Fix2


NamedTuple{names, T}(map(nt /> getfield, names)))

# same as

NamedTuple{names, T}(map(Base.Fix1(getfield, nt), names))



Base.values(x::MyStruct{K1,<:Any,K2,<:Any}) where {K1,K2} = 
    (values(x.a)..., map(x.b /> getfield, filter(K1 \> ∉, K2))...)

# same as

Base.values(x::MyStruct{K1,<:Any,K2,<:Any}) where {K1,K2} = 
    (values(x.a)..., map(Base.Fix1(getfield, x.b), filter(Base.Fix2(∉, K1), K2))...)

Fixing and Composition

f = (", " \> join) ∘ ((2 \> ^) /> map) ∘ ((Int /> parse) /> map) ∘ (r",\s*" \> split)
f("1, 2, 3, 4")
    == "1, 4, 9, 16"

Fun Properties

Frontfix /> and backfix \> produce partially applied functions where one argument is pre-filled. Because they’re right-associative and have higher precedence than function call, they can progressively fill out a partial function until it’s fully applied.

To illustrate how this works, frontfix /> pops arguments out from the left side (or front) of the parentheses:


f(a, b, c, d) ==
a/>f(b, c, d) ==
b/>a/>f(c, d) ==
c/>b/>a/>f(d) ==
d/>c/>b/>a/>f()

the dual operation is backfix \>, which pops arguments out from the right side (or back):


f(a, b, c, d) ==
d\>f(a, b, c) ==
c\>d\>f(a, b) ==
b\>c\>d\>f(a) ==
a\>b\>c\>d\>f()

It’s not likely that this capability will be fully realized, but it’s nifty to understand. As a result of it, the operators have some overlap in their use:


[1, 2, 3, 4] \> filter(iseven) \> map(sqrt) /> join(", ") ==
[1, 2, 3, 4] /> iseven /> filter() /> sqrt /> map() \> ", " \> join() ==
    join(map(sqrt, filter(iseven, [1, 2, 3, 4])), ", ")

and /> can pipe an argument into the second position of a many-argument function:


arg2 /> arg1 /> method(args[3:end]...)

note that for single-argument functions, fixing into the front and fixing into the back are equivalent.

As a weird bonus, you can also evaluate expressions in Reverse Polish Notation:


3 \> 4 \> -() \> 5 \> +() == (3 - 4) + 5 == 4

and also in whatever notation this is:


3 /> 4 /> -() /> 5 /> +() == 5 + (4 - 3) == 6

Common Pushback & Responses

You just want syntax sugar to make OOP programmers happy.

True. OOP languages landed on a really good idea: the ability to feed successive transformations of an object through a sequential chain of function calls in a convenient and powerful way.

But also, Not true. This isn’t method encapsulation; this is just an elegant use of functional ideas that also does a good job solving a pattern that OOP languages accidentally found themselves good for, in a way that fills a void in Julia’s supported idioms.

In other words, it’s a generic solution to multiple classes of problem.

I don’t see how object→method1()→method2() chaining is better than method2(method1(object)).

One is not better than the other; they are complementary.

We frequently think of the method before the object when we are focused on a technique: perhaps a composition of functions to perform a task, or perhaps when combining several dissimilar objects. That’s where nesting objects in method argument lists is natural. However, when we are focused on transforming a single complex object in preparation for something else, we think of the object before the method.

Which format is better depends on context, and in contexts relevant to fixing an argument, one of the most common and compelling reasons for doing so is to thread an object through a sequence of transformations.

Part of the desire for dot-syntax comes from tab autocomplete. But soon Julia will have tab autocomplete for methods that match an argument type signature anyway, making this a moot point.

I have my doubts. Remembering to type a question mark first, then get halfway through typing the arguments before pressing tab, and then scanning a large list of methods all because you forgot a function name, is much more mental overhead than our OOP brethren suffer in this matter.

Maybe @tim.holy can debunk my stance, but it seems hard to beat typing an object name, pressing ., immediately seeing a list of all available methods, and watching the list filter down as characters of the method name are typed in.

I’ve tried ?(x, <tab> in the REPL, and if we can expect anything resembling its current behavior, it’s unlikely I’ll use it.

Julia is a multiple-dispatch language, so we don’t treat any argument as more special than any other.

Not true; who made this up? Julia even has a special do statement to insert an anonymous function into the *first* argument, proving we don’t actually believe this claim anyway.

Furthermore, the first or last ordered (non-keyword) argument of any function is usually chosen to be “important;” to define a function otherwise is usually poor design, regardless of the language paradigm. This is reflected in Clojure’s threading macros -> and ->>, wherein objects are threaded either through the first or last argument.

Sidenote: currently args... syntax can only slurp the last arguments of a function, not the middle or first. This could cause frontfix /> to be used more frequently than backfix \>.

Why not just use xyz.jl macro package? That should be good enough for anybody!

I like Julia in part because generators, matrix multiplication, regular expressions, complex numbers and so on are builtin and standard. What’s so wrong about wanting a better chaining operator to be builtin?

I’ve counted a dozen packages that implement chaining in some form or other. For such a common and generic idiom, maybe leaning on the package ecosystem for this is the wrong approach.

But the xyz.jl package uses _ to mark argument insertion. That’s more flexible; why don’t you get behind that instead?

If _ placeholder syntax stands a chance of being implemented in the language, I might. That approach has been proposed for half a decade now with no progress, e.g. #24990; so forgive me.

I also think frontfix and backfix will likely play nicer with autocomplete, and provide more interesting types for multiple dispatch.

Why introduce new operators /> and \>? Why not just use a pre-existing operator such as ⇝?

Two reasons:

  1. To avoid changes that might break existing packages that already use the symbol.
  2. To use ASCII characters, as these operators will likely be used often so should be easily accessible.

I’m open to other ideas for what characters should be used, but after exploring numerous options this seemed best.

It’s ugly. Should we require spaces around /> and \>?

I’m hoping that good syntax coloring will solve this.

Appendix

Packages Addressing Piping/Chaining

List of Complaint & Query Threads

Proposals to Fix (not Base.Fix) it

These lists are a WIP, as I continue to find more examples.

Demo

This implements a simple proof of concept so you can play with these operators and measure how much you like (or dislike) them. It’s a hack, so it should NOT be used for anything serious.

Many thanks to @bertschi for providing the code I copypasted and for helping brainstorm.

Demo Code:

using MacroTools: postwalk, @capture

struct FixFirst{F,X} f::F; x::X end
struct FixLast{F,X} f::F; x::X end
(fixer::FixFirst)(args...; kwargs...) = fixer.f(fixer.x, args...; kwargs...)
(fixer::FixLast)(args...; kwargs...) = fixer.f(args..., fixer.x; kwargs...)

frontfix(x, f) = FixFirst(f, x)
backfix(x, f) = FixLast(f, x)

# ugly hack: it's easiest to use `+` and `++` because they take varargs
macro fixdemo_str(expr)
    let + = frontfix, ++ = backfix
        expr = replace(expr, "++"=>"⤔", "+"=>"⤕", "/>"=>"+", "\\>"=>"++")  
        expr = Meta.parse(expr)
        expr = postwalk(expr) do ex
            if @capture(ex, +(args__))
                unchain(+, args, [])
            elseif @capture(ex, ++(args__))
                unchain(++, args, [])
            else
                ex
            end
        end
        expr = replace(string(expr), "⤕"=>"+", "⤔"=>"++")
        Meta.parse(expr)
    end
end

function unchain(op, terms, stack)
    if isempty(terms)
        :(foldr($op, [$(stack...)]))
    elseif @capture(terms[1], f_(args__)) && f != :foldr  # don't touch nested transforms again
        arg = :(foldr($op, [$(stack...), $f])($(args...)))
        unchain(op, terms[2:end], push!([], arg))
    else
        unchain(op, terms[2:end], push!(stack, terms[1]))
    end
end

Some fun examples to try:

fixdemo""" "Hello, world!" /> replace("o" => "e", "l" => "r") /> split(",") \> map(uppercase) /> join(":") """
fixdemo""" [1, 2, 3, 4] \> filter(iseven) \> map(sqrt) /> join(", ") """
fixdemo"1 /> +(2)" # you will never do this (or will you?)
fixdemo"rand(10) \> filter(≤(0.5))" # define-map-filter-also-as-functionals/87401
fixdemo"(1:10)\>filter(iseven)\>map(sqrt)/>collect()" # define-map-filter-also-as-functionals/87401/3
fixdemo"rand(10) \> filter(≤(0.5)) \> map(1 \> +)" # define-map-filter-also-as-functionals/87401/8
fixdemo"(1:100) \> mapreduce(2 \> ^, +) /> √()" # root of sum of squares
fixdemo"(1:5) \> filter(isodd)"
fixdemo"(1:5) /> isodd /> filter()"
oddfilt = fixdemo"isodd /> filter"
fixdemo"(1:5) /> oddfilt()"
hello_replace = fixdemo""" "Hello, world!"/>replace """
hello_replace("l"=>"r")
hello_replace("o"=>"a", "l"=>"y")
fixdemo""" "1 2, 3; hehe4" \> eachmatch(r"(\d+)") \> map(first) \> map(Int/>parse) /> join(", ") """
fixdemo"3 \> 4 \> -() \> 5 \> +()" # rpn for the funz
f = fixdemo""" x -> x /> split(r",\s*") \> map(Int /> parse) \> map(2 \> ^) /> join(", ") """
f("1, 2, 3, 4")
g = fixdemo""" (", " \> join) ∘ ((2 \> ^) /> map) ∘ ((Int /> parse) /> map) ∘ (r",\s*" \> split) """
g("1, 2, 3, 4")

Play around with it and share your thoughts please! Note that when partial functions start getting busy, it’s often better just to define a conventional anonymous function; some of the examples are mildly excessive just for illustration.

54 Likes
2

Very interesting. My first reaction looking at the examples: “this is an unacceptable break of referential transparency!” because in document/>root()/>firstelement()/>setnodecontent!("Hello, world!") for example you cannot replace root() by its result without changing the meaning of the code. But I was wrong!

As you explain, the new operators have higher precedence than function call so the example can be read as

(((document /> root)()) /> firstelement)()) /> setnodecontent!)("Hello, world!")

I’m not 100% sold on the readability of the proposed feature, but at least there’s a solid non-magical way to understand it, and it does cover an impressive range of use cases.

For readability I think #24990 is the absolute best we can do (the simple version implemented in the PR, not the alternative rule consuming “one call followed by zero or more operator calls”). But the feature proposed here is obviously much more powerful.

4 Likes
3

I appreciate that you still like the /> and \> notation :slight_smile:

I think you outlined the tradeoffs well. I anticipate most people would agree that the notation is powerful, useful, and elegant; however with great power comes great responsibility, and it adds the potential for unreadable golfed code.

I guess the question comes down to if the usability it provides is worth the skill floor it raises? I would say yes, just so I can stop importing Chain into every script I ever write, but I am partial to this kind of expression (pun intended).

IMO the observation that this can help tab-complete be more smooth is a good one.

4 Likes
4

Quite a few of these examples work already with |> and .|>:

# (1:100) \> mapreduce(2 \> ^, +) /> √()"
(1:100) .|> abs2 |> sum |> sqrt  # three existing named functions
sum(abs2, 1:100) |> sqrt  # lazier version

# (1:5) \> filter(isodd)
(1:5) |> filter(isodd)  # filter just works (Julia 1.9)

# (1:10) \> filter(iseven) \> map(sqrt) /> collect()  # collect not necc.
(1:10) |> filter(iseven) .|> sqrt
[sqrt(x) for x in 1:10 if iseven(x)]  # lazier existing idiom

Almost all of the others would be well-served by the simplest _ proposal. Perhaps it’s familiarity but these all seem easier to read:

# document /> root() /> firstelement() /> setnodecontent!("Hello, world!")
document |> root |> firstelement |> setnodecontent!(_, "Hello, world!")

# [1, 2, 3, 4] \> filter(iseven) \> map(sqrt) /> join(", ")
[1, 2, 3, 4] |> filter(iseven) .|> sqrt |> join(_, ", ")

# "Hello, world!" /> replace("o" => "e", "l" => "r") /> split(",") \> map(uppercase) /> join(":")
"Hello, world!" |> replace(_, "o" => "e", "l" => "r", "," => ":") |> uppercase

# "1 2, 3; hehe4" \> eachmatch(r"(\d+)") \> map(first) \> map(Int/>parse) /> join(", ")
"1 2, 3; hehe4" |> eachmatch(r"(\d+)", _) .|> first .|> parse(Int, _) |> join(_, ", ")

# g = (", " \> join) ∘ ((2 \> ^) /> map) ∘ ((Int /> parse) /> map) ∘ (r",\s*" \> split)
g = join(_, ", ") ∘ map(abs2, _) ∘ map(parse(Int, _), _) ∘ split(_, r",\s*")

How easy is it to invent (or ideally, find in the wild) examples for which this is not true? I.e. for which this proposal does things you couldn’t easily do with 24990? Both work “one level deep”, I think, so perhaps they are always close.

If the objective is mainly tab-completion, then x /> has more information than |> as it knows that x will be the first argument. So one possible use of this idea would be that "str" /> jo<tab> could complete to "str" |> join(_,?

16 Likes
5

Whatever we do, TAB completion should be an/the (main) objective. I think that’s the main reason OOP people like OOP, not because it’s single-dispatch. Maybe we can get it in VS Code (and the REPL) without (or with some) new syntax.

I’m focusing mainly on the /> operator for compatibility with other languages (except for how it looks), do we need the corresponding \>? Would it be really useful (other languages do without similar)?

Whatever we do, IF we adopt one or both, what would be good Unicode aliases, IF we want those…?

3 Likes
6

Admittedly, functions like map, filter, and reduce were some of the primary drivers motivating the backfix operator \>.

Maybe the practice of offering alternate curried function forms such as filter(f) = x -> filter(f, x) will become standard practice in Julia, so that |> will be all that’s ever needed? My gut feel is that a convention of specifying method variations solely for the sake of currying could begin to inhibit other uses of multiple dispatch and unduly burden the task of specifying interfaces, so maybe not.

Ideally the frontfix and backfix operators will work with broadcasting (e.g. [1,2,3] /> sqrt.()), but the demo macro doesn’t yet support it. This is why in my examples I used map so frequently. But the intent eventually is to support broadcasting.

admittedly some of the examples are a bit contrived just to illustrate what’s possible.

Perhaps the biggest difference is in assisting tab autocomplete by specifying argument(s) before starting on the function name.

They’re both useful for chaining, and they’re both useful for specifying partially evaluated functions. It seems like the most prominent difference is in storing and passing partial functions: whereas /> and \> partially evaluate on one argument, _ syntax partially evaluates on every argument except one. EDIT: THIS IS INCORRECT. This means that it’s possible to create partial functions with /> and \> that cannot be created with _, yet every partial function from _ can be made with /> and \> (although whether it’s worthwhile to do so is another question entirely).

An example of this is among my “fun examples” list: the partial function hello_replace can take multiple arguments, as illustrated by the second call to it. If I’m not mistaken, a function partially evaluated by _ syntax cannot do this.

On the other hand, when you want to partially evaluate a multi-argument function until it has only one argument left, curry underscore can be much easier. It’s possible to do the same with /> and \>, but not before pulling out your hair in the process. Then again, most of the time the goal when creating single-argument partial functions is for |> chaining, which /> and \> handle already without the need for |>.

So it seems a cost/benefit analysis is in order.

If it helps, one can think of frontfix /> and backfix \> as being infix variants of Clojure’s thread-first -> and thread-last ->> macros. Perhaps we could instead use /> and />> for a greater sense of consistency with Clojure :wink: I’m attracted to /> and \> because of the inclusion of the backslash character \, which one can associate with the backfix operation: which threads the object into the back of the argument list. Silly perhaps to fixate on such a minor detail, but I like it.

Including backfix \> was largely for want to assist functions such as filter, map, reduce, and mapreduce, which take the threadable object as a last argument, but it’s useful anywhere one would have used Base.Fix2 as well.

If ever args... slurping becomes available for the first or middle arguments, use of backfix \> could become more generally useful as people begin to see the last argument position as a viable location for important arguments (similar to filenames coming last in command line arguments). But until then…

This sentiment is important.

Encapsulating methods in classes is rendered largely unnecessary by multiple dispatch; the remaining wall to tear down is a respectable autocomplete, as illustrated by this thread.

1 Like
7

For most functions there is one argument that is the “data” argument that you want to pipe in.
What about a package of versions of functions with the data argument as first argument e.g.

_filter(a,b)    = filter(b,a)
_map(a,b)       = map(b,a)
_eachmatch(a,b) = eachmatch(b,a)
_parse(a,b)     = parse(b,a)

Then you could do the first 3 examples all with Lazy @>. which has far less text.

@>   "Hello, world!"   replace('o'=>'e')        split(',')      join(':')       uppercase
@>   [1  2  3]         _filter(isodd)           _map(x->x^2)    sum             sqrt
@>   "1 2, 3; hehe4"   _eachmatch(r"(\d+)")     first.()        _parse.(Int)    join(", ")

This is probably too extreme but what about pushing for all functions to have the data argument first for Julia 2.0

1 Like
8

This worldview is natural for someone with an object orientation, but it’s actually a pretty restrictive view of functions and can needlessly limit your problem-solving creativity.

Notably, it’s not a good perspective for solving the sorts of problems that Julia is uniquely good at solving. Imposing this worldview on Julia is an affront to its core competencies and identity, sort of like demanding a Michelin chef to cook at McDonald’s.

Can more functions be written with “the” data object in first argument position? Sure, there’s a subset of functions for which it might be appropriate. But I wouldn’t wish for such a requirement to be imposed with a broad brush if I can help it; I’d rather that the function author has creative license to express ideas in a manner most natural and performant for its context, and for me to have the expressive capacity to accommodate and integrate it as smoothly as possible into my own context.

This would be a nightmare in the long run.

Where by “far less text” you just mean not having to type /> a few times, in exchange for having to prefix the entire line with a macro call and impose that all function arguments be in the order you need at that particular moment? Imagine feeling this way toward other infix operators like + or -…

As mentioned in the OP, /> and \> are operators that implement infix variants of Clojure’s -> and ->> threading macros (which Lazy.jl implements with @> and @>>). The /> and \> operators, if implemented, bring the additional benefits of being able to be intermingled, of being able to return partial functions, and of not requiring a macro call—i.e., they can be considered “base” Julia. In summary, they’re simply better.

Way too extreme.

1 Like
9

This is an impressive proposal.

These are more convenient if you need to switch between \> and />. However, if you’re using functions like map that mostly take their data argument last, or DataFrames.jl functions that mostly take the data argument first, it’s more convenient to see the control flow macro at the beginning and only once. For instance, it’s annoying in R to have to put %>% at the start of each line in a chain. If your syntax is added, I would like to also have @\> and @/> behaving like Clojure’s prefix threading macros.

3 Likes
10

Options are good :wink:

11

First, thanks for putting together such a well-written and thorough proposal!

I think this would be a great way to resolve #36181. Yes, Fix1 and Fix2 were intended to be internals, but now I see a lot of packages using them, and a generalization with some dedicated syntax would not hurt.

FWIW, I don’t think this has much to do with OOP, it is just a useful syntax for a commonly occurring situation. Chaining can lead to very clean code in some contexts.

18 Likes
12

I actually really like @chain and its flexibility and ability to specify a general argument location using _ and the ability to insert @aside and thus don’t think I see a reason why I would use this proposal at all. Of course maybe it doesn’t affect me then… But having yet another chaining methodology doesn’t seem super attractive. IMHO chaining is syntactic sugar and a macro is the right place for sugar… What are the compelling arguments against that point of view?

3 Likes
13

mylist_of_lists .\> filter(my_custom_predicate)

Seems like a very handy syntax. @chain is great for long threads and I use it quite frequently in conjuction with DataFrame transformations, for example, but for short things like the above it’s a lot more typing.

Also the integration with tab-complete could be huge, I don’t think that should go understated. I know theoretically it could just complete to |> f(_ ...) but if the functionality is going to be added may as well go all the way.

2 Likes
14

I would just type this as

filter.(my_predicate,mylists)

and find that way easier to understand, particularly that the filter is being broadcast, as opposed to the \> being broadcast which I find tremendously confusing.

1 Like
15

One could say the exact same thing about |> and .|> though.

2 Likes
16

which I also never use :sweat_smile:

1 Like
17

This already works:

julia> [i:4i for i in 1:3] .|> filter(isodd)
3-element Vector{Vector{Int64}}:
 [1, 3]
 [3, 5, 7]
 [3, 5, 7, 9, 11]

On 1.9, but someone should make a PR adding it to Compat.jl.

1 Like
18

yes it happens to work because someone defined a partial application for filter, but this is not the case for every function, hence why you see Fix1 and Fix2 in the wild. For example, the modulo operator % does not have a partial application defined the same way that ==() does.

For example, with the backfix operator, I can get (inefficiently) the divisors of x by

x \> range(1) .\> %(x) |> findall(==(0))

Using existing arrows requires

x |> (j -> range(1, j)) .|> (j -> %(x, j)) |> j -> findall(==(0), j)

Which is obviously sufficiently ugly that I would just write that over multiple lines. Also note above that due to operator precedence interaction of |> and -> it will only work if I am explict about how I parenthesize the expression.

2 Likes
19

Sure. I just remain surprised how hard it seems to be to come up with neat examples where the proposal is clearly useful. Note that this one uses x again, so it’s not quite a chain, and seems clearer without pipes:

julia> let x = 10
         #   x \> range(1) .\> %(x) |> findall(==(0))
         a = x |> (j -> range(1, j)) .|> (j -> %(x, j)) |> j -> findall(==(0), j)
         b = findall(iszero, x .% (1:x))
         a == b
       end
true
2 Likes
20

If you type .\> it implies broadcasting this way:

FixLast.(filter, mylist_of_lists)(my_custom_predicate)

Maybe it’s because I’m sleep deprived, but I don’t think this makes sense, as we now have a collection of FixLast functors that we are attempting to call.

My thought was to use syntax like this for broadcasting:

mylist_of_lists \> filter.(my_custom_predicate)

which would mean

FixLast(filter, mylist_of_lists).(my_custom_predicate)

I think a method of broadcast would have to be written (specialized on FixFirst and FixLast) such that filter is broadcasted across mylist_of_lists.

1 Like