Fixing the Piping/Chaining Issue

Internals & Design
, , , ,
102

Literally the only reason the compiler was able to fold your computation was because the input to your Fix thing was part of the benchmarking expression. Here, let me help you out:

julia> @benchmark Fix{(1,2,-3,-1)}(f, vals, z=5)(args...; kw...) setup=(f=(args...;kwargs...)->(args...,(;kwargs...));vals=(:a,:b,:getting_close,:END); args=(:y, 1, 2); kw=(:k=>2,))
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  52.667 μs … 176.066 μs  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     53.412 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   55.087 μs ±   5.945 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ▆█▅▂▁                 ▃                                      ▁
  █████▆▅█▆▆▇▆▄▅█▆▅▆▆▄▄██▇▅▅▇▆▆▇▆▅▅▆▅▅▆▅▅▅▄▄▄▄▄▂▄▄▄▄▄▅▂▃▄▄▅▅▅▆ █
  52.7 μs       Histogram: log(frequency) by time      84.9 μs <

 Memory estimate: 2.59 KiB, allocs estimate: 45.

julia> @btime Fix{(1,2,-3,-1)}(f, vals, z=5)(args...; kw...) setup=(f=(args...;kwargs...)->(args...,(;kwargs...));vals=(:a,:b,:getting_close,:END); args=(:y, 1, 2); kw=(:k=>2,))
  52.683 μs (45 allocations: 2.59 KiB)
(:a, :b, :y, 1, :getting_close, 2, :END, (k = 2, z = 5))

And ok, you might argue “but the function is fixed!”, then here, have this benchmark where only the input is not known and thus can’t be folded:

julia> @benchmark Fix{(1,2,-3,-1)}((args...;kwargs...)->(args...,(;kwargs...)), (:a,:b,:getting_close,:END), z=5)(args...; kw...) setup=(args=(:y, 1, 2); kw=(:k=>2,))
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  51.705 μs … 203.229 μs  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     52.434 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   54.838 μs ±   7.742 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ██▄▂  ▁        ▁  ▂▂  ▁                                      ▂
  ███████▇▇█▁▇█▆▆█▅▅██▇▆█████▇▇██▇▇▇▇▇▆▆▆▇▆▇▆▆▆▆▅▆▆▆▇▇▆▆▅▆▆▆▄▅ █
  51.7 μs       Histogram: log(frequency) by time        89 μs <

 Memory estimate: 2.55 KiB, allocs estimate: 44.


julia> @btime Fix{(1,2,-3,-1)}((args...;kwargs...)->(args...,(;kwargs...)), (:a,:b,:getting_close,:END), z=5)(args...; kw...) setup=(args=(:y, 1, 2); kw=(:k=>2,))
  51.934 μs (44 allocations: 2.55 KiB)
(:a, :b, :y, 1, :getting_close, 2, :END, (k = 2, z = 5))

It’s not better. The only case where it could be better is if everything is tuple, always, which is just not a realistic use case.

To me your reply just shows that you’re ignorant about what you’re actually defining and how that actually works. This is not convincing.

This again requires the autocomplete to know that d is a Beta, which is information you don’t have. Constructors are (sadly) not even required to return an object of their type, meaning you have to run type inference, as I’ve repeated multiple times now, to even begin checking which methods can possibly receive d.

Again, _ has no relation whatsoever to the “no argument” case because you can’t pipe anything into functions that don’t take any arguments. There is no generalization possible here because the 0 argument case is not a generalization of the 1 argument case, and neither of the n-argument case.

This already has syntax in the form of f(d...) = my_func(x, d...), which is MUCH clearer about what’s going on than having to read the second part since ... already has the established meanings.

Because they are in a statically typed language where the type of every single object is determined just by having a method/function in the first place. In a statically typed language, not knowing the type of a variable is a compiler error. It is not in julia.

It cannot, because Beta at the syntax level is just another function that can return an object of any type. Running at least type inference to know this is required.

Whether or not the function was created with _ or not has no bearing on which methods are selected. To the compiler, these two things are almost exactly the same, save for the name:

julia> f = (a,b) -> a+b
#3 (generic function with 1 method)

julia> g(a,b) = a+b
g (generic function with 1 method)

Whether f was created like f = _ + _ or with the explicit anonymous function syntax is irrelevant - the compiler never even sees the difference.

This already exists - annotate your types on every variable. Of course, this completely removes the ability of your code to be reused & generic, which is what you typically get in a static language. If you don’t want to do that and run type inference on every step, we AT LEAST require partial type inference to run up until invalid expressions, but again, that has NOTHING to do with whether you have fancy syntax for defining partially applied functions or not.

I will just repeat myself from above: You want a specific syntax and ascribe to that syntax mythical abilities that have nothing to do whatsoever with the actual semantic problems underneath. OOP languages can have their easy autocomplete on their syntax because they are statically typed; not the other way around.

103

First, even if you disagree with a proposal, please do not use language like that. Well-written and detailed proposals like this deserve to be heard and discussed in good faith, even if they are not implemented.

Simply lowering something without too much transformation is common in Julia. Consider eg [] (lowered as hvcat or variants, depending on content).

I think that lowering \> and /> with the proposed associative and precedence rules into a neutrally named generic function would be great. Then a package could take that and implement FixFirst and FixLast as described, or, if it pleases, do something completely different.

12 Likes
104

We could do all the Julia benchmarks and include compile time if you prefer.

Is running type inference problematic?

Never said it did.

The first bit of information you need, before you can begin your search for methods that dispatch on your object, is a) what the object is, and b) the fact that you are about to call a function on it. That’s what piping + fancy partial application syntax provide, in an order that’s convenient (and therefore accessible and likely to be used) for the human-machine interaction in question.

I’m not claiming to solve autocomplete, not by a long shot. I’m just hoping to get over one of the first hurdles so that it can be in reaching distance.

Perhaps I shall inform this person that they are not allowed to use autocomplete in Python, because it’s not statically typed.

105

This (long) discussion now features

  1. a proposed syntax,
  2. claims about the necessity and feasibility of autocompletion,
  3. various side-discussions about performance and implementation issues.

Personally, I think that the interesting part is 1., and 2. and 3. are distractions and sidetrack the readers from something that would be great to add to the language in its essential form (a pair of operators with the proposed syntax and precedence, that lower into a generic function, which packages could then define methods for).

3 Likes
106

Welcome back!

Sorry, but you missed that I jumped ship, and am now in support of essentially the proposal of #24990 (see reasoning here), with proposal for a truly generalized Fix type (description here). Still a rough draft.

In short, if done right, underscore placeholder syntax can lower into typed partial application functors, and more generally (and legibly), while treating the parser with greater kindness, than my original proposal for \> and />.

I hope it doesn’t disappoint :crossed_fingers:

2 Likes
107

Not necessarily, but right now there is no capability to run it on broken/incomplete expressions. That has to be built first, after which you’re still left with the question of what to do in functions that don’t have their types annotated, where type inference won’t help in the slightest because it’ll end up as giving you every possible function the IDE could know about.

And I’m arguing that no, they don’t solve either problem. You need (at least partial) type inference to know what type an object is. Once you’ve created an object, of course you’re going to pass it to a function (or return it, but I’ll assume that this is not wanted), what else are you going to do? Even operators are just functions in julia. Pretty much anything can be made callable and is thus potentially the thing to use the object on.

That’s ok - but do see that the surface level syntax is not the “first hurdle” to overcome and that it doesn’t solve the fundamental problem of “there is not enough information available to the IDE to autocomplete with”.

In fact, if you start writing a function name, editors using LanguageServer.jl are already suggesting potential names, so autocomplete in that regard already works - even without type inference. It does need that additional context of “here’s the first few characters” to filter out potential matches, but that’s already how autocomplete in other languages works anyway.

Polemic rhetoric aside, you could also just ask them how they continue to develop one of the big plotting packages in julia, Makie.jl, if the autocomplete is so unbearable :person_shrugging: They’re a main contributor after all and even published a paper about it.

That post is also 4(!) years old by now and a lot has changed since then. Atom/Juno was still widely used. There was no VSCode extension. LanguageServer.jl was barely beginning to start to use the very first versions of SymbolServer.jl, which is what’s used right now for providing the symbols for autocompletion in the first place. I don’t think the thread is representative of the problems faced today.

In regards to python’s autocomplete - try this and tell me what you get:

[sukera@tempman ~]$ python3
Python 3.10.8 (main, Oct 13 2022, 21:13:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
...     x = 1
... 
>>> Foo().<TAB>

or this:

[sukera@tempman pytest]$ python3
Python 3.10.8 (main, Oct 13 2022, 21:13:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
...     def test(self):
...             return 1
... 
>>> Foo().tes<TAB>

I can press <TAB> as many times as I like, I can’t get it to autocomplete. I can get that to work in ipython, mind you, but the exact same also already works in our REPL. What I can’t get to work even in ipython is then writing a function:

def baz(x):
    x.<TAB>

and it won’t autocomplete. How could it? It has no idea what type x might be. This is the exact problem I’m trying to convey is hard. It will do it if I write def baz(x: Foo):, because then it knows the type again, which is exactly the behavior I see in julia with e.g. VSCode and LanguageServer.jl once it knows the type - just like in any other statically typed language. Annotating the type however removes exactly that genericity we so desire for dynamic workflows.

2 Likes
108

Was not intending anything, just a poor choice of words, perhaps “rigamarole” closer to what I meant.

I think the proposal to have a bunch of currying capacity is a good idea, I just think the piping situation is basically a syntactic transform and should be attacked at that level.

3 Likes
109

Indeed, I’m glad to see someone bring this up, the assertion that somehow piping is going to improve autocomplete seems a bit optimistic to put it mildly.

In an IDE like vscode if you type something like

foo |> TAB

Under what circumstances does it gain any information over just

TAB

only when it knows the type of foo and can search through all the methods of all the functions that are specialized for that type OR take a generic type for the first argument. The number of functions that take a generic type for a first argument will be very high because of generic duck typing in Julia so it isn’t going to be super helpful. You’ll wind up with 26000 options often enough.

Furthermore there will be many if not most cases where the type of foo can’t be figured out. Such as

function bar(foo, baz)
   foo |> TAB

What it might do is help with top level scripts where global vars are being used. But if you care in any way about performance you’re still writing function barriers for top level scripts. Oddly the function barrier helps the compiler but makes the IDE have no idea what’s up

2 Likes
110

@Sukera If I understand your concern correctly, you are saying that adding a syntax like this does not fundamentally solve the problem of autocomplete since it’s still necessary to do type inference. That makes sense to me. However, it does seem plausible to me that syntax like this could make for a more convenient autocomplete UI experience at the user level, conditional on further engineering work under the hood—would you agree with that?

I think multiple contributors to the discussion have expressed what is also my sentiment very well, that the original proposal seems like a surprisingly elegant pair of operators, purely for syntactic convenience, and does not have to necessarily have the full power of being a perfectly consistent currying functor etc. etc.

They would not be the first operators to bind more tightly than function calls; both type annotation :: and broadcast . and do keyword do so I believe.

So, it seems there are many who would be excited for syntactic sugar for FrontFix and BackFix (or FrontPipe and BackPipe ? )

Most of the concerns I’m seeing seem like they revolve around either 1. some of the desired power of the operator or 2. some of the claimed second-order benefits of the operator, when neither of these points are actually the key selling feature, which is a nice way to thread objects through some common functional patterns.

What if the transformation is the following? Using the same $ as function application notation:

any_expr /> any_callable $ {args...} becomes any_callable $ {any_expr, args...}

And any_expr /> any_callable becomes args -> any_callable $ {any_expr, args...}. And vice versa for backfix. Thus for one-argument functions foo, then x /> foo() == x |> foo == x \> foo()

Using the example that breaks the powerful bells-and-whistles version from @CameronBieganek
[4, 9, 16] \> map $ {sqrt} \> filter $ {iseven}

We transform each \> right-associatively, so

([4, 9, 16] \> map $ {sqrt}) \> filter $ {iseven}

then

filter $ {iseven, ([4, 9, 16] \> map $ {sqrt})}

then inner transformation

filter $ {iseven, map $ {sqrt, [4, 9, 16]}} #evaluates to [2, 4]

So it looks like treating the operations just as syntax transformation no longer breaks. Did I do this correctly?

1 Like
111

No, I do not. \> or /> only exclude functions that don’t take any arguments at all from being possible matches, which aren’t considered for piping in the first place - they don’t take arguments after all. The potential engineering work by itself can inform possible method completions, but those are not going to be improved/further culled by knowing that one of the arguments to be piped is of a given type (and if it can, only in very limited cases, where the first/last argument is not already of the same type in all methods and dispatch is used for disambiguation).

The trouble with your proposal is that I can make any object callable, and that is not information available just from parsing:

julia> struct FooBar end

julia> (f::FooBar)(x) = "A `FooBar` instance got called with argument $(x)!"

julia> foobar = FooBar()
FooBar()

julia> foobar(42)
"A `FooBar` instance got called with argument 42!"

So to resolve that, you already need to know the boundary of where to stop resolving/applying the call you want to move around, which is the most contentious point of the _ proposal (julia will happily parse any symbol followed directly by parentheses as a Expr(:call, ...), but that has no bearing on whether or not the call will succeed. You either need to know ahead of time with some other information or pick semantics of when to stop moving calls around).

Another problem is that composing that with |> (which we can’t remove, would be breaking) is pretty hard - you can’t easily change lowering for it because it already has defined semantics:

julia> Meta.parse("a |> b |> c |> d")
:(((a |> b) |> c) |> d)

julia> Meta.parse("a |> b |> c |> d") |> dump
Expr
  head: Symbol call
  args: Array{Any}((3,))
    1: Symbol |>
    2: Expr
      head: Symbol call
      args: Array{Any}((3,))
        1: Symbol |>
        2: Expr
          head: Symbol call
          args: Array{Any}((3,))
            1: Symbol |>
            2: Symbol a
            3: Symbol b
        3: Symbol c
    3: Symbol d

Possibly? The problem is that we don’t have any notion of “function application operator” and thinking of foo() as doing that is not correct, as (I think) has been pointed out either somewhere far above or in the other thread.

If you want to see it as just a syntax transform, the best way to test your idea out is to write a macro to do it, since that’s exactly what macros already do. Whether or not such a transform is wanted/useful in Base is then a different discussion, unrelated to whether the idea works in principle.

1 Like
112

This is probably my own ignorance, but I admit I don’t understand why this is more difficult than do-block parsing.

can A(w) /> B(x) /> C(y) /> D(z) parse to something like

D(z) do
   C(z) do
        B(x) do
            A(w)
        end
    end
end

Although I know that do-block syntax wants anonymous functions passed into first argument not arbitrary expressions.

113

it would parse to A(w) /> B(x) /> C(y) /> D(z) as it should, just like do parses to a do block:

julia> Meta.parse("map([1,2,3]) do x ; x+1 end")
:(map([1, 2, 3]) do x
      #= none:1 =#
      x + 1
  end)

The question ultimately is can it mean something other than application of functions.

So basically these proposals are proposals to make operators work like operators (ie. lower to calls to functions) EXCEPT if the operator is \> or /> in which case it should specially lower to something else. This is very similar to introducing a “special form” such as “for” or “let” or “begin” or “do” except the special form would look like operators.

In some sense, for and such are exactly the same, they lower to stuff that doesn’t look anything like a for loop:

julia> f(x) = for i in 1:10 ; println(i); end
f (generic function with 1 method)

julia> @code_lowered(f(1))
CodeInfo(
1 ─ %1  = 1:10
│         @_3 = Base.iterate(%1)
│   %3  = @_3 === nothing
│   %4  = Base.not_int(%3)
└──       goto #4 if not %4
2 ┄ %6  = @_3
│         i = Core.getfield(%6, 1)
│   %8  = Core.getfield(%6, 2)
│         Main.println(i)
│         @_3 = Base.iterate(%1, %8)
│   %11 = @_3 === nothing
│   %12 = Base.not_int(%11)
└──       goto #4 if not %12
3 ─       goto #2
4 ┄       return nothing
)

In general I think it’s a very very bad idea to introduce a “special form” that looks like function / operator application. One of the absolutely best things about Julia is that whenever you look at the code you know exactly what the hell is going on. One of the absolute worst things about R is that absolutely anywhere you could run into something that looks like a function call but is actually “nonstandard evaluation”.

2 Likes
114

Fair enough, then what if they just use different keywords so it looks less like a function? e.g. into and over for front/backfix

[4, 9, 16] over map(sqrt) over filter(iseven) # evalutes to [2, 4]

Then syntax highlighting makes it purple and it’s clear it’s nonstandard evaluation. I anticipate most people would care more about the functionality than the exact semantics of expression

115

as opposed to

filter(iseven,map(sqrt,[4,9,16]))

or something like:

[4,9,16] |> ComposeChain(Curry(map,sqrt),Curry(filter, iseven))
117

@dlakelan I can only speak for myself, but I would prefer a threading syntax over any of those other options. I usually default to Chain.jl in these situations.

@chain [4, 9, 16] begin
    map(sqrt, _)
    filter(iseven, _)
end

But if into / over-like syntax existed I would probably use that.

I very much enjoy thinking functionally, but when I type something like this
filter(iseven,map(sqrt,[4,9,16]))

I will quite literally type it backwards right-to-left starting with the innermost function call. It is not super ergonomic but it’s how my brain works haha. I have full conviction that somewhere buried in this (long) thread we have the pieces to build a useful piece of syntax to make these patterns more ergonomic.

118

Indeed, I really like Chain and it’s imported by DataFramesMeta.jl so I wind up with it by default almost everywhere I’d want it.

But if we want syntax then I think the right thing to do is propose an alternative macro:

@piperator [4,9,16] |> map(sqrt,_) |> dothing(_,[1,2,3]) |> somesuch(a,_,b)

In fact I think we already have Pipe.jl that does this right?

119

and if you want a curried function, how about

@currierandives f(_,b) |> q(a,_,c) |> filter(bar,vcat(b,_))

Which results via syntax transformation in something like:

x-> filter(bar,vcat(b,q(a,f(x,b),c))
120

I like Chain.jl a lot, and I have nothing against Pipe.jl. But I can’t help but come back to one of @uniment 's opening statements:

Wouldn’t it be nice to type

x = mystruct into foo(arg1, arg2) into bar(arg3, arg4)

As a built-in ? Even using these macros we need to remember to include it everywhere, it’s still a few more characters to type, we need to trust that they will continue to be maintained as necessary, and other packages will not necessarily be designed to interop smoothly by default with whatever syntax is chosen by these third-party macros.

Obviously the following comparison is extreme maybe to the point of uselessness, but imagine if all we had were while loops and you had to use a third-party macro to run a for loop. This is somewhat how I feel about the current underpowered state of currying/piping.

2 Likes
121

That’s quite impressive. Perhaps they can update their original question with the answer, as I have yet not found it.

Let’s see if we can do with Julia what we can do with Python…

Python:

Let’s define a class:

class Foo:
    x = 1
    def bar(self):
        return self.x

Tab-complete shows the property x as well as the member method bar.

image
image1058×505 22.9 KB

Julia:

The Julian way of writing this is to recognize that Foo’s member method bar probably generalizes across a range of other Foo-like types, so we define an AbstractFoo type and specialize our methods on it. (We could specialize to Foo, and that would do exactly the same thing as having a class member method, but that’s not as Julian.) First we write Foo-specialized methods which directly access its fields, and then write methods of AbstractFoo to call them.

abstract type AbstractFoo end
Base.@kwdef struct Foo <: AbstractFoo
    x = 1
end
getx(foo::Foo) = foo.x
bar(foo::AbstractFoo) = getx(foo)

image
image1205×612 38.9 KB

Because we have syntax sugar for getproperty (namely . dot), and because I have typed Foo(), it knows to call propertynames on this object (or something similar). Because I have typed the object description, and typed a dot, autocomplete has the information it needs to help me discover x.

However, it’s not Julian to access the object’s properties directly; the preferred idiom is instead to call methods on it. So let’s find those methods.

Unfortunately, the situation isn’t so good for helping me discover either bar(::Foo), or getx(::Foo):

image
image1199×994 47.2 KB

What I claim is simple: that one day, autocomplete will recognize that I intend to call a function that specializes on a Foo, and it will help me find it. However, because the pipe operator can only call single-argument functions, at the moment such an autocomplete would not be very useful; we need a preferred partial application technique first, so that we can capture the whole range of possible method signatures.

Unless, of course, we should just solve the problem by making every method a dotted member method:

abstract type AbstractFoo end
Base.@kwdef struct Foo <: AbstractFoo
    x = 1
    bar = function(self) self.x end
end
Base.getproperty(x::AbstractFoo, n::Symbol) = begin
    if getfield(x,n) isa Function
        return (a...;k...)->getfield(x,n)(x,a...;k...)
    end
    getfield(x,n)
end

Now we finally get method discoverability in Julia, but it shouldn’t be this hard (or entirely non-Julian) to do.

image
image1468×733 55.3 KB

That’s not the problem I care about, because I do not care about solving impossible problems. I would have no time left for living life if I did.

You can consider a “member method” to be simply a function which is specialized to exactly the concrete type of the class. When hoping to tab-complete on member methods, I’m trying to find the most specialized functions! Why should I wish to hit tab to see all the methods which are not specialized to this object, nor to any object like it? I can just start typing random function names anyway.

Recall from inception the purpose of Julia, which is essentially to be a scripting language which compiles: to have the benefits of quick dev time and then quick run time. Scripting languages are quick and easy to develop in, partly because of not needing to assign types to objects, sure ok, but also because if you want to scratchpad something you can just casually start scratchpadding. You can hop into a REPL and experiment, and when you’ve found the appropriate methods, algorithms, and fragments of glue code you can copypaste back into the editor; or you can hit CTRL+ENTER in VSCode, or etc. It’s part of the preferred development style.

For sections of code where arguments are fully generic to ::Any, and if you’re also not scratchpadding the code and getting global vars from it, I don’t see why anyone would expect autocomplete to help.

122

I think these are two separate questions. If this is really a concern, then lobbying to get Chain.jl and Pipe.jl into Base seems reasonable. At the moment I don’t think there’s a real concern here. Both Chain.jl and Pipe.jl are under some kind of MIT license, it would always be possible for Julia Base or the julialang organization to absorb them in the future.

IMHO no, why do we need special syntax for this when macros are perfect for special syntax? and the _ syntax is actually more understandable imho. and it’s going to be annoying as hell to write macros that work with this syntax, also now you will never be able to use into or over as variable names and so it’ll break people’s code who have used those variable names. Again I think it’s “julia is no longer at that stage” territory.

2 Likes