# Fixing the Piping/Chaining Issue

**URL:** <https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654>\
**Category:** Internals & Design\
**Tags:** proposal, piping, chaining, partial-evaluation, threading\
**Created:** [November 2, 2022, 11:04am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654 "2022-11-02T11:04:36Z")\
**Posts on this page:** 20\
**Page:** 10

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 9, 2022, 9:15pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/184 "2022-11-09T21:15:40Z")

</div>

Just some thoughts, it’s been a long time since I used Common Lisp but if I remember correctly you can actually replace the parser at run time with something else. There’s something called the “readtable” and you can make “read” do basically anything… This can be super useful for constructing entire honest to goodness domain specific languages which are not a subset of the basic syntax of common lisp. Similarly I could imagine something like:

```julia
@with_reader foo ...

```

and in the ‘…’ it would actually call the reading function foo to read and parse the stuff in …

i’m imagining that JuliaSyntax might actually enable that sort of thing to happen? Whether that’s a good idea or not is up in the air 😅

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 9, 2022, 9:18pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/185 "2022-11-09T21:18:31Z")

</div>

> [@c42f](#):
>
> But might be an interesting alternative take.

Yes, in fact I might be inclined to say it’s better! I think it seems to sacrifice some of the most powerful functor-currying-partial-application power of the original proposal, but in return it gets to be a lot simpler and more obvious, and probably easier on the compiler, while still satisfying probably the majority of use cases. I think we can still do things like `map(/> myfunc(myparam), myvec)`

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [November 9, 2022, 10:00pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/186 "2022-11-09T22:00:28Z")

</div>

Counterproposal, rewriting yours:

### Example Possible Autocomplete Behavior

Let’s create an object `df = DataFrame(a=1, b=2)`. When I type

```julia-auto
df |> 

```

I should see a (very very long) list of methods appear: those which specialize on `typeof(df)`, followed by methods which specialize on `supertype(typeof(df))`, followed by `supertype(supertype(typeof(df)))`, etc., sorted by degree of specialization. The list is about two thousand entries long, something like this:

```julia-auto
df |>
  append!(DataFrame,
  copy(DataFrame;

  ⋮ other methods of `DataFrame`
  ⋮ (vdots here to shorten my explanation)

  Array(AbstractDataFrame)
  ==(AbstractDataFrame,
  (Matrix)(AbstractDataFrame

  ⋮ other methods of `AbstractDataFrame`

  ArgumentError(msg)
  AssertionError(msg)
  BoundsError(a)

  ⋮ other methods of `Any`

```

And then I can scroll down the list to find what I’m looking for. One neuron fires in my brain and I remember that the first character is a `p`. So I type `p` and I see:

```julia-auto
df |> p
  pop!(DataFrame)
  popat!(DataFrame,
  popfirst!(DataFrame)
  prepend!(DataFrame,
  push!(DataFrame,
  pushfirst!(DataFrame,

```

I think maybe for first level we should show two levels, `typeof(df)`, but not followed by `supertype(typeof(df))` rather disregarding the distinction (unless the supertype is Any?), and order alphabetically as one group?

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 9, 2022, 10:32pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/187 "2022-11-09T22:32:53Z")

</div>

Okay so since working through that example I basically did all the work needed to figure out how to it anyway, I threw together a quick and simple script that does what I described for a simple autocomplete (you knew I would, didn’t you 😤)

> **Code here**
>
> ```julia
> 
> function propose_method_autocompletions(obj, func_name_fragment::String=""; 
> type_depth = 1:typemax(Int), only_same_module = false, 
> prioritize_firstlast = false, github_inference = false, personalized_inference = false)::Vector{Method}
> 
> @assert first(type_depth) ≥ 1 && last(type_depth ≥ 1)
> recs = Vector{Method}[]
> get_type_at_depth(type, depth=1) = depth == 1 ? type : type == Any ? nothing : get_type_at_depth(supertype(type), depth-1)
> 
> for i ∈ type_depth
> stype = get_type_at_depth(typeof(obj), i)
> isnothing(stype) && break
> 
> meths = filter(only_same_module ? methodswith(stype, parentmodule(typeof(obj))) : methodswith(stype)) do m
> length(func_name_fragment) > length(string(m.name)) && return false
> string(m.name)[1:length(func_name_fragment)] == func_name_fragment
> end
> 
> prioritize_firstlast || true # do cool sorting stuff, add later
> github_inference || true # do cool sorting stuff, add later
> personalized_inference || true # do cool sorting stuff, add later
> 
> recs = [recs; meths]
> end
> 
> recs
> end
> 
> ```

To invoke:

```julia
using DataFrames
df = DataFrame(a=1, b=2)

propose_method_autocompletions(df)

```

By default, it returns all available methods that can act on object `df`. As you start to type in a function name, the list gets narrowed down quickly:

```julia
propose_method_autocompletions(df, "p")
propose_method_autocompletions(df, "pu")
propose_method_autocompletions(df, "pushfirst!")

```

You can also change the search depth. By default, it searches specializations on `typeof(df)`, as well as `supertype(typeof(df))`, `supertype(supertype(typeof(df)))`, and so on. This can be changed like so:

```julia
propose_method_autocompletions(df, "p"; type_depth=1:2)

```

This gives methods of `DataFrame` as well as `AbstractDataFrame`, but not `Any`. The iterator can also be reversed `2:-1:1` to show the abstract type’s methods first.

You can also restrict the search to only methods that are defined in the same module as `DataFrame`:

```julia
propose_method_autocompletions(df, "p"; only_same_module = true)

```

And, one day, it’ll work with the `arg_order_inference`, `github_inference` and `personalized_inference` options too. 😉

Someday it could be interesting to make it feed its results into a `Channel`, so that it can offer results with greater immediacy.

_Unfortunately it doesn’t solve \*my\* problem, because the module didn’t export its methods so they don’t appear in `methodswith` 😅_

_Oddly, when calling `methodswith(DataFrame, DataFrames)` (trying this to see if it improves speed over filtering \*after\* the `methodswith` call), it simply doesn’t return a bunch of methods of `DataFrame` that are indeed declared by the `DataFrames` module. For example, `pushfirst!` is missing. Strange._

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [November 9, 2022, 11:50pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/188 "2022-11-09T23:50:08Z")

</div>

> [@uniment](#):
>
> The motivation for right-associativity was not this; it was for being able to fix multiple arguments. For example, `arr \> filter(isodd)` could also be `arr /> isodd /> filter()`. This would enable things like `myfilt = isodd /> filter` and then `arr /> myfilt()`.

Yes I see. It’s very funny that we’ve done the opposite things here in terms of precedence, but with the end result being almost the same from the user’s perspective 😆

I think `/>` having high precedence would be surprising to anyone who’s trying to use it for piping. For example, what does this mean?

```julia
x /> Mod1.g[1].h(y)(z)

```

- Low precedence: `Mod1.g[1].h(y)(x, z)`
- High precedence: `Mod1.g[1].h(x,y)(z)` … presumably?

> [@CameronBieganek](#):
>
> I think it would be semantically cleaner to just have “front pipe” and “back pipe” than to have “fix every argument but the first” and “fix every argument but the last”, with an implicit chain thrown in.

Couple of thoughts here:

- Is there a flexible lowering which might allow pipelines of `/>` to be optimized more completely. For example, turning `x \> map(_^2) \> reduce(+)` into `mapreduce(_^2, +, x)`. In particular I’m thinking of how transducers are a lot better for compilation than iterators, as described here [Comparison to iterators · Transducers.jl](https://juliafolds.github.io/Transducers.jl/dev/explanation/comparison_to_iterators/) and that we could make use of the iterator-transducer duality if we could give the compiler the right insight. @tkf 🙂
- What about first class pipelines without data such as `\> map(isodd) \> filter(_>10)`? Well ok I guess these can be lowered to a lambda `x -> filter(y->y>10, map(isodd, x))`

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [November 10, 2022, 12:04am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/189 "2022-11-10T00:04:19Z")

</div>

> [@c42f](#):
>
> Is there a flexible lowering which might allow pipelines of `/>` to be optimized more completely. For example, turning `x \> map(_^2) \> reduce(+)` into `mapreduce(_^2, +, x)`.

To answer my own question, yes! The `fixbutfirst`/`fixbutlast` lowering already allows this. Adding the following definition to `chain` will turn a `map \> reduce` pipeline into a call to `mapreduce`:

```julia
function chain(x, f1::FixButLast{typeof(map)}, f2::FixButLast{typeof(reduce)}, fs...)
    chain(x, fixbutlast(mapreduce, f1.args..., f2.args...; f1.kwargs..., f2.kwargs...), fs...)
end

```

Presumably this is not the best such lowering for such transformations, as this is a fairly special case. But it does give a tantalizing taste of possibility.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [November 10, 2022, 1:34am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/190 "2022-11-10T01:34:23Z")

</div>

Great, thanks for the implementation!

A nice thing is that such autocomplete can be added without any changes to julia syntax: just expand it to `obj |> it->func(<cursor here>, it)` instead of `obj |> func(<cursor here>, _)`. So, it can already be implemented in a package using eg ReplMaker.

I tried your `propose_method_autocompletions`, and some kind of sorting would definitely help.  
For a simple table, `tbl = [(a=1,)]`, the first suggestions without typing the method name are:

```julia
ulia> propose_method_autocompletions(tbl)
[1] ArgumentError(msg) in Core at boot.jl:325
[2] AssertionError(msg) in Core at boot.jl:341
[3] BoundsError(a) in Core at boot.jl:277
[4] BoundsError(a, i) in Core at boot.jl:278
[5] ConcurrencyViolationError(msg) in Core at boot.jl:290
[6] DomainError(val) in Core at boot.jl:296
[7] DomainError(val, msg) in Core at boot.jl:297
[8] ErrorException(msg) in Core at boot.jl:267
[9] InexactError(f::Symbol, T, val) in Core at boot.jl:318
[10] InitError(mod::Symbol, error) in Core at boot.jl:354
[11] InitError(mod, error) in Core at boot.jl:354
[12] LineNumberNode(l::Int64, f) in Core at boot.jl:407
[13] LoadError(file::AbstractString, line::Int64, error) in Core at boot.jl:348
[14] LoadError(file, line, error) in Core at boot.jl:348
[15] MethodError(f, args) in Core at boot.jl:338
[16] MethodError(f, args, world::UInt64) in Core at boot.jl:335
[17] (::Type{T})(itr) where T<:Tuple in Base at tuple.jl:317
[18] NamedTuple(itr) in Base at namedtuple.jl:123
[19] OverflowError(msg) in Core at boot.jl:321
[20] Pair(a, b) in Core at boot.jl:825
...

```

None of these methods are commonly used on tables/collections.  
Typing `propose_method_autocompletions(tbl, "push")` does find the `push!` function of course, but still doesn’t show the expected `push!(tbl, ...)` method among the first suggestions. The first is `push!(s::Set, x)`, so it wants to put `tbl` as the 2nd argument.

Autocomplete based on the ideas from this topic can already be implemented in a package, and would be great if it turns out actually useful! But for now I’m skeptical, given a huge amount of methods specifying few to no argument types.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 10, 2022, 3:37am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/191 "2022-11-10T03:37:58Z")

</div>

> [@aplavin](#):
>
> But for now I’m skeptical, given a huge amount of methods specifying few to no argument types.

Oops! It appears that calling `methodswith` on a parameterized type doesn’t give the methods of the unparameterized type! My autocomplete was failing to find any methods of `Vector` because it was being parameterized with the type of its contents.

> **See this example.**
>
> ```julia
> 
> julia> struct MyThing{T} end
> 
> julia> foo(::MyThing) = 1
> foo (generic function with 1 method)
> 
> julia> methodswith(MyThing)
> [1] foo(::MyThing) in Main at REPL[157]:1
> 
> julia> methodswith(MyThing{1})
> 
> julia> bar(::MyThing{1}) = 2
> bar (generic function with 1 method)
> 
> julia> methodswith(MyThing)
> [1] foo(::MyThing) in Main at REPL[157]:1
> 
> julia> methodswith(MyThing{1})
> [1] bar(::MyThing{1}) in Main at REPL[160]:1
> 
> ```

As a result, because an object such as `[(a=1,)]` is of type `Vector{NamedTuple{(:a,), Tuple{Int64}}}`, none of the methods of `Vector` appear.

I’ve rewritten it so it’ll ~~detect if there are no methods of the parameterized type, and if so, give methods of the unparameterized type (with special case handling for vectors). I think it’ll have poor behavior in some particular case, but for a quick hack I think it’s not too bad. Might consider doing something more sophisticated later.~~ find methods of the parameterized type, and append methods of the unparameterized type with special case handling for vectors and matrices.

> **Here's the code now.**
>
> ```julia
> 
> function propose_method_autocompletions(obj, func_name_fragment::String=""; 
> type_depth = 1:typemax(Int), only_same_module = false, only_imported_modules = false,
> arg_order_inference = false, github_inference = false, personalized_inference = false)::Vector{Method}
> 
> @assert first(type_depth) ≥ 1 && last(type_depth) ≥ 1
> recs = Vector{Method}[]
> get_type_at_depth(type, depth=1) = depth ≤ 1 ? type : type == Any ? nothing : get_type_at_depth(supertype(type), depth-1)
> 
> for i ∈ type_depth
> stype = get_type_at_depth(typeof(obj), i)
> isnothing(stype) && break
> 
> meths = methodswith(stype)
> if !(stype isa UnionAll) && length(stype.parameters) > 0
> if stype <: AbstractVector # special case handling for vectors
> meths = [meths; methodswith(getfield(parentmodule(stype), nameof(stype)){T,1} where T)]
> elseif stype <: AbstractMatrix
> meths = [meths; methodswith(getfield(parentmodule(stype), nameof(stype)){T,2} where T)]
> end
> meths = [meths; methodswith(getfield(parentmodule(stype), nameof(stype)))]
> end
> 
> meths = filter(meths) do m
> length(func_name_fragment) > length(string(m.name)) && return false
> string(m.name)[1:length(func_name_fragment)] == func_name_fragment || return false
> only_same_module && (parentmodule(typeof(obj)) == m.module || return false)
> # How to detect only modules that have been explicitly imported?
> true
> end
> 
> arg_order_inference || true # do cool sorting stuff, add later
> github_inference || true # do cool sorting stuff, add later
> personalized_inference || true # do cool sorting stuff, add later
> 
> recs = [recs; meths]
> end
> 
> recs
> end
> 
> ```

Calling it on the object of `[(a=1,)]`:

```julia

julia> propose_method_autocompletions([(a=1,)])
[1] CapturedException(ex, bt_raw::Vector) in Base at task.jl:12
[2] \(A::LinearAlgebra.Transpose{<:Complex, <:LinearAlgebra.Hermitian{<:Complex, <:SparseArrays.AbstractSparseMatrixCSC}}, B::Vector) in SparseArrays at C:\Users\unime\AppData\Local\Programs\Julia-1.8.0\share\julia\stdlib\v1.8\SparseArrays\src\linalg.jl:872
[3] \(L::SuiteSparse.CHOLMOD.FactorComponent, b::Vector) in SuiteSparse.CHOLMOD at C:\Users\unime\AppData\Local\Programs\Julia-1.8.0\share\julia\stdlib\v1.8\SuiteSparse\src\cholmod.jl:1521
[4] append!(a::Vector, items::AbstractVector) in Base at array.jl:1105
[5] deleteat!(a::Vector, i::Integer) in Base at array.jl:1485
[6] deleteat!(a::Vector, r::AbstractUnitRange{<:Integer}) in Base at array.jl:1491
[7] deleteat!(a::Vector, inds::AbstractVector{Bool}) in Base at array.jl:1590
[8] deleteat!(a::Vector, inds::AbstractVector) in Base at array.jl:1533
[9] deleteat!(a::Vector, inds) in Base at array.jl:1532
[10] empty!(a::Vector) in Base at array.jl:1738

⋮

[3209] groupview(f, X; restype, kwargs...) in FlexiGroups at C:\Users\unime\.julia\packages\FlexiGroups\1ItB2\src\base.jl:41

```

Beyond that, for _truly_ generic methods that don’t specialize on _anything_, whose behavior can be thought of as really generalizable across _any_ object, they shouldn’t float to the top of such a simple autocomplete anyway; if a method is general enough not to specialize on any type, it’s probably general enough to have memorized.

It’s possible to make them float to the top anyway, using a sorting technique based on statistical inference of a model fitted to a codebase (imagine if you could train your autocomplete to your own codebase!), but I’m not going to make that at the moment.

Without such inference built-in, for methods to gain visibility, they need to be specialized to such types as `AbstractArray`, `AbstractDict`, or etc.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [November 10, 2022, 3:55am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/192 "2022-11-10T03:55:11Z")

</div>

> [@uniment](#):
>
> It’s possible to make them float to the top anyway, using a sorting technique based on statistical inference of a model fitted to a codebase (imagine if you could train your autocomplete to your own codebase!), but I’m not going to make that at the moment.

This would be super cool. IMHO compiler tooling should do a lot more of this kind of thing. I want some kind of data-driven approach to the JuliaSyntax.jl diagnostics system. What exactly, I’m not sure yet 🙂 I wish for some happy medium between traditional compilers (where all the diagnostic messages are hand-coded at huge engineering effort and are _still_ hard to interpret), vs hugely bloated natural language models which seem hard to train and deploy, but can perform amazing feats of inferring what the user intended.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 10, 2022, 4:26am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/193 "2022-11-10T04:26:44Z")

</div>

I think this is a relatively straightforward problem, conceptually anyway. One would wish to collect the frequencies in the codebase with which `Tᵢ` typed objects are fed to `mⱼ` methods. Then, when autocompleting methods for a `Tᵢ` object, simply sort the methods by order of call frequency.

I think this would probably be an optional mode: for example, one mode of autocomplete might be to sort based on type specialization; another mode might be to sort by frequency of use in your personal codebase; another mode might be to sort by frequency of use in all GitHub repos.

I think it’s probably decent to use the GitHub repo method use frequencies as a starting point, and then do some sort of weighted average with the frequencies they’re used in personal use. That way you don’t overfit to your own codebase.

It’s also possible to find the frequencies associated with certain packages being loaded (e.g., somebody who has loaded `LinearAlgebra` is more likely to call `qr`), and do a weighted average of the frequencies associated with the packages you’ve loaded in your project to get an autocomplete that’s more likely to be perfect for your specific use case. The possibilities are endless.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 10, 2022, 4:45am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/194 "2022-11-10T04:45:38Z")

</div>

> [@aplavin](#):
>
> A nice thing is that such autocomplete can be added without any changes to julia syntax: just expand it to `obj |> it->func(<cursor here>, it)` instead of `obj |> func(<cursor here>, _)`. So, it can already be implemented in a package using eg ReplMaker.

Indeed, although [as we’ve shown here](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/162), compile times for lambdas can get prohibitive if it’s done a lot, sometimes two orders of magnitude greater compile time than the alternative. Much better to have dedicated syntax to make a partial application functor.

It’s also simply cleaner to have dedicated syntax.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 10, 2022, 5:04am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/195 "2022-11-10T05:04:26Z")

</div>

I think the piping and the autocomplete are really orthogonal, though some might prefer certain syntax as a UI to autocomplete.

Imagine something like vscode if you could go to the Julia workspaces panel and just right click a variable and ask it to “explore methods for this variable” maybe it pops up a new panel with methods and you can filter them by characters you type into a search box as well as select which modules you want to search etc etc.

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 10, 2022, 4:11pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/196 "2022-11-10T16:11:35Z")

</div>

Cool idea!

I like that we’ve moved the conversation from “autocomplete is infeasible” to “this is how I would do it.” 😉

In the same way that `.` dot-syntax is syntax sugar that motivates tab-complete for property names, a proper chaining syntax motivates tab-complete for method names. Hopefully this should make idiomatic Julian style easier and more accessible.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [November 10, 2022, 4:29pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/197 "2022-11-10T16:29:00Z")

</div>

> [@uniment](#):
>
> I like that we’ve moved the conversation from “autocomplete is infeasible” to “this is how I would do it.” 😉

I don’t think people claimed “autocomplete is infeasible”, but we did give reasons for why the syntax proposal doesn’t really help the core problem and that the core problem can be solved without new syntax (as you’ve shown, by implementing a rudimentary autocomplete yourself).

I’m also pretty sure that [LanguageServer.jl](https://github.com/julia-vscode/LanguageServer.jl) already has that feature, provided it knows the type of the object. That is a requirement that hasn’t changed, as your code literally has `typeof(obj)` in it, which is not necessarily something an editor can do because the editor would potentially have to run your code to figure that out, which you surely don’t want (or run type inference on all of your code, which runs into the problem of “partial inference is not implemented”).

Not every piece of code will be executed in the REPL, where the object already exists (possibly…).

---

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 10, 2022, 4:51pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/198 "2022-11-10T16:51:32Z")

</div>

> [@Sukera](#):
>
> that the core problem can be solved without new syntax (as you’ve shown, by implementing a rudimentary autocomplete yourself).

My point was that the actual core problem was one of _motivation_: people need a _reason_ to work on an autocomplete, hopefully beyond just being an academic pursuit. Which syntax sugar provides. In the same way, somebody was motivated to write an autocomplete for `.` but not `getproperty`, despite the former being just sugar for the latter, in the confidence that the autocomplete on `.` would find heavy use and the effort would not go to waste.

> [@Sukera](#):
>
> That is a requirement that hasn’t changed, as your code literally has `typeof(obj)` in it, which is not necessarily something an editor can do because the editor would potentially have to run your code to figure that out, which you surely don’t want (or run type inference on all of your code, which runs into the problem of “partial inference is not implemented”).

With sufficient motivation, somebody smarter than me will figure this out I’m sure. In the meantime, I’d be pretty happy to have autocomplete in REPL mode.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [November 10, 2022, 10:22pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/199 "2022-11-10T22:22:18Z")

</div>

I don’t think it has yet been mentioned that many of the pipes in this thread can be implemented without a macro by using the [Transducers.jl](https://juliafolds.github.io/Transducers.jl/dev/) package. For example:

```julia
julia> using Transducers

julia> [4, 9, 16] |> Map(sqrt) |> Filter(iseven) |> collect
2-element Vector{Float64}:
 2.0
 4.0

```

In fact, I suspect that the Clojure threading macros (`->` and `-->`) that were mentioned in the original post were largely superseded by [transducers](https://clojure.org/reference/transducers) when transducers were introduced to Clojure. (That’s just a guess based on my limited knowledge of transducers in Clojure. I’m not a Clojure programmer.)

I need to study Clojure transducers and Transducers.jl more, but my current understanding is that the technically correct way to combine transducers is via function composition. So the piping syntax above is a bit of a pun. Clojure uses `compose` for this. For some reason, Transducers.jl uses “opposite compose”, rather than regular composition. A Unicode operator for “opposite compose” is provided, so the above example can be written like this:

```julia
julia> t = Map(sqrt) ⨟ Filter(iseven);

julia> collect(t, [4, 9, 16])
2-element Vector{Float64}:
 2.0
 4.0

```

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 10, 2022, 10:54pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/200 "2022-11-10T22:54:20Z")

</div>

I really need to take a closer look at transducers. I used them once to filter a lot of data read from big CSV files but I didn’t really understand how they were different from iteration

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [November 10, 2022, 11:07pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/201 "2022-11-10T23:07:02Z")

</div>

Can’t wait for the inevitable Transducers.jl x Diffractor.jl crossover episode

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [November 10, 2022, 11:43pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/202 "2022-11-10T23:43:27Z")

</div>

Transducers seem to create some Functors structs and then composition is by opcompose and that relies on Julia to do smart inlining? Because of the way they are composed it winds up producing some extremely efficient code is that a correct understanding?

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [November 10, 2022, 11:59pm UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654/203 "2022-11-10T23:59:03Z")

</div>

Nope, it’s actually not very reliant on a ‘sufficiently smart compiler’, it’s instead reliant on defining data traversal in terms of folds rather than iteration.

So for instance, if you write

```julia
f(x) = 2x
foldl(+, Iterators.map(f, filter(iseven, input))) 

```

with iterators, this becomes something like

```julia
function map_filter_iterators(xs, init)
    ret = iterate(xs)
    ret === nothing && return init
    acc = init
    @goto filter
    local state, x
    while true
        while true # input
            ret = iterate(xs, state) #
            ret === nothing && return acc #
            @label filter #
            x, state = ret #
            iseven(x) && break # filter :
        end # :
        y = 2x # imap : :
        acc += y # + : : :
    end # : : : :
    # + <-- imap <-------- filter <-- input
end

```

whereas the transducers equivalent

```julia
foldl(+, Filter(iseven) |> Map(f), input, init=0)

```

becomes

```julia
function map_filter_transducers(xs, init)
    acc = init
    # input -> Filter --> Map --> +
    for x in xs # input : : :
        if iseven(x) # Filter : :
            y = 2x # Map :
            acc += y # +
        end
    end
    return acc
end

```

This doesn’t rely on smart inlining or the compiler being clever at all to get that algorithmic shape.

[Previous page](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654.md?page=9)

[Next page](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654.md?page=11)