[RFC] PipelessPipes.jl (now Chain.jl)

this one.

This is exactly what I want! I love it!

1 Like

This looks great! I think the move from @_ to @chain improves the clarity. Along the same lines, is @! set in stone? In Julia, I usually associate ! with something mutating in place but the usage here feels pretty different to me. Maybe @hop , @skip , @over , or some other short descriptive word?

4 Likes

Love the idea reminds me of R.

Weird questionā€¦ would it be possible to emit intermediate values in say a let-block? Maybe something like: @emit

Iā€™ve seen tee used for this elsewhere ā€“ e.g. at the shell, tee writes output to a file but also passes it on to the next step in the pipeline, and R has the ugly but useful ā€œtee pipeā€ %T>% for the same thing. So maybe @tee.

2 Likes

Nothingā€™s really set in stone, Iā€™m always open for good arguments. I thought that @! was short and conveyed something like attention, unusual, out of the ordinary. I didnā€™t connect it with mutation, but if that seems confusing I could change to something like @skip. For me personally, @tee doesnā€™t convey any meaning, but Iā€™m not an R user :slight_smile:

4 Likes

tee is from the Linux/Posix command I would say. I thought also about @aside, but @tee should also be meaningful for a Linux user ?

I donā€™t quite understand the question, could you explain further?

Iā€™m also not familiar with the tee syntax but from a quick search it seems to imply writing to std out and/or files, which is not necessarily the case with the current @!.

Iā€™m not sure if mutating was the right word choice here, I was referring to this part of the style guide about functions like push! that modify their arguments. Given the popularity of @. for broadcasting I think it might be common for people to expect @! to make an entire line modify arguments in some way. Either way, itā€™s a pretty minor syntax point in my opinion. I really like the overall feel!

I guess I hadnā€™t thought so much about a valid use case of diverting the pipeline stream to some other target, like tee does. In that case it would make sense to choose a more descriptive word. I had envisioned @! more for temporary debugging. So the proposals so far are:

@skip
@hop
@over
@aside
@tee

Any other ideas? Given that you can do anything after this macro, not just divert into a different sink, I wonder if a more general word is better

1 Like

@without? I also like @skip, fwiw.

maybe @also? I wonder if @skip sounds a bit like, skip that part altogether

Yeah, I think @skip is a little confusing, but @also seems pretty good.

or maybe @btw for by the way, although maybe not everybody knows that acronym

1 Like

I would go for @tee: itā€™s really the same concept, and itā€™s well established. The name is also distinctive and short.

The picture on tee (command) - Wikipedia explains the origin of the name: the concept is to add a ā€œTā€ in a pipeline, that makes the data available to two consumers.

It looks oddly specific to talk about files, but files are a very general concept in Unix. Here is a valid pipeline in Linux (Bash):

ls /usr |
tee >(grep ^lib | xargs du -hs >&2) |
grep ^s

In English:

Show the content of the /usr directory
Pass a copy to a sub-pipeline that keeps only entries starting with `lib` and prints their size to the second output
Pass the original data to `grep`, which keeps only entries starting with `s` and prints them to the first output

The output on my computer is:

sbin
share
src
7.7G	lib
17M		lib32
4.0K	lib64
13M		libexec
4.0K	libx32

The first three lines are from the main pipeline (first output). The last 6 lines are from the sub-pipeline (second output). (By default, both outputs are shown in the terminal.)

The trick in this example is >(...), which creates a sub-pipeline that can be written to, like a file. Many things behave like ā€œfilesā€ in Unix so tee is really about passing data to two consumers.

Isnā€™t this very similar to what we want here?

1 Like

To me, @tee indicates something like youā€™re splitting the chain into two branches, each of which you can pass through a sequence of commands. But it seems like @! doesnā€™t give a whole subchain (in which you can chain outputs together) but rather a single command that doesnā€™t participate in the chain.

I think you could actually do a whole chain, but at some point it might not be worth using @chain at all when it gets too complexā€¦

In principle, this would work:

@chain 1:5 begin
    _ .^ 2
    @tee @chain begin
        sum
        println("The side result is $(_)")
    end
    diff
end
2 Likes

One could of course also just do @T if tee stands for that anyway

Yes, in complex cases I would rather write @tee myfun(_) and implement the second pipeline in myfun.

Regarding @T: indeedā€¦ Iā€™m probably partial to tee because of my experience in shell programming. I find it more distinctive than @T (itā€™s a ā€œrealā€ word with a meaning). So itā€™s nice for people who already know tee. But itā€™s also nice the other way: learning @tee in Julia might help someone understand a shell pipeline one day :slight_smile:

I vote for @aside, itā€™s meaningful and the most clear and transparent to me.

3 Likes