# \[ANN\] DataPipes.jl 0.3.0

**URL:** <https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734>\
**Category:** Package Announcements\
**Tags:** data, piping\
**Created:** [May 7, 2021, 5:29pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734 "2021-05-07T17:29:26Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [July 8, 2021, 1:59am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/42 "2021-07-08T01:59:38Z")

</div>

It’s worth noting that this exact same `DataPipes.mutate` syntax works with `Underscores.@_` (and has done since shortly after it was released). So this is further evidence that these packages are very similar in design. If we could agree on a general solution for implicit `__` vs `↑` and naming of arguments, it may be that these packages can join together in some way.

```julia
julia> using TypedTables
julia> t = Table(a=[1,2,3], b=[4,5,6]);

julia> @_ mutate(exp_a_square=exp(_.a^2), a_square=_.a^2, t)
Table with 4 columns and 3 rows:
     a b exp_a_square a_square
   ┌─────────────────────────────
 1 │ 1 4 2.71828 1
 2 │ 2 5 54.5982 4
 3 │ 3 6 8103.08 9

julia> @_ mutate(comp_val=_.a > _.b ? _.a^2 : _.a, t)
Table with 3 columns and 3 rows:
     a b comp_val
   ┌───────────────
 1 │ 1 4 1
 2 │ 2 5 2
 3 │ 3 6 3

```

`mutate` seems pretty handy. Perhaps something like it could go into `SplitApplyCombine`?

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [July 8, 2021, 3:41am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/43 "2021-07-08T03:41:02Z")

</div>

`mutate` seems straight from [dplyr](https://dplyr.tidyverse.org/); if a package is going to go that route I think it would be good to look more comprehensively at the dplyr API so Julia’s version can get a cohesive look-and-feel.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [July 8, 2021, 6:22am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/44 "2021-07-08T06:22:15Z")

</div>

> [@c42f](#):
>
> So this is further evidence that these packages are very similar in design. If we could agree on a general solution for implicit `__` vs `↑` and naming of arguments, it may be that these packages can join together in some way.

Totally agree that there is a significant overlap, and short single-step examples work completely the same in these packages (maybe even in `Chain.jl`).  
I myself don’t see a general and still no-boilerplate interpretation of the differences, but would be very curious to know if there is any. DataPipes clearly implements a less general approach, but is more convenient for piped data analysis (hence the name (: ). Also, there are pipe-related features on top, like `@export` macro, and I also plan to add `@aside` macro like in `Chains.jl`. They don’t seem fit for a really general `_`-package like `Underscores.jl`, but I may be mistaken here.

> [@c42f](#):
>
> `mutate` seems pretty handy. Perhaps something like it could go into `SplitApplyCombine` ?

Currently, this function (and some other short ones) is defined in `DataPipes`, but not documented. The reason is I don’t know where it is best to put them, and they may be changed/removed at any time. Maybe you are right and `SAC.jl` is the right place for them to go…

> [@jzr](#):
>
> `mutate` seems straight from [dplyr](https://dplyr.tidyverse.org/); if a package is going to go that route I think it would be good to look more comprehensively at the dplyr API so Julia’s version can get a cohesive look-and-feel.

I agree with Chris, that specific functions like `mutate` should really be out of scope of `DataPipes` and similar packages. Don’t know or use dplyr myself, but in would be interesting to see someone attempt to implement a similar interface in Julia, if there is none yet. Currently available functions (Base, SAC.jl, …) may be less “cohesive”, but are more general than dplyr and dataframes.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [July 8, 2021, 7:31am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/45 "2021-07-08T07:31:24Z")

</div>

> [@aplavin](#):
>
> DataPipes clearly implements a less general approach, but is more convenient for piped data analysis (hence the name (: ). Also, there are pipe-related features on top, like `@export` macro, and I also plan to add `@aside` macro like in `Chains.jl` . They don’t seem fit for a really general `_` -package like `Underscores.jl` , but I may be mistaken here.

That’s true, there’s some things which will only apply to pipes but are super handy such as having variables for partial results assigned within the pipeline.

Underscores.jl actually does have a small accommodation for `|>` syntax (also `∘`, `<|`, `.|>` and `.<|`), but only in the sense that it recurses into such expressions and applies the same `_` replacement rules inside them, rather than treating them as normal call expressions.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [July 8, 2021, 7:44am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/46 "2021-07-08T07:44:11Z")

</div>

> [@c42f](#):
>
> Underscores.jl actually does have a small accommodation for `|>` syntax (also `∘` , `<|` , `.|>` and `.<|` )

Interesting, especially the reverse pipe. Do you have any neat examples with these?

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [July 8, 2021, 10:55am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/47 "2021-07-08T10:55:45Z")

</div>

> [@aplavin](#):
>
> reverse pipe. Do you have any neat examples with these?

Not with the reverse pipe! I think `<|` was added at the suggestion of someone else. Maybe in the thread at [ANN: Underscores.jl: Placeholder syntax for closures](https://discourse.julialang.org/t/ann-underscores-jl-placeholder-syntax-for-closures/36482)

Composition is normal composition. Like in your `@f` macro, but you don’t need a separate macro:

```julia-auto
@f map(_^2) |> filter(exp(_) > 5)

# vs

@_ map(_^2, __) ∘ filter(exp(_) > 5,__)

```

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [July 8, 2021, 2:23pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/48 "2021-07-08T14:23:18Z")

</div>

That’s a clever and general solution!  
Indeed, having a pure syntax indication such as `__` in the first pipe step helps distinguish between a function definition and application.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [July 9, 2021, 4:08am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/49 "2021-07-09T04:08:17Z")

</div>

> [@aplavin](#):
>
> having a pure syntax indication such as `__` in the first pipe step helps distinguish between a function definition and application.

Yes it kind of neatly falls into place in this case.

Another approach to “the problem of `__`” is to define versions of common functions which return a function:

```julia-auto
Base.map(f::Function) = xs->map(f, xs)
Base.filter(f::Function) = xs->filter(f, xs)

```

Then we’d have

```julia-auto
julia> data = 5:12
5:12

julia> @_ data |>
           filter(_>10) |>
           map(_^2)
2-element Vector{Int64}:
 121
 144

```

This is type piracy, of course! But this version of `Base.filter` isn’t defined, and though `Base.map(f::Function)` is defined the existing definition of mapping over zero collections seems pretty useless.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [July 9, 2021, 8:34am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/50 "2021-07-09T08:34:58Z")

</div>

Funny, this is already implemented in Transducers.jl (all the more reasons to use this awesome package)

```julia
using Transducers
using Underscores

julia> data = 5:12
5:12

julia> @_ data |>
           Filter(_>10) |>
           Map(_^2) |>
           collect
2-element Vector{Int64}:
 121
 144

```

and no type piracy here.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [July 9, 2021, 11:41am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/51 "2021-07-09T11:41:58Z")

</div>

Yes Transducers.jl is really cool for many reasons.

But sometimes you just want to do some quick data processing without any extra dependencies, which is why I kind of wish we had versions of normal `map()` and `filter()` as above.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [September 17, 2021, 4:17pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/52 "2021-09-17T16:17:43Z")

</div>

Out of my habit, I find it easier to write/read code where the functions are embedded (# 2) rather than the form # 1.  
But I like the ability to shorten the syntax.  
The first form I tried was # 5, which doesn’t work. Then I found the other shapes that give the expected result, but I’m not sure I understand why # 5 doesn’t work and # 4 does.  
But maybe there is an even more correct way to get what I was looking for.

```julia

@p 1:4 |> map(_^2) |> filter(exp(_) > 5) #1

filter(x->exp(x)>5,map(x->x^2,1:4)) #2

@p filter(exp(_)>5, @p begin map(_^2,1:4) end ) #3

@p filter(exp(_)>5, @p(map(_^2,1:4))) #4

@p filter(exp(_)>5, @p map(_^2,1:4) ) #5

```

I would like to have some considerations on the different efficiency of the various forms

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 18, 2021, 3:10am UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/53 "2021-09-18T03:10:46Z")

</div>

All these five variants work for me and give exactly the same results with DataPipes.jl 0.1.7. Could you please share the error you are getting?

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [September 18, 2021, 12:44pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/54 "2021-09-18T12:44:10Z")

</div>

> [@rocco\_sprmnt21](#):
>
> But I like the ability to shorten the syntax.

Maybe you could get rid of `map` and broadcast, for example (_respecting the number of steps)_:  
`@p 1:4 .|> _^2 |> filter(exp(_)>5)`

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [September 18, 2021, 12:51pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/55 "2021-09-18T12:51:37Z")

</div>

ok. I have the same version (downloaded yesterday) and now I get the same results as you. I don’t know what to think.  
I apologize for the wrong report.  
I had initially tried with the expression

```julia
@p filter(exp(_)>5, map(_^2,1:4) ) 

```

which I then corrected in form # 5 and since it seemed to me (perhaps confused with the initial form) that this didn’t work, so I tried first with # 3 and then with # 4.

Could you in this case use the nested form with the placeholder \_1?

PS  
is it possible to retrieve the log of the outputs (the one of the inputs I have) of yesterday’s session made in the vscode environment?

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 18, 2021, 1:42pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/57 "2021-09-18T13:42:03Z")

</div>

> [@rocco\_sprmnt21](#):
>
> Could you in this case use the nested form with the placeholder \_1?

The `_1` placeholder has a very different semantics now, don’t think it is possible to combine them somehow.  
And explicit delimiters (like `@p`) are necessary anyway with the nested function style you use. Otherwise the meaning would be ambigous.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [September 18, 2021, 2:16pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/58 "2021-09-18T14:16:38Z")

</div>

Thanks for taking the time.  
Maybe I’m still missing something to understand how @p works.  
If I did a brutal substitution of f (\_) with x-\> f (x), then again the naive form I used first would work.  
Even using the same variable name, since the scope is different

```julia
 filter(x->exp(x)>5, map(x->x^2,1:4) )

```

Instead from the following two tests it seems that @p acts, in case of nested functions only at the high level.

```julia
julia> @p filter(exp(_)>5, map(x->x^2,1:4) )
3-element Vector{Int64}:
  4
  9
 16

julia> @p filter(x->exp(x)>5, map(_^2,1:4) )
ERROR: MethodError: no method matching filter(::var"#137#139", ::var"#138#140")

```

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 18, 2021, 3:28pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/59 "2021-09-18T15:28:55Z")

</div>

Unfortunately, the naive replacement you suggest doesn’t really work well in general. The main issue is to determine function boundaries: what does `a(b(c(_), _), d(e(f(_) + _)))` get converted to?  
That’s why `DataPipes.jl` takes the approach of converting only top-level function arguments (containing `_`) to anonymous functions.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [December 12, 2021, 7:58pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/60 "2021-12-12T19:58:07Z")

</div>

In the meantime, `DataPipes.jl` has got a significant new feature, and the placeholder style changed to a more convenient one.

You can continue using the old syntax in 0.1.x versions, or switch to the 0.2 version with the updated placeholders. Future features will only appear in 0.2 and won’t be backported.

# Improved docs

The README at [Alexander Plavin / DataPipes.jl · GitLab](https://gitlab.com/aplavin/DataPipes.jl) is now shorter and doesn’t go into depths. A more detailed documentation together with a set of worked out data processing tasks is available as a Pluto notebook: see [HTML version](https://aplavin.github.io/DataPipes.jl/examples/notebook.html).

# New feature: `@aside` macro

_Available in both 0.1 and 0.2 versions._  
Perform a side computation without breaking the pipeline:

```julia-auto
@p begin
	data
	@aside avg = mean(_.age)
	map((; _.name, _.age, above_average=_.age > avg))
end

```

Also plays nice with `@export` to export the variable to the outer scope:

```julia-auto
@p begin
	data
	@aside @export avg = mean(_.age)
	map((; _.name, _.age, above_average=_.age > avg))
end

# avg is available here

```

_Idea for the `@aside` macro is taken from Chain.jl_

# New placeholder syntax

Old `DataPipes@0.1`:

- `_` - first/only lambda argument
- ` __`, `___ `, … - second, third, and further lambda arguments
- `↑` - result of the previous step
- `_1` - lambda argument `_` of the outer pipe

New `DataPipes@0.2`:

- _`_` - first/only lambda argument, same as before_
- `_2`, `_3`, … - second, third, and further lambda arguments (_`_1` also works, and is equivalent to `_`_)
- `__` - result of the previous step
- `_ꜛ` - lambda argument `_` of the outer pipe (type the arrow with `\^uparrow`)

Motivation:

- Referring to the previous step result turned out to be more common than I originally thought. The old `↑` symbol is more difficult to type compared to the new `__`, which is important for common operations. Also, `↑` parses as an operator name in Julia, which required putting it into brackets sometimes.
- `_2` instead of ` __` is about the same typing effort, and `__ ` got already occupied in the previous bullet.
- `_ꜛ` is somewhat more difficult to type, but accessing the outer pipe argument is needed much less often in my experience. So, a compromise here is acceptable. Still, `_ꜛ` parses as a regular name in Julia, and doesn’t require extra parens unlike the old `↑`.  
Suggestions of other placeholder to replace `_ꜛ` are welcome!

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [December 13, 2021, 8:00pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/61 "2021-12-13T20:00:33Z")

</div>

Didn’t think I’ll push another significant update soon, but here it is (:  
_upd: DataPipes@0.2.1 is registered in General._

I perform essentially all data manipulation tasks with `DataPipes`, and didn’t encounter many pain points with it. Still, there are a couple of common scenarios that can be made cleaner with less boilerplate. They mostly revolve around working with nested data, and now I addressed some of these scenarios.

A common pattern is lambda functions consisting only of inner “pipes” (`@p`), especially with the `map` function. Like this simple parsing of a key-value string into a named tuple:

```julia
@p begin
	"a=1 b=2 c=3"

	split
	map() do kv
		@p begin
			split(kv, '=')
			Symbol(__[1]) => parse(Int,__[2])
		end
	end
	NamedTuple
end

```

Now, it has a more succinct syntax in `DataPipes`: the lambda function body is automatically wrapped with an inner pipe when the only argument is `__` (double underscore). The intuition is that `__` refers to the previous pipeline step in `DataPipes`, and by assigning to `__` we effectively start a new pipe.  
Here is the same example using the new feature:

```julia
@p begin
	"a=1 b=2 c=3"

	split
	map() do __
		split(__, '=')
		Symbol(__[1]) => parse(Int,__[2])
	end
	NamedTuple
end

```

Essentially, we got rid of one nesting level and the `@p begin end` boilerplate.

_Idea that such nesting can be simplified is taken from a post on Julia slack. Unfortunately, cannot find that post anymore._

---

<div class="post-metadata">

**Author:** ![Maximilian\_Roos](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maximilian_roos/32/18910_2.png) [@Maximilian\_Roos](https://discourse.julialang.org/u/Maximilian_Roos)\
**Post date:** [December 13, 2021, 10:57pm UTC](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734/62 "2021-12-13T22:57:09Z")

</div>

@aplavin this looks great!

> [@aplavin](#):
>
> The README at [Alexander Plavin / DataPipes.jl · GitLab](https://gitlab.com/aplavin/DataPipes.jl) is now shorter and doesn’t go into depths. A more detailed documentation together with a set of worked out data processing tasks is available as a Pluto notebook: see [HTML version](https://aplavin.github.io/DataPipes.jl/examples/notebook.html).

Is the Readme up to date there? I see it was created 11 months ago; or am I looking at the wrong thing?

[Previous page](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734.md?page=2)

[Next page](https://discourse.julialang.org/t/ann-datapipes-jl-0-3-0/60734.md?page=4)