# DataFramesMeta.jl and the state of the DataFrames ecosystem
**URL:** https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221
**Category:** Data
**Created:** [March 19, 2020, 7:25pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221 "2020-03-19T19:25:49Z")
**Posts on this page:** 17
**Page:** 2
### Author:  [@jmboehm](https://discourse.julialang.org/u/jmboehm)
#### Post date: [March 21, 2020, 8:48pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/21 "2020-03-21T20:48:05Z")
> [@nalimilan](#):
>
> What could be done is that another macro called e.g. `@skipmissing` could be added to DataFrameMeta pipelines to automatically wrap columns in `skipmissing` , to avoid repeating it and thinking about where you need to put it.
I think that’s a good idea, though then it would make sense to have an equivalent `@completecases` too (and the name is probably not ideal-- it’s not immediately clear what the macro does).
> [@nalimilan](#):
>
> Overall, I encourage you to discuss these issues with us here or on DataFrames/DataFramesMeta GitHub issues rather than starting a new package. In the end everyone will benefit from having convenient APIs in a single package.
I agree. The point is not to have a better API to `DataFrames`. Rather, the point is to have an API that Stata users are very familiar with. I hope to feed some ideas into the design of the interface in DataFrames.jl, but am well aware that data manipulation habits are very persistent, and what is a good design for some may not be a good one for others (at least in the short run).
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [March 22, 2020, 6:53am UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/22 "2020-03-22T06:53:46Z")
> [@jmboehm](#):
>
> Rather, the point is to have an API that Stata users are very familiar with.
This may be true from your perspective, but please keep in mind that Julia users come from various backgrounds, which does not necessarily include Stata; also, Stata should not be considered an API to imitate.
Stata’s approach of missing values can be described as trying to guess what the uses wants in common cases. The problem with this is that this involves a lot of implicit assumptions and can lead to silent mistakes in analysis. A prominent example is missing values in logical statements: in simple cases people learn tricks like
```stata
gen var2 = (var < 10) if var !=.
```
but it is easy to forget about them for more complex code. Even seasoned Stata users make these mistakes all the time (and are usually unaware of it, unless the analysis is replicated with other programs).
Julia’s approach can be described as propagating `missing` unless the user deals with it explicitly. This may seem inconvenient at first glance, because some people expect that something “obvious” can be done to missing values. This is not true for any nontrivial code though: discussions reveals that what people consider the “right” approach can be very different. Also, this approach does not compose well.
Choosing how missing values are to be handled _explicitly_ leads to much cleaner code. It also meshes well with Julia’s design: in most cases it can be done at low or zero cost using various wrappers and iterators.
---
### Author:  [@jmboehm](https://discourse.julialang.org/u/jmboehm)
#### Post date: [March 22, 2020, 12:31pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/24 "2020-03-22T12:31:58Z")
Tamas, I agree with all your points. This is _exactly_ why I am hesitant to suggest changes to DataFrames.jl itself, but see value in a different interface.
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [March 22, 2020, 1:59pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/25 "2020-03-22T13:59:11Z")
On the one hand, I can see why having a familiar interface would be useful, and if you want to make a package like `StataDataFrames.jl` that wraps DataFrames with a familiar API, no one is going to stop you.
On the other hand, that approach could lead to a lamentable fragmentation of the ecosystem. One might imagine someone doing something similar for dyplr syntax, pandas syntax etc. In some ways, the presence of Query.jl ecosystem already represents such fragmentation, and that ecosystem uses a different paradigm for missing values etc.
I’m some ways, this is inevitable I suppose, especially as the community grows, and I don’t fault the Query folks for wanting to take a different approach. But, unless there’s something actually wrong with the DataFrames approach (as opposed to just being unfamiliar), it might be better in your case to write some Stata-\>DataFrames.jl cheat sheets or “getting started with DataFrames for Stata users” blog posts or something.
I really appreciate how thoughtful your posts have been, and I think this is a great discussion to have. I definitely think it’s worth learning from the things other languages do right. But it sounds like, for the most part, the DF1.0 interface is going to be able to do what you want, though perhaps with a bit more verbosity. It would be a shame IMO to write a wrapper around that for the sake of saving a couple lines of code.
---
### Author:  [@jmboehm](https://discourse.julialang.org/u/jmboehm)
#### Post date: [March 22, 2020, 2:26pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/26 "2020-03-22T14:26:41Z")
> [@kevbonham](#):
>
> On the other hand, that approach could lead to a lamentable fragmentation of the ecosystem.
I think this is an important issue, and it would be great if we could discuss this. I’m most grateful for all the work on DataFrames.jl and related packages, and I certainly don’t want to have a negative impact on their development (though, frankly, I doubt that I could!). My impression (also based on @floswald 's view) is that people that would prefer a Stata-like interface to DataFrames.jl are a small minority among Julia users. Hence, there would be no risk of a split in interface that the community uses. And, to re-iterate, I’m well aware of the fact that the Stata-type approach has deficiencies as well. I just think that one shouldn’t throw the baby out with the bathwater.
> [@kevbonham](#):
>
> it might be better in your case to write some Stata-\>DataFrames.jl cheat sheets
That’s actually how I started thinking of all this. But in the end I’m using all this to write papers, so I thought I might as well turn it into code that I can use.
---
### Author:  [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [March 22, 2020, 2:46pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/27 "2020-03-22T14:46:33Z")
Great thread, thanks all! Actually already learned some new tricks 🙂
Not much to contribute though over and above reiterating what I told @jmboehm offline: creating that stata-like interface for Dataframes.jl won’t do any harm.
I think some place with such tips and tricks would be a great resource. The tutorials by @bkamins are awesome for that, I should spend more time with them.
---
### Author:  [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [March 22, 2020, 3:02pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/28 "2020-03-22T15:02:15Z")
just for completeness and to be fair, you can do stuff like that in R.
1. You can always `paste` together a string exactly as you want and `eval` that
2. in particular, a `formula` can be given as just a string, so all regression stuff is easy to construct programmatically
3. slightly more advanced is `tidy` and `purrr` etc like [here](https://jessecambon.github.io/2019/12/08/practical-tidy-evaluation.html) for example.
of course none of this is proper metaprogramming will remain a hack forever.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [March 22, 2020, 4:22pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/29 "2020-03-22T16:22:36Z")
I’ve filed a github issue [here](https://github.com/JuliaData/DataFrames.jl/issues/2161) to discuss `skipmissing`-related improvements with the new `select` methods.
---
### Author:  [@floswald](https://discourse.julialang.org/u/floswald)
#### Post date: [March 22, 2020, 4:40pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/30 "2020-03-22T16:40:36Z")
Great! It would be good to collect an issue full of stata like statements like that which people think are useful?
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [March 22, 2020, 4:48pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/31 "2020-03-22T16:48:02Z")
The new `select` _just_ got merged yesterday. And `transform` is being worked on but has not gotten merged yet. People should check out master and play around and file issues as needed. We really want to hear your feedback before 1.0, but it’s important to understand existing features well first.
---
### Author:  [@matthieu](https://discourse.julialang.org/u/matthieu)
#### Post date: [April 12, 2020, 5:12pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/32 "2020-04-12T17:12:20Z")
Experimenting with a REPL mode for data cleaning/analysis in Julia would be interesting. I wonder if it has advantages compared to macros etc.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 12, 2020, 7:24pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/33 "2020-04-12T19:24:09Z")
you have column names autocompletion for data frames
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [April 12, 2020, 7:52pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/34 "2020-04-12T19:52:49Z")
this could be cool. a more stata-esque workflow. It would probably be a very hard package to write but it’s feasible using something like [HeaderREPLs.jl](https://github.com/timholy/HeaderREPLs.jl).
---
### Author:  [@jmboehm](https://discourse.julialang.org/u/jmboehm)
#### Post date: [April 12, 2020, 8:10pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/35 "2020-04-12T20:10:28Z")
In some ways that’s what I have in mind and what I’m trying to work on ([GitHub - jmboehm/Douglass.jl: Stata-like toolkit for data wrangling on Julia DataFrames](https://github.com/jmboehm/Douglass.jl) but progress is slow; I doubt that I’m the most qualified person for this). First step is to write a parser, then implement the commands, then the REPL mode.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [April 13, 2020, 9:47am UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/36 "2020-04-13T09:47:43Z")
I _love_ the REPL idea. If someone wanted to put that ontop of [Query.jl](https://github.com/queryverse/Query.jl), I’d be happy to help with advice and guidance.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [April 16, 2020, 1:19pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/37 "2020-04-16T13:19:33Z")
An update on this, the following just got merged into DataFrames master:
EDIT: This was broken earlier but is fixed now.
```julia
julia> by(df, [:id, :val], :, :income => mean)
```
Notice the `:` in the function call. This preserves all existing columns. The resulting mean of income by group will be “spread”, so to speak, across each group.
This is exactly equivelent to
```julia
bysort id val: egen income_mean = mean(income)
```
in Stata. And imo pretty much just as pretty.
---
### Author:  [@jack\_rabbit](https://discourse.julialang.org/u/jack_rabbit)
#### Post date: [April 24, 2020, 1:18pm UTC](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221/38 "2020-04-24T13:18:34Z")
Just a small fix, the syntex should be
```julia
julia> by(df, [:id, :val], :, :income => mean)
```
See [here](https://github.com/JuliaData/DataFrames.jl/issues/2208) for an example.
[Previous page](https://discourse.julialang.org/t/dataframesmeta-jl-and-the-state-of-the-dataframes-ecosystem/36221.md?page=1)