# DataFrames.jl development survey
**URL:** https://discourse.julialang.org/t/dataframes-jl-development-survey/44022
**Category:** Data
**Tags:** question, dataframes
**Created:** [July 31, 2020, 7:49am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022 "2020-07-31T07:49:21Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [July 31, 2020, 6:32pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/21 "2020-07-31T18:32:41Z")
Just to pile on to my previous comment, making DataFrame operations nearly as performant as operations on Vectors or Arrays would be I think a dream outcome. For example, I am doing an optimization problem involving simulating panels of data. There is a nontrivial cost to doing calculations on this when the simulated data is in a DataFrame versus the approach where keep everything in arrays like I’m a MATLAB-loser. Of course it is much nicer to write the DataFrames-style code, which is a huge plus, but when the performance gap is hit millions of times it starts to really add up.
It is all still faster than MATLAB anyway so why am I even complaining?
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [July 31, 2020, 6:44pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/22 "2020-07-31T18:44:16Z")
In any place where you hit this kind of performance bottleneck, use one of (depending on the use case):
1. a function barier extracting the columns you need
2. `Tables.columns`
3. `Tables.namedtupleiterator`
to get the columns you need to do the computation on fast.
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [July 31, 2020, 7:17pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/23 "2020-07-31T19:17:47Z")
Just make it as fast as possible.
---
### Author:  [@anon92994695](https://discourse.julialang.org/u/anon92994695)
#### Post date: [August 1, 2020, 11:47am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/24 "2020-08-01T11:47:46Z")
Thank you for all the hard-work you’ve put into this package - I use it pretty much daily and love it.
To me the order of priorities goes like this:
Make a thing → make that thing fast → make it easy to use → make it modular for developers
I see a lot of workflows like the following
Make a thing fast → make it modular → now I have a thing
I think the first workflow is best for “utility” packages ie ones that provide essential tools used across a variety of fields and that newcomers immediately install or are interested in. I see the second workflow as best for the developers who want to build off of your work to strengthen Julia itself.
I am torn about the “utility” functions question… Part of me thinks its ridiculously important, especially for people who haven’t been monitoring DataFrames.jl development, for all the goodies to be in one place. Someone quickly groking some packages will see DataFrames.jl and know they need that, but they won’t neccessairily know they need something like “DataFramesUtils.jl” unless they read about and understand the design decisions. Which not to be negative - most people using the package probably don’t want to do that - people working on a similar package definitely would. And then there’ll be really cringey pythonista blog posts, by yes people not as smart, about how “slow julia is because you have to compile 10 packages to join and filter on 2 csvs”.
That being said I know the julian way of building packages is to make them as modular as possible while still having each unit retain its meaning. But sometimes I do worry what that does to readability of codebases. No one wants to jump across 5 packages to see if they can use your code (not talking about developing off of your code), finding documentation, and increased surface area for things to go wrong.That whole death by abstraction thing (ex: [https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition](https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition))…
---
### Author:  [@dlakelan](https://discourse.julialang.org/u/dlakelan)
#### Post date: [August 1, 2020, 2:35pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/25 "2020-08-01T14:35:45Z")
> [@pdeffebach](#):
>
> Not requiring `:x` and letting users write `x`
No no no no no
writing `x` means and always should mean “the value of the variable x” so
```julia
x="foo"
mydf[!,x]
```
means and ALWAYS should mean grab column `foo`
this is an anti-pattern in R that it uses the text you type instead of the values of variables which makes it nearly impossible to reason about any serious R code
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [August 1, 2020, 3:00pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/26 "2020-08-01T15:00:26Z")
This is true! I definitely agree with you about the perils of using tidy evaluation in this way and it is a _constant_ source of frustration for me in R.
The use of literals would _only_ apply inside DataFramesMeta macros, i.e. `@transform`, `@select`, `@with`, and `@byrow`. Therefore readers of the code will always be alerted to the presence of non-standard evaluation with a macro.
Second, we would work very hard to ensure that
1. It is 100% as easy to use `x = "foo"` as a column name as it is to use code literals. Testing will ensure that there is feature parity between the two and that escaping rules are far more clear than R’s `quo` and `enquo`.
2. It is always obvious when something is a local variable that is _not_ a column name. Perhaps via `$` or some other Julia convention.
Maintaining these rules and clarity is very important. It is in my mind the major benefit that we can implement in Julia over R. If I can’t find a way to get consistent escaping rules, then this change won’t get implemented.
Nonetheless, I often use Parameters.jl’s `@unpack` and `@pack` macros when working with data in Julia simply because I find using `Symbol`s and indexing cumbersome. Users have routinely complained about the verbosity of DataFrames, including the use of `:` in certain places.
Finally, note that we currently have a (manageable) level of ambiguity about code literals being used in DataFames with `df.x` always returning the column `:x` no matter what the variable `x` represents.
---
### Author:  [@dlakelan](https://discourse.julialang.org/u/dlakelan)
#### Post date: [August 1, 2020, 4:12pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/27 "2020-08-01T16:12:52Z")
> [@pdeffebach](#):
>
> The use of literals would _only_ apply inside DataFramesMeta macros, i.e. `@transform` , `@select` , `@with` , and `@byrow` . Therefore readers of the code will always be alerted to the presence of non-standard evaluation with a macro.
Yes, this I can get on board with. Inside a macro code means something else than outside a macro, and that’s always true, so the only way to know how a thing works is to read the docs on the macro. I’m ok with that. The problem is in R that you never know where nonstandard evaluation will just appear, there’s no indication in the text of the code.
> [@pdeffebach](#):
>
> It is 100% as easy to use `x = "foo"` as a column name as it is to use code literals
Exactly. I can’t tell you how many times I’ve been bitten by trying to use the value of a variable in a plot or a data summarization or whatever and instead getting a blank plot because the name of the variable was used instead of the **value** of the variable… it’s a disaster in R, one of the straws that broke the camels back for me regarding R, and it’s gotten progressively worse as Hadley has polluted the R ecosystem with weird evaluation.
consider in ggplot2 `aes()` vs `aes_()` vs `aes_str()`
we’ve had like 10 years of people futzing with this, no one I know is really sure how to make any given thing they’re trying to do work, except the very simplest thing like ggplot(foo, aes(x,y)) … only to find that if you get the help on aes\_ that
> Life cycle:
>
> ```
> All these functions are soft-deprecated. Please use tidy
> evaluation idioms instead (see the quasiquotation section in
> ‘aes()’ documentation).
>
> ```
> [@pdeffebach](#):
>
> we currently have a (manageable) level of ambiguity about code literals being used in DataFames with `df.x` always returning the column `:x` no matter what the variable `x` represents.
But that’s how it’s supposed to work right? I mean
```julia
struct Foo
x::Int
end
x="bar"
foo = Foo(1)
foo.x
```
should always return the x field of foo, not throw an error “There is no field named bar in a struct Foo”
So this is more or less the way it works everywhere else. I’m going to have to go watch the talk on DataFrames indexing though. I had intended to do that all along, but need to make the time. Glad they’re all recorded!
Also thanks all for the work making DataFrames extremely useful. I’m opinionated here, but not ungrateful!
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [August 1, 2020, 4:39pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/28 "2020-08-01T16:39:55Z")
I absolutely agree with you about ggplot, I even asked discourse in R about it [yesterday](https://community.rstudio.com/t/most-up-to-date-way-to-use-ggpot-with-variables-representing-strings/74985/4)!
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 19, 2020, 2:13am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/29 "2020-09-19T02:13:33Z")
> [@tbeason](#):
>
> For example, I am doing an optimization problem involving simulating panels of data
Would matrix work for u?
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [September 19, 2020, 2:54am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/30 "2020-09-19T02:54:24Z")
I actually did switch back to the “manual” method using matrices, for performance reasons.
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [September 20, 2020, 10:48pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/31 "2020-09-20T22:48:23Z")
How much performance potential still untapped for DataFrames? Is there any hope to catch up with data.table?
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [September 21, 2020, 6:02am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/32 "2020-09-21T06:02:47Z")
There are different cases: in some we are slower, in some we are on par, in some we are faster. Definitely where we are slower is time of a first run. In order to reduce it you need to build a custom system image.
Now - in general, in cases where we are much slower (joins) a faster implementation is currently under development. There are also incremental improvements in split-apply-combine.
Finally - we currently do not do multi threading, as data.table does. It is also planned to be implemented in the future.
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [September 21, 2020, 9:03am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/33 "2020-09-21T09:03:13Z")
Are there no cases in which we are faster? 😉
(I assume/hope there’s a typo in your first sentence!)
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [September 21, 2020, 12:48pm UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/34 "2020-09-21T12:48:55Z")
Yes - typo. In some we are faster
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [September 27, 2020, 1:06am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/35 "2020-09-27T01:06:24Z")
I still find DataFrames quite verbose. Not as verbose as pandas but more verbose than data.table.
For example, `df[df.sex .== "male", :]` would be `df[sex == "male"]` in data.table. Within the DataFrame environment, there is no confusion between df.sex and sex. I am OK with `df[:sex .== "male"]` .
Also, the filter function seems very strange, because it takes a DataFrame on the second argument while the other functions take it on the first argument.
For example,
```julia
@pipe df |>
filter(:sex => ==("male"), _) |>
groupby(_, :pclass) |>
combine(_, :age => mean)
```
This just does not seem consistent.
If all the functions take a DataFrame on the first argument, then it would be:
```julia
@pipe df |>
filter(_, :sex => ==("male")) |>
groupby(_, :pclass) |>
combine(_, :age => mean)
```
Now it looks more consistent. If in the future it can be reduced to:
```julia
@pipe df |>
filter(:sex => ==("male")) |>
groupby(:pclass) |>
combine(:age => mean)
```
There is no confusion.
To me, I hope it can be reduced to:
@pipe df |\>
filter(:sex .== “male”) |\>
groupby(:pclass) |\>
combine(mean(:age))
Or even better, just make |\> a function of DataFrames, so people do not need to type @pipe every time they want to use |\>.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 27, 2020, 2:19am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/36 "2020-09-27T02:19:59Z")
> [@Yifan\_Liu](#):
>
> df[:sex .== “male”]
these kinda syntax can only be achieved macros
```julia
@dt df[sex == "male"]
```
but no one’s written the `@dt` macro yet.
you can do `@where` from DataFramesMeta so
```julia
@where(df, :sex .== "male")
```
> [@Yifan\_Liu](#):
>
> the filter function seems very strange, because it takes a DataFrame on the second argument while the other functions take it on the first argument.
This has been discussed elsewhere, you can load `using DataConvenience` instead of use the `@where` macro.
> [@Yifan\_Liu](#):
>
> @pipe df |\>
> filter(:sex .== “male”) |\>
> groupby(:pclass) |\>
> combine(mean(:age))
Just use `DataConvenience.jl` or `using Lazy: @>`
then you can do
```julia
using DataConvenience: filter, @>
using DataFramesMacro: @based_on
@> df begin
filter(:sex .== “male”)
groupby(:pclass)
@based_on(mean(:age))
end
```
> [@Yifan\_Liu](#):
>
> ust make |\> a function of DataFrames,
this was discussed elsewhere and the proposal was to make `groupby` etc into curried versions but it was voted down.
I also voted it down, because you can just use a macro.
The macrosystem in Julia is not as convenient as R’s as it makes a distinction between macro and normal function whereas in R every function has the potential to be a macro.
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [September 27, 2020, 2:41am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/37 "2020-09-27T02:41:06Z")
Maybe I am wrong, but this extra verbosity does not seem to bring any clarity, performance, or consistency.
I mean no offense, but the workaround you mentioned:
```julia
using DataConvenience: filter, @>
using DataFramesMacro: @based_on
@> df begin
filter(:sex .== “male”)
groupby(:pclass)
@based_on(mean(:age))
end
```
seems really strange. The mixture of macros and functions makes the syntax really strange.
I really appreciate the efforts by the DataFrames team, but I think I will stick to DataFramesMeta. To be honest, the built-in functions in DataFrames confuse me a lot.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 27, 2020, 2:54am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/38 "2020-09-27T02:54:53Z")
> [@Yifan\_Liu](#):
>
> The mixture of macros and functions makes the syntax really strange.
I agree. That is a problem. So I think DataFramesMeta should introduce an all macros approach so it becomes
```julia
using DataConvenience: @>
using DataFramesMacro
@> df begin
@filter(:sex .== “male”)
@groupby(:pclass)
@combine(mean(:age))
end
```
> [@Yifan\_Liu](#):
>
> built-in functions in DataFrames confuse me a lot.
Espeically if you come from R. But Julia has a better chance of being logically and aesthetically consistent.
Actually, `dplyr` is really odd and `data.table` is also really odd. I remember being really confused when I first learned them. No doubt, it’s similiar to how u feel abt Julia now.
Do you use dplyr or data.table?
> [@Yifan\_Liu](#):
>
> but this extra verbosity
```julia
using DataConvenience
using DataFramesMacro
@> df begin
@where(:sex .== “male”)
@groupby(:pclass)
@combine(mean(:age))
end
```
this is not much more verbose once it’s done.
But if you use it alot then the extra verbosity is just two lines of using. So averaging over hundreds of data manipulation code. It’s no big deal.
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [September 27, 2020, 3:01am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/40 "2020-09-27T03:01:46Z")
I prefer data.table in my work due to its speed and memory-efficiency.
To my best knowledge, many R data science classes use dplyr because it is easier.
I have always used DataFramesMeta and I thought the plan is to test functions in it and then absorb the good ones into DataFrames. Now I realized that DataFrames started a lot of new things by itself…
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 27, 2020, 3:03am UTC](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022/41 "2020-09-27T03:03:11Z")
> [@Yifan\_Liu](#):
>
> DataFramesMeta and I thought the plan is to test functions in it and then absorb the good ones into DataFrames.
Maybe in the future. My reading is that DataFramesMeta.jl is a convenience layer on top of DataFrames.jl which will remain macro-less in the foreseeable future so the core devs can focus on core functionalities and leave convenience features to other packages, at least for now.
[Previous page](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022.md?page=1)
[Next page](https://discourse.julialang.org/t/dataframes-jl-development-survey/44022.md?page=3)