How is the data ecosystem right now for large datasets?

[nalimilan]

@aaowens

Regarding joins, for DataTables:

dt2 = by(dt, :B, d -> mean(d[:A]))
join(dt, dt2, on = :B)

For pandas:

df2 = mean(groupby(df, “B”))
df3 = merge(df, df2, left_on = “B”, right_index = true)

I suspect there is some way to do this all in one go, like with broadcasting and the dot notation in Julia.

DataTables: 459s
Pandas: 14s

Just an update to note that with another pull request, I was able to reduce the time needed by DataTables for this operation to 20s, so reasonably close to Pandas, and about as fast as DataFrames (my machine appears to be slighly slower than yours).