Stack overflow in DataFrames group by

[nalimilan]

@ExpandingMan

This may go somewhat against the intended design philosophy but I think that we need a few different functions available depending on the specific context. The issue of allocating appropriate types is difficult, but in a huge number of cases the function passed to by will return dataframes all of which have the same types. Then, when the user calls this function, we can use the first output to determine the types of the rest. In that case it would be up to the user to ensure the function always outputs the proper type. We would of course still have by for inferring types. Ideally we shouldn’t sacrifice performance for edge cases, though perhaps I’m wrong about what the most common use cases are.

Yes, maybe, but there are several levels of complexity regarding what the function returns for each group:

1. a single scalar
2. several scalars (e.g. as a tuple, NamedTuple or a single-row DataFrame)
3. a multiple-row DataFrame (possibly with a varying number of rows depending on the group)

Each of these cases can be either type-stable or not, which makes things even more tricky.

FWIW, Pandas provides three different functions:

• aggregate to return a single value for each column and each group (similar to our aggregate, which is a bit more general)
• transform to return a DataFrame with the same shape as the original for each group
• apply for general transformations (similar to our by)

The advantage of transform over apply is that you know in advance the size of the result, so you can avoid allocating a temporary copy if you can predict the output type. Same for aggregate, which can be even more efficient since it can operate by columns (making inference and specialization easier). The fact that we allow aggregate to return either a scalar or a vector makes it harder to optimize, cf. this PR.

@ExpandingMan

I also like @nalimilan’s ideas. It also seems to me that Base really ought to have a version of vcat which takes an iterable as an argument.

See `stack(vec_of_vecs)` for `vcat(vec_of_vecs...)` · Issue #21672 · JuliaLang/julia · GitHub.