Query.jl fails handle CSV.Rows

General Usage
,
1

I got an dataset too large for memory, I want to combine CSV.Rows and Query.jl to handle data, but after @groupby, I can’t use @map to summarize data.

using DataFrames 
using CSV 
using Query 
data_path = "D:\\chen\\detail.csv" 
data_itr = CSV.Rows(data_path, reusebuffer=true) 
data_itr |> @groupby(_.hospital) |> @map({Key=key(_), Total=sum(_.amount)}) |> DataFrame

ERROR: type Row2 has no field amount
Stacktrace:
[1] getproperty(g::Grouping{Any, CSV.Row2}, name::Symbol)
@ QueryOperators C:\Users\gxjk009.julia\packages\QueryOperators\dF1vq\src\enumerable\enumerable_groupby.jl:29
[2] (::var"#94#99")(333::Grouping{Any, CSV.Row2})
@ Main C:\Users\gxjk009.julia\packages\Query\85Sw7\src\query_translation.jl:58
[3] iterate
@ C:\Users\gxjk009.julia\packages\QueryOperators\dF1vq\src\enumerable\enumerable_map.jl:25 [inlined]
[4] iterate
@ C:\Users\gxjk009.julia\packages\Tables\T7rHm\src\tofromdatavalues.jl:45 [inlined]
[5] buildcolumns
@ C:\Users\gxjk009.julia\packages\Tables\T7rHm\src\fallbacks.jl:202 [inlined]
[6] columns
@ C:\Users\gxjk009.julia\packages\Tables\T7rHm\src\fallbacks.jl:265 [inlined]
[7] DataFrame(x::QueryOperators.EnumerableMap{Union{}, QueryOperators.EnumerableIterable{Grouping{Any, CSV.Row2}, QueryOperators.EnumerableGroupBy{Grouping{Any, CSV.Row2}, Any, CSV.Row2, QueryOperators.EnumerableIterable{CSV.Row2, CSV.Rows{Vector{UInt8}, Tuple{}, PosLen, PosLenString}}, var"#91#96", var"#92#97"}}, var"#94#99"}; copycols::Nothing)
…

2

When the dataset is too large for memory, you may want to avoid materialization and use single-pass algorithms. Try OnlineStats.jl:

fit!(GroupBy(String, Sum()), data_itr)  # assuming there are two columns: hospital is a string, and amount is a number