Get on GroupedDataFrame

Marc.Cox · March 17, 2020, 6:03pm

Issue using the Base.get Get a group based on the values of the grouping columns example code
here >> Functions · DataFrames.jl

Base.get — Function.
get(gd::GroupedDataFrame, key, default)
Get a group based on the values of the grouping columns.

julia>

df = DataFrame(a = repeat([:foo, :bar, :baz], outer=[2]),
                             b = repeat([2, 1], outer=[3]),
                             c = 1:6);

julia> gd = groupby(df, :a)
GroupedDataFrame with 3 groups based on key: a

julia> get(gd, (a=:bar,), nothing)
ERROR: MethodError: no method matching get(::GroupedDataFrame{DataFrame}, ::NamedTuple{(:a,),Tuple{Symbol}}, ::Nothing)
Closest candidates are:
get(::REPL.Terminals.TTYTerminal, ::Any, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/REPL/src/Terminals.jl:176
get(::StatsModels.Schema, ::Any, ::Any) at /home/marc/.julia/packages/StatsModels/8N8nM/src/schema.jl:49
get(::StatsModels.FullRank, ::Any, ::Any) at /home/marc/.julia/packages/StatsModels/8N8nM/src/schema.jl:253

Julia Base v1.04
(v1.0-EXP) pkg> st
** [a93c6f00] DataFrames v0.20.2**

Any thoughts ,suggestions or work arounds appreciated.

bkamins · March 17, 2020, 6:32pm

It works on v0.20.2, so probably you have an old version of DataFrames.jl

nalimilan · March 17, 2020, 6:51pm

Yet his post shows that he’s using 0.20.2. Can you confirm all of these results are from the same Julia session? Can you post the full copy/paste, and add the output of methods(get)?

(BTW, please use triple backticks around code blocks.)

bkamins · March 17, 2020, 7:37pm

Ah - I have not noticed that (but I have checked on my machine that the code worked as expected, though on Julia 1.3.1)

Marc.Cox · March 17, 2020, 8:43pm

Thank You for your help.

Seems I “hit the seam” between Julia Base and Dataframes,
so is there a practical more general way that Dataframes development
can stay independent/orthogonal of the Julia Base version ?
Or at least automated testing / CBI / assertions that would catch bugs like this
in the use case example code in the documentation…

Ok so the presently suggested solution is upgrade to New Julia version v1.3.1 ?
Do we have more info handy about any other lower Julia versions that fixed the issue
such as v1.2 , v1.1 etc. ?

I just did an image backup so I’m not opposed to New Julia version v1.3.1
UNLESS v1.3.1 presents OTHER rude surprise “gotchas”
like the upgrade from v0.7 to v1.0 where the Atom-Uber-Juno IDE plugin
would NOT allow me to easily switch back to Julia v0.7
^^ and failures to “just rollback” are quite unpleasant surprises,
which I “forgave once” because by unspoken convention << 1.0
is considered BETA ergo subject to change.

Any pointers to known issues or advantages with Atom IDE upgrading from v1.04
to new Julia version v1.3.1 ?

Also @bkamins I’m interested in alternative Dataframe soln NOT using Base.get(gd, (a=:bar,)
for the following use case:

# df_Hinge_Segments:=6×3 DataFrame
# │ Row │ sym    │ left     │ right   │
# │     │ Symbol │ Float64  │ Float64 │
# ├─────┼────────┼──────────┼─────────┤
# │ 1   │ w      │ -3.34382 │ 2.68347 │
# │ 2   │ x      │ -3.12436 │ -1.7407 │
# │ 3   │ x      │ -1.7407  │ 1.71305 │
# │ 4   │ x      │ 1.71305  │ 3.54989 │
# │ 5   │ y      │ -11.3052 │ 13.0877 │
# │ 6   │ z      │ -3.12796 │ 3.66479 │

NOTE DROP sym=:y , so Essentially Need a total of Three(3) Dicts that
SPLIT integration_limits:=Dict(:w=>(-3.34382, 2.68347),:z=>(-3.12796, 3.66479),:x=(-3.12436, 3.54989))
because ( Count(:w)=1 times c(:x)=3 times c(:z)=1 ) = 3 combinations WITH 3 sym=:x LEFT,RIGHT ranges

btw I know I prefer understanding the “why” of things , so fyi this is part of an Integration by Parts plan using MultivariateFunctions Piecewise_Functions

nalimilan · March 17, 2020, 9:49pm

I had also tried Julia 1.0.1 and it works there, so AFAICT that’s not a problem with the Julia version.

bkamins · March 17, 2020, 10:54pm

Yes - we test everything against Julia 1.0 in CI and the tests of get pass.

bkamins · March 17, 2020, 10:55pm

Regarding your question get is actually provided only for completeness when you expect that you might not find the key you are looking for. In your case writing:

gd[(:bar,)]

or

gd[(a=:bar,)]

should just work

Marc.Cox · March 17, 2020, 11:23pm

No luck so far with >>
julia> gd[(a=:bar,)]
ERROR: MethodError: no method matching getindex(::GroupedDataFrame{DataFrame}, ::NamedTuple{(:a,),Tuple{Symbol}})
Closest candidates are:
getindex(::GroupedDataFrame, ::Integer) at /home/marc/.julia/packages/DataFrames/yH0f6/src/groupeddataframe/grouping.jl:164
getindex(::GroupedDataFrame, ::AbstractArray) at /home/marc/.julia/packages/DataFrames/yH0f6/src/groupeddataframe/grouping.jl:168
getindex(::GroupedDataFrame, ::Colon) at /home/marc/.julia/packages/DataFrames/yH0f6/src/groupeddataframe/grouping.jl:182

bkamins · March 18, 2020, 7:01am

Can you please double check that you have DataFrames 0.20.2 installed. The " /home/marc/.julia/packages/DataFrames/yH0f6" directory name seems not to indicate 0.20.2 version of the DataFrames.jl (but some other version) as the last part of the path should be “S3ZFo”.

Marc.Cox · March 18, 2020, 8:51pm

Thanks for your help.

OK so presently my plan is the following steps:

rm --project DataFrames

rm --manifest DataFrames

add DataFrames

ad let you know the results of julia>gd[(a=:bar,)]

Marc.Cox · March 18, 2020, 11:43pm

Thanks Bogumił @bkamins !

Good catch on the version mismatch with installation folder !!

Thanks for you help. Now it Works Hurray !

For future reference I took the following steps:

rm --project DataFrames

rm --manifest DataFrames

add DataFrames

## fyi locked up and needed a reboot at this point, which is odd for Linux …

## After reboot , when restarted Atom received Info : about recompile ; and also received some strange Warnings and Errors Like so >>

Press Enter to start Julia.

Starting Julia…

[ Info : Recompiling stale cache file /home/marc/.julia/compiled/v1.0/Atom/w9XOh.ji for Atom [c52e3926-4ff0-5f6e-af25-54175e0327b1]

.

[ Info: Recompiling stale cache file /home/marc/.julia/compiled/v1.0/MultivariateFunctions/ig9dm.ji for MultivariateFunctions [54e48022-12d0-513b-b20d-e273c072c6db]

┌ Warning: Error requiring Juno from PlotThemes:

│ LoadError: MethodError: no method matching hex(::UInt32, ::Int64, ::Bool)

│ You may have intended to import Base.hex

│ Stacktrace:

## But Tests are working AOK, thanks again

julia> df = DataFrame(a = repeat([:foo, :bar, :baz], outer=[2]),

b = repeat([2, 1], outer=[3]),

c = 1:6);

julia> gd = groupby(df, :a)

GroupedDataFrame with 3 groups based on key: a

First Group (2 rows): a = :foo

│ Row │ a │ b │ c │

│ │ Symbol │ Int64 │ Int64 │

├─────┼────────┼───────┼───────┤

│ 1 │ foo │ 2 │ 1 │

│ 2 │ foo │ 1 │ 4 │

⋮

Last Group (2 rows): a = :baz

│ Row │ a │ b │ c │

│ │ Symbol │ Int64 │ Int64 │

├─────┼────────┼───────┼───────┤

│ 1 │ baz │ 2 │ 3 │

│ 2 │ baz │ 1 │ 6 │

julia> get(gd, (a=:bar,), nothing)

2×3 SubDataFrame

│ Row │ a │ b │ c │

│ │ Symbol │ Int64 │ Int64 │

├─────┼────────┼───────┼───────┤

│ 1 │ bar │ 1 │ 2 │

│ 2 │ bar │ 2 │ 5 │

julia> gd[(a=:bar,)]

2×3 SubDataFrame

│ Row │ a │ b │ c │

│ │ Symbol │ Int64 │ Int64 │

├─────┼────────┼───────┼───────┤

│ 1 │ bar │ 1 │ 2 │

│ 2 │ bar │ 2 │ 5 │

Good catch on the version mismatch with installation folder !!

Marking above as solution per Tests are now working AOK,

Thanks again !!