Bayesian model comparison / selection

Specific Domains Statistics
, ,
3

Ah, thanks, this is a bug. I have opened an issue to track this. ArviZ converts the arviz.ELPDData returned by arviz.loo to a DataFrame on the Julia side, but the Python compare function expects an ELPDData input, and currently ArviZ doesn’t convert it back.

There are 2 quick workarounds.

First, we’ll load an example InferenceData:

julia> using ArviZ

julia> idata = load_arviz_data("radon");

Option 1: pass a Dict of InferenceData to compare:

julia> compare_dict1 = Dict(
           "model 1" => idata,
           "model 2" => idata,
           "model 3" => idata,
       );

julia> compare(compare_dict1; ic="loo")
3×10 DataFrame
 Row │ name     rank   loo       p_loo    d_loo    weight    se       dse      warning  loo_scale 
     │ String   Int64  Float64   Float64  Float64  Float64   Float64  Float64  Bool     String    
─────┼────────────────────────────────────────────────────────────────────────────────────────────
   1 │ model 1      0  -1027.12  26.7643      0.0  0.333333  28.8476      0.0    false  log
   2 │ model 3      1  -1027.12  26.7643      0.0  0.333333  28.8476      0.0    false  log
   3 │ model 2      2  -1027.12  26.7643      0.0  0.333333  28.8476      0.0    false  log

Option 2: bypass ArviZ.jl’s conversion to DataFrame by calling the Python function directly:

julia> loo_result = ArviZ.arviz.loo(idata)
PyObject Computed from 2000 by 919 log-likelihood matrix

         Estimate       SE
elpd_loo -1027.12    28.85
p_loo       26.76        -
------

Pareto k diagnostic values:
                         Count   Pct.
(-Inf, 0.5]   (good)      919  100.0%
 (0.5, 0.7]   (ok)          0    0.0%
   (0.7, 1]   (bad)         0    0.0%
   (1, Inf)   (very bad)    0    0.0%


julia> compare_dict2 = Dict(
           "model 1" => loo_result,
           "model 2" => loo_result,
           "model 3" => loo_result,
       );

julia> compare(compare_dict2; ic="loo")
3×10 DataFrame
 Row │ name     rank   loo       p_loo    d_loo    weight    se       dse      warning  loo_scale 
     │ String   Int64  Float64   Float64  Float64  Float64   Float64  Float64  Bool     String    
─────┼────────────────────────────────────────────────────────────────────────────────────────────
   1 │ model 1      0  -1027.12  26.7643      0.0  0.333333  28.8476      0.0    false  log
   2 │ model 3      1  -1027.12  26.7643      0.0  0.333333  28.8476      0.0    false  log
   3 │ model 2      2  -1027.12  26.7643      0.0  0.333333  28.8476      0.0    false  log

Whenever ArviZ.jl wraps a Python function with no modification, rather than manually convert all docs to Julia and try to keep them up-to-date, we refer the users to the Python docs. In the future these will all be pure Julia functions with their own documentation.

2 Likes