ParetoSmooth.jl psis_loo crashes with ERROR: The number of lines in `row_names` must match the number of lines in the matrix

Specific Domains Probabilistic programming
1

Hi!

I found an example use of ParetoSmooth.jl psis_loo here, but when I try to run the example it crashes with

[ Info: No source provided for samples; variables are assumed to be from a Markov Chain. If the samples are independent, specify this with keyword argument `source=:other`.
┌ Warning: Some Pareto k values are extremely high (>1). PSIS will not produce consistent estimates.
└ @ ParetoSmooth ~/.julia/packages/ParetoSmooth/A6x5U/src/InternalHelpers.jl:43
Results of PSIS-LOO-CV with 1000 Monte Carlo samples and 1 data points. Total Monte Carlo SE of 0.29.
Error showing value of type PsisLoo{Float64, Array{Float64, 3}, Vector{Float64}}:
ERROR: The number of lines in `row_names` must match the number of lines in the matrix.

Has something changed? I am running ParetoSmooth v0.7.1 and Turing 0.22.0.
I also tried the new/other(?) Turing syntax where I wrote to model to condition on the observations psis_loo(test(Xₜ, 1) | (;Yₜ), ch) but it gave the same error. @ParadaCarleton

full stack trace

ERROR: The number of lines in row_names must match the number of lines in the matrix.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] _print_info(data::PrettyTables.ColumnTable; alignment::Symbol, cell_alignment::Nothing, cell_first_line_only::Bool, compact_printing::Bool, filters_row::Nothing, filters_col::Nothing, formatters::PrettyTables.var"#52#54"{Vector{String}}, header::Tuple{Vector{Symbol}}, header_alignment::Symbol, header_cell_alignment::Nothing, limit_printing::Bool, renderer::Symbol, row_names::Vector{Symbol}, row_name_alignment::Symbol, row_name_column_title::String, row_number_column_title::String, show_row_number::Bool, title::String, title_alignment::Symbol)
@ PrettyTables ~/.julia/packages/PrettyTables/nQZHQ/src/private.jl:220
[3] #_pt#81
@ ~/.julia/packages/PrettyTables/nQZHQ/src/private.jl:458 [inlined]
[4] #_pretty_table#80
@ ~/.julia/packages/PrettyTables/nQZHQ/src/private.jl:378 [inlined]
[5] #pretty_table#68
@ ~/.julia/packages/PrettyTables/nQZHQ/src/print.jl:706 [inlined]
[6] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol(“text/plain”)}, loo_object::PsisLoo{Float64, Array{Float64, 3}, Vector{Float64}})
@ ParetoSmooth ~/.julia/packages/ParetoSmooth/A6x5U/src/LeaveOneOut.jl:57
[7] (::REPL.var"#43#44"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol(“text/plain”)}, Base.RefValue{Any}})(io::Any)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:267
[8] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:521
[9] display(d::REPL.REPLDisplay, mime::MIME{Symbol(“text/plain”)}, x::Any)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:260
[10] display(d::REPL.REPLDisplay, x::Any)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:272
[11] display(x::Any)
@ Base.Multimedia ./multimedia.jl:328
[12] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[13] invokelatest
@ ./essentials.jl:726 [inlined]
[14] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:296
[15] (::REPL.var"#45#46"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:278
[16] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:521
[17] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:276
[18] (::REPL.var"#do_respond#66"{Bool, Bool, REPL.var"#77#87"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:857
[19] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[20] invokelatest
@ ./essentials.jl:726 [inlined]
[21] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ REPL.LineEdit ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/LineEdit.jl:2510
[22] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
@ REPL ~/software/julia/1.8.2/share/julia/stdlib/v1.8/REPL/src/REPL.jl:1248
[23] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
@ REPL ./task.jl:484

2

This error usually crops up when you describe your entire dataset as being drawn from a single multivariate normal. However, the multivariate normal is supposed to represent a single observation of multiple variables (e.g. you might model height and weight as jointly multivariate normal), rather than multiple observations of a single variable (e.g. modeling the heights of many participants as independent draws from a Normal distribution).

3

Thank you for explaining! That would not have been easy for me to debug.

I absolutely understand the distinction between an N dim MvNormal and N Normal distributions, the reason I use it in Turing for the observations is that the alternative, loops, are slow. Are you saying that ParetoSmooth.jl will only work if one loops over the observed variable?

4

You can either loop or broadcast, but basically yes. Unfortunately, ParetoSmooth.jl uses Turing itself to figure out what counts as an observation. Future versions should fix this, though! This was quite difficult with the older interface, but I think now that Turing’s switching over to using SimpleVarInfo for everything it should be a lot easier.

Thanks for letting me know the error message is unclear! I’ll edit it when I have time.

5

That makes sense. Thank you for your work on Bayesian inference tools in Julia :slight_smile:
I know this is off topic but will the SimpleVarInfo refactoring have any implications for end users of Turing? I see it mentioned here and there in discussions and issues and it sounds like a big deal.

6

It’s a very big deal internally, but not so much for users, except insofar as it means “Turing will be faster and development for it will be easier.”

1 Like