I’ve written my own MCMC code. MCMCChains provides some helper functions wrt working with already completed chains.
As I iterate through the chain, what is the best way to store the each pass? E.g., if I complete non-burn pass 1757/10,000, and state contains vectors and scalars, say state is (;a=3.0, b=[4.0,6.0,7.0]), should I just push the draws to a DataFrame, or is there a better purpose-built solution?
Without code it is hard to answer. But my advice is to allocate the whole data frame before running anything and then filling it instead of pushing.
Code @lrnv :
#put state into records
records = #some data type
recordstate(;i=1757, state=(;a=3.0, b=[4.0,6.0,7.0]),records)
Yes, that’ll work.
What I would do is have record preallocated (maybe with a bunch of zeros) before launching the chain.
Isn’t there some actual API for pushing new values into the chain itself rather than producing some intermediate data structure
@dlakelan That would be great, but I can’t seem to find it in the docs- am I missing something obvious?
It looks like even the AdvancedMH creates an intermediate structure that subtypes AbstractTransition, then all at once bundles the samples afterwards AdvancedMH.jl/mcmcchains-connect.jl at master · TuringLang/AdvancedMH.jl · GitHub
Might just be a case where a standardized solution is more trouble than it is worth. To be fair, it is not like an intermediate structure is a huge burden to code up.
After some additional experimentation, it does seem possible to keep a handle to the underlying arrays in the chain e.g.
a=rand(10); c=Chains(a,:a); creates the chain without copying. Then indexing directly into the three dimensional arrays seems to work (e.g.
A bit cumbersome but serviceable if needed. However, seems like the interface intends the user to convert at the end so I might just do that, especially since I’m not sure how the copying/referencing will work in future releases.
In fact it might well be less code burden than trying to fill in an array… But it is a lot of memory, particularly if you’re doing something like sampling 10k samples in 4 chains of a model with 10k parameters.
Yeah- I ended up creating a wrapper object around the chain instead of an intermediate object. The wrapper contains views for accessing the chains for each of my parameters. Then I overloaded
getproperty so I can use the “.” syntax for further ease of access- best of all I can still use the MCMCChain analytics without needing to create a new object.
Thanks for helping me think this through.