I am using DrWatson, and many times I run Monte Carlo simulation which result in large data. As an example you can think of a time sequence, where each element in the sequence is a matrix.
Sometimes, the simulation prepares many such time sequences, which results in large data.
When I load the results using collect_results, I get that each row hold a lot of information.
I can then handle the proper separation in code.
My main issue is that if I forget to add ; at the end of a line which returns the DataFrame, the jupyter notebook get stuck as it need a lot of memory to present the information.
This causes me to think that I am implementing my Monte Carlo data saving wrong. If anyone has good suggestions on how to save or load things properly it would be nice to hear!
(BTW, I am aware of this question, however I do not really want to use a database. I don’t think it necessary for my case).