I have been using DrWatson for a project at work for months now and have found it mostly helpful.
However, there is one thing I do not understand:
As far as I understand, it is more efficient in many cases to use NamedTuples for handling structured, dict-like data like simulation results. However, DrWatson seems not to be able to store and load NamedTuples directly, and you need a conversion step to a Dictionary.
Main question is: Why - and is there a way to handle NamedTuples more directly.
My second question: Right now, DrWatson is complaining about me using “key instead of symbol instead of a string”. Apart from the log-pollution caused by this warning there is no problem at all though. Can this be fixed/silenced (without just using Suppressor around any Code that calls produce_or_load)?
It isn’t DrWatson that is complaining. It is JLD2. Or I assume at least, since you haven’t given any exact copy of the messages that clog your logs.
To use string-based dictionaries use @strdict instead of @dict. Or just initialize/create your dictionary with string keys instead of symbols.
To silence the warning you need to open an issue at JLD2.jl requesting a keyword to silence this warning. YOu can propagate this keyword then via the keyword arguments that DrWatson propagates like save_kwargs.
Can you provide some context or more information? DrWatson doesn’t store anything, it is not a IO package. It provides functions that themselves wrap other IO packages. When you call e.g., tagsave by default the JLD2.jl package will be used. So I can’t really say to exactly what thing you are referring when you are saying “it doesn’t store named tuples”. I can easily do tagsave(filename, Dict("data" => NamedTuple())) so the named tuple will be stored correctly.
Do you want to do instead tagsave(filename, (data = "example",))? If so, I unfortunately do not know of any IO backend that does this. You can open an feature request to an existing package, although I doubt it would be implemented. Dictionaries are better suited for this.
What do you mean by “simulation results”?
Generally speaking, I think it is the opposite from what you said. A dictionary is better suited data structure for a named list of many elements. Tuples do not scale well with performance for many elements and to my experience simulation results would have multiple outputs.
Thanks for your answer - I think there are multiple valid suggestions for solutions in your post, the easiest being simply to wrap a tuple inside a dictionary, the most complicated to find an output package supporting named tuples.
I think you are also right about JLD2, not DrWatson being the origin of the error messages.
The next question you raised is whether NamedTuples are suitable at all. I do not have good intuition about what “many elements” means in this context… My result structures should usually have not more than 20 entries on the top level. Those might in turn contain larger structures like 2D arrays, but I would naively assume that those are stored as pointers while the results are still in memory? Off course when writing to a file, pointers would have to be resolved, but this should be the case for dictionaries and tuples alike.