JDF is the Julia DataFrames serialization format. It’s a specialised serialization format and hence doesn’t support arbitrary objects like JLD2 and JLSO. This loss of generality is more than made up for in gains in speed and reliability for saving and loading DataFrames.
JDF now supports DataFrames containing these types
where T can be String, Bool, and isbits types i.e. UInt*, Int*,
and Float*Date* types etc.
RLEVectors support will be considered in the future when missing support
arrives for RLEVectors.jl.
Also, there is now the ability to load only the columns you select. For example
At the moment though I don’t think we can move into JDF, because of the type limitations. I’ll keep watching this post to see if there are less restrictions on types as time progresses. ( @JonasIsensee I’m tagging you this may be interesting for you )
By the way, I think Symbol should also have a “fast” implementation. AT the moment I use Symbols as parameters to represent complicated functions that I don’t want to save in my DataFrame, and during my simulations I @eval those symbols.
I have one comment on the types:
At the moment we have columns with custom types (or collections of types - > Array of Any)
in our aggregated DataFrames but i suppose this is not really necessary if we find a different proper representation.
Strings are probably not effective as you can’t query into the values easily anymore.
Converting to namestuples could be an option or would that also not work fast?