JDF - an experimental DataFrame serialization format is ready for beta testing

There’s a whole story of it which I have pieced together over the time I’ve been involved with it, which I’m not sure is worth getting into here. I think what they are getting at is that Feather was basically deprecated in favor of another arrow file format, which has no name and they never bothered to publicize (it’s basically just the arrow streaming format written into a file). Feather seems to have been a pretty early attempt at an arrow data format, and as such is only slightly more specialized to arrow than parquet is.

From what I can tell, they are unlikely to continue to evolve the Feather format itself, unless they decide to slap that label on the arrow file format (which I think would be a bad idea, because it would be very confusing). If this is true, it makes Feather pretty safe for long term storage.

Anyway, ok I’d have to admit I’d be a little nervous about deliberately writing out a feather file, not being allowed to touch it for 10 years, and then having to load it up without a problem 10 years later, but I wouldn’t exactly say it’s “unreliable”.

(By the way, I can already serialize and deserialize the arrow file format from my dev branch of Arrow.jl, but I haven’t finished getting it into a state that’s appropriate for a registered package and I don’t know when I might go back to it.)

3 Likes