BSONqs.jl v0.5.0 - high speed fork of BSON.jl

Package Announcements
1

In short this library is intended to make loading Julia structs/types from their BSON representation[1] very efficient. I already wrote a post about this package, detailing some of my reasoning and optimisations.

I have now also added a new interface that allows for partial lazy loading and should be completely type stable if you know the concrete type you want to deserialise in advance. Please see the links above for an explanation.

[1] As defined by the original BSON.jl which can serialise almost any Julia type

4 Likes
2

Thanks for posting this. I would like to ask these questions:

  1. Will you support this package / fork actively, or your plan is to leave it in the state it is, and maybe once per semester merge a PR?
  2. What about bugs of the original BSON, e.g. Array{Array{T, N}, 1} saved as Array{Any, 1}? · Issue #35 · JuliaIO/BSON.jl · GitHub ? What if they get solved at the original repo, are you willing to also update your own fork with the fixes?
2 Likes
3

I am currently using it for the foreseeable future and will definitely fix problems which I have. If someone posts an issue I don’t have, I might fix it depending on the nature of the problem. PRs are welcome of course. If it is a case of just cherry picking from the original, then that is easy enough. I haven’t really touched the write side, so in most cases I can just merge those fixes. On the read side, the two libraries probably don’t share that many problems. I can’t guarantee that I will see PRs on the original though, so if no one complains I might miss it.

3 Likes
4

I guess the bug that you linked to is caused by the write side because it appears to be writing a plain BSON array not tagged with any julia type. I might rewrite that part at some point, but I have no idea when.

5

@Datseris I came up with a partial solution to the linked problem. I suppose that there is a similar problem with hash maps which this would partially fix as well.

See https://github.com/richiejp/BSONqs.jl/commit/8315b11408bb146a5f203bc4303b8965224c5f67
for details.

6

Hi! I’m planning to register https://github.com/felipenoris/BSONSerializer.jl soon. I would love to see benchmarks comparing them.

7

Hello. Sure, I created some benchmarks (biased towards nested structs) https://github.com/richiejp/serbench. It probably needs some modification to remove bits (and maybe other) types though?

8

Apparently not, or I already fixed it by accident.

9

I found a bug while experimenting with Julia 1.3/4, seems like Julia now sometimes uses the none generated versions of the generated methods and one of these was buggy. So version will be bumped to v0.6.1.

I’m pondering whether to add multithreading support for really big arrays/dicts… I already split my data into multiple BSON documents though, so don’t really need it.