# \[ANN\] LightBSON.jl

**URL:** <https://discourse.julialang.org/t/ann-lightbson-jl/67437>\
**Category:** Package Announcements\
**Tags:** announcement, bson, serialization\
**Created:** [August 31, 2021, 4:35pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437 "2021-08-31T16:35:41Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Christian\_Rorvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_rorvik/32/965_2.png) [@Christian\_Rorvik](https://discourse.julialang.org/u/Christian_Rorvik)\
**Post date:** [August 31, 2021, 4:35pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/1 "2021-08-31T16:35:41Z")

</div>

Borne out of a need for high performance and allocation free BSON processing, announcing [LightBSON.jl](https://github.com/ancapdev/LightBSON.jl).

## What It Is

- Allocation free API for reading and writing BSON data.
- Natural mapping of Julia types to corresponding BSON types.
- Convenience API to read and write `Dict{String, Any}` or `OrderedDict{String, Any}` (default, for roundtrip consistency) as BSON.
- Struct API tunable for tradeoffs between flexibility, performance, and evolution.
- Configurable validation levels.
- Light weight indexing for larger documents.
- [Transducers.jl](https://github.com/JuliaFolds/Transducers.jl) compatible.
- Tested for conformance against the [BSON corpus](https://github.com/mongodb/specifications/blob/master/source/bson-corpus/bson-corpus.rst).

## What It Is Not

- Generic serialization of all Julia types to BSON. See [BSON.jl](https://github.com/JuliaIO/BSON.jl) for that functionality. [LightBSON.jl](https://github.com/ancapdev/LightBSON.jl/blob/master/README.md) aims for natural representations, suitable for interop with other languages and long term persistence.
- Integrated with [FileIO.jl](https://github.com/JuliaIO/FileIO.jl). [BSON.jl](https://github.com/JuliaIO/BSON.jl) already is, and adding another with different semantics would be confusing.
- A BSON mutation API. Reading and writing are entirely separate and only complete documents can be written.
- Conversion to and from [Extended JSON](https://docs.mongodb.com/manual/reference/mongodb-extended-json/). This may be added later.

See [README](https://github.com/ancapdev/LightBSON.jl/blob/master/README.md) for more examples and further information.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [August 31, 2021, 5:03pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/2 "2021-08-31T17:03:18Z")

</div>

Not being integrated with FileIO is a hit for interplay with DrWatson’s `tagsave, produce_or_load` and co. ☹ Do you think it is possible to actually implement this? Had many problems with the existing BSON.jl, unlikely to use it in the future, so your package seems like a nice alternative.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [August 31, 2021, 9:44pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/3 "2021-08-31T21:44:19Z")

</div>

Yay.  
I am so glad to have an option for plain BSON that doesn’t try and serialize arbitrary Julia objects.  
This maximizes compatibility,  
and also means it can be a secure format.  
(In contrast is is seemingly impossible to make anything like pickle, or the Julia serializer not able to run arbitrary code on loading. This includes BSON.jl, JLD2, and JLSO)

---

<div class="post-metadata">

**Author:** ![Christian\_Rorvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_rorvik/32/965_2.png) [@Christian\_Rorvik](https://discourse.julialang.org/u/Christian_Rorvik)\
**Post date:** [September 1, 2021, 9:18am UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/4 "2021-09-01T09:18:36Z")

</div>

> Not being integrated with FileIO is a hit for interplay with DrWatson’s `tagsave, produce_or_load` and co. ☹ Do you think it is possible to actually implement this?

It doesn’t look difficult to implement, but I’m concerned if it would cause confusion with users. From how I understand FileIO to work, it will try each package for a format in the order they were registered, fail if the package can’t be loaded (e.g., not in your project), and try the next one. If I add LightBSON to FileIO it becomes difficult to determine which package will be used to load `.bson` files outside of the top level project where you have control of your environment. Given the semantic difference in how BSON.jl and LightBSON.jl treat the data, I’m not sure how users would actually handle that.

If there are good solutions to this I’d be happy to add the functionality. You probably have more experience with the ecosystem, so how do you see it best working?

---

<div class="post-metadata">

**Author:** ![jackyjoy123](https://avatars.discourse-cdn.com/v4/letter/j/d6d6ee/32.png) [@jackyjoy123](https://discourse.julialang.org/u/jackyjoy123)\
**Post date:** [September 5, 2021, 11:44am UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/5 "2021-09-05T11:44:50Z")

</div>

> [@Christian\_Rorvik](#):
>
> It doesn’t look difficult to implement, but I’m concerned if it would cause confusion with users. From how I understand FileIO to work, it will try each package for a format in the order they were registered, fail if the package can’t be loaded (e.g., not in your project), and try the next one. If I add LightBSON to FileIO it becomes difficult to determine which package will be used to load `.bson` files outside of the top level project where you have control of your environment. Given the semantic difference in how BSON.jl and LightBSON.jl treat the data, I’m not sure how users would actually handle that.
> 
> If there are good solutions to this I’d be happy to add the functionality. You probably have more experience with the ecosystem, so how do you see it best working?

thanks for the awesome information.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [September 9, 2021, 2:09pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/6 "2021-09-09T14:09:31Z")

</div>

Hm, I guess having the ending `.lbson` for light-bson is not good? because it means the files wouldn’t be able to load with other BSON formats in other programming environments…?

---

<div class="post-metadata">

**Author:** ![Christian\_Rorvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_rorvik/32/965_2.png) [@Christian\_Rorvik](https://discourse.julialang.org/u/Christian_Rorvik)\
**Post date:** [September 9, 2021, 6:08pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/7 "2021-09-09T18:08:23Z")

</div>

It would work for the FileIO dispatch, but I think it would be a bit strange to change extension only for that when it’s the same format in the file

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [September 9, 2021, 6:27pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/8 "2021-09-09T18:27:19Z")

</div>

I suspect can hand-code a case for `Format{:BSON}` into FileIO which does a bit of fiddling to guess if it is BSON.jl using extended features (I think there will be a types top-level key or something?)  
and if so using BSON.jl, or if not uses LightBSON.jl

---

<div class="post-metadata">

**Author:** ![Christian\_Rorvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_rorvik/32/965_2.png) [@Christian\_Rorvik](https://discourse.julialang.org/u/Christian_Rorvik)\
**Post date:** [September 9, 2021, 6:33pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/9 "2021-09-09T18:33:57Z")

</div>

That’s a good suggestion, I can see that working for load. What would you do on the save path?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [September 9, 2021, 6:47pm UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/10 "2021-09-09T18:47:19Z")

</div>

I have no super bright ideas, maybe look at the data and see if it only contains things that follow the BSON spec?  
It is a recursive search; might want a max limit before guessing it doesn’t.

---

<div class="post-metadata">

**Author:** ![Christian\_Rorvik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christian_rorvik/32/965_2.png) [@Christian\_Rorvik](https://discourse.julialang.org/u/Christian_Rorvik)\
**Post date:** [September 10, 2021, 6:28am UTC](https://discourse.julialang.org/t/ann-lightbson-jl/67437/11 "2021-09-10T06:28:59Z")

</div>

I’m hesitant to implement heuristics that aren’t robust and clear, and LightBSON.jl also lets you write custom types, so instead I’ve added a keyword argument instead to select the saver.

With [these additions](https://github.com/ancapdev/LightBSON.jl/commit/1479c86834ae559e950d34db0f4920751264231f) in LightBSON.jl:

```nohighlight
function fileio_save(f, x; kwargs...)
    get(kwargs, :plain, false) || error("Pass plain = true to select LigthBSON.jl over BSON.jl")
    bson_write(f.filename, x)
    nothing
end

function fileio_load(f)
    buf = read(f.filename)
    reader = BSONReader(buf)
    hastag = false
    hastype = false
    foreach(reader) do field
        hastag |= field.first == "tag"
        hastype |= field.first == "type"
        nothing
    end
    hastag && hastype && error("BSON.jl document detected, aborting LightBSON.jl load")
    reader[Any]
end 

```

and registering LightBSON.jl at higher priority than BSON.jl in FileIO, you can pass `plain = true` to save in plain BSON format, whilst the load path detects automatically as you suggested. Would that work or be made to work with DrWatson?