# Serialization format allow incremental write to file
**URL:** https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464
**Category:** General Usage
**Tags:** serialization
**Created:** [March 2, 2023, 7:07pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464 "2023-03-02T19:07:05Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [March 2, 2023, 7:07pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/1 "2023-03-02T19:07:05Z")
Is there any data format that supports incremental writes and allows flush to the file after the write? I find julia-arrow support incremental writes here: [refactor Arrow.write to support incremental writes by baumgold · Pull Request #277 · apache/arrow-julia · GitHub](https://github.com/apache/arrow-julia/pull/277/files)
but it does not allow me to flush io to the file after the write without closing the file io. Any idea if this is supported? Or is there a different format allows me to do this?
---
### Author:  [@bilderbuchi](https://discourse.julialang.org/u/bilderbuchi)
#### Post date: [March 2, 2023, 7:37pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/2 "2023-03-02T19:37:21Z")
ASDF - the Advanced Scientific Data Format AFAIK has streaming writes, at least according to [Low-level file layout — ASDF Standard 1.6.0 documentation](https://asdf-standard.readthedocs.io/en/1.6.0/file_layout.html?highlight=stream#exploded-form) / [Introduction — ASDF Standard 1.6.0 documentation](https://asdf-standard.readthedocs.io/en/1.6.0/intro.html?highlight=stream#introduction)
Don’t know about the flushing, though.
There’s a Julia package by @schnetter at [GitHub - eschnett/ASDF.jl: A Julia implementation of the Advanced Scientific Data Format (ASDF)](https://github.com/eschnett/ASDF.jl), but I don’t know its status.
---
### Author:  [@quinnj](https://discourse.julialang.org/u/quinnj)
#### Post date: [March 2, 2023, 8:09pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/3 "2023-03-02T20:09:22Z")
Why can’t you `flush` after an incremental write w/ Arrow.jl? You can pass your own `IO` to `Arrow.append` and then call `flush(io)` yourself?
---
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [March 2, 2023, 8:35pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/4 "2023-03-02T20:35:28Z")
I tried that but somehow didn’t work? e.g
```julia
using Arrow
using Tables
row_A = (field=[1.0, 2.0], temp=[1.0, 1.0], energy=[-0.0, -0.0])
row_B = (field=[3.0, 2.0], temp=[1.0, 1.0], energy=[-0.0, -0.0])
io = open("test.arrow", "w")
Arrow.append(io, row_A)
flush(io)
tbl = Arrow.Table("test.arrow") # this has two rows
Arrow.append(io, row_B)
flush(io)
tbl = Arrow.Table("test.arrow") # this still have two rows does not have B
close(io)
```
---
### Author:  [@schnetter](https://discourse.julialang.org/u/schnetter)
#### Post date: [March 3, 2023, 1:16pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/5 "2023-03-03T13:16:24Z")
`ASDF.jl` should be working, but I am not using it any more. I switched to using [`ADIOS2.jl`](https://github.com/eschnett/ADIOS2.jl) as file format, which has many more features.
I am using ADIOS2 when running simulations of PDEs. Every few iterations one writes some variables to the file and flushes them. This use case is very efficient with ADIOS2. In other respects, ADIOS2 is similar to HDF5, in that it is designed to hold multi-dimensional arrays with attributes.
-erik
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [March 3, 2023, 1:19pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/6 "2023-03-03T13:19:31Z")
I suspect what’s happening there is file is flushed (check file size?), but the metadata isn’t updated until file is closed.
this is very common, we don’t want to re-locate / re-write metadata chunk every time we flush I think?
---
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [March 3, 2023, 3:33pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/7 "2023-03-03T15:33:05Z")
I mean if that’s the case how do I read my data back if my program crash without metadata?
---
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [March 3, 2023, 3:33pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/8 "2023-03-03T15:33:57Z")
> [@schnetter](#):
>
> I am using ADIOS2 when running simulations of PDEs. Every few iterations one writes some variables to the file and flushes them. This use case is very efficient with ADIOS2. In other respects, ADIOS2 is similar to HDF5, in that it is designed to hold multi-dimensional arrays with attributes.
Thanks! This seems to be what I want
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [March 3, 2023, 4:19pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/9 "2023-03-03T16:19:33Z")
I don’t know if Arrow.jl is at fault here (our implementation is bad) or it’s a general Arrow design issue – they may not have crash recovery as a design goal.
For the closely related Parquet format, it seems to be a thing: [Error Recovery | Apache Parquet](https://parquet.apache.org/docs/file-format/data-pages/errorrecovery/)
---
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [March 12, 2023, 7:00pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/10 "2023-03-12T19:00:35Z")
OK, I think I just did this on my own - a custom data format that given the data I’d like to flush to disk is quite simple. I don’t believe Arrow works out for me in the end. But still thanks to everyone’s replies here.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [March 12, 2023, 7:11pm UTC](https://discourse.julialang.org/t/serialization-format-allow-incremental-write-to-file/95464/11 "2023-03-12T19:11:50Z")
@quinnj I think it’s pretty important to support incremental write?