# DataFrames: Which formats (if any) support saving of metadata

**URL:** <https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [November 27, 2024, 12:24am UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134 "2024-11-27T00:24:44Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [November 27, 2024, 12:24am UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/1 "2024-11-27T00:24:44Z")

</div>

Are there file formats that support saving of metadata (global and columnar)?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [November 27, 2024, 1:33am UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/2 "2024-11-27T01:33:08Z")

</div>

Arrow.jl does.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 27, 2024, 2:35pm UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/3 "2024-11-27T14:35:57Z")

</div>

Just as a side note, TableMetaDataTools.jl has utilities to save metadata as toml files and then re-load them and attach them to a data frame. So in theory you could save a .csv and a .toml together and load them both.

It won’t handle cross-language metadata obviously (though maybe someone should write an R package…)

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [November 27, 2024, 5:01pm UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/4 "2024-11-27T17:01:15Z")

</div>

> [@quinnj](#):
>
> [Arrow.jl](https://juliahub.com/ui/Packages/General/Arrow) does.

Are you sure? I’ve tried and it doesn’t seem to retain the metadata:

```julia
using Arrow
using DataFrames

df = DataFrame(a = 1:3, b= 'A':'C')
Arrow.write("test.arrow", df)
df = DataFrame(Arrow.Table("test.arrow"))

colmetadata!(df, :a, "test", "hope this works"; style = :note)
colmetadata(df, :a, "test")

Arrow.write("test2.arrow", df)
df = DataFrame(Arrow.Table("test2.arrow"))
colmetadata(df, :a, "test")

```

```julia
ERROR: ArgumentError: no column-level metadata found for column "a"
Stacktrace:
 [1] colmetadata(df::DataFrame, col::Symbol, key::String, default::DataFrames.MetadataMissingDefault; style::Bool)
   @ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/other/metadata.jl:367
 [2] colmetadata
   @ ~/.julia/packages/DataFrames/kcA9R/src/other/metadata.jl:360 [inlined]
 [3] colmetadata(df::DataFrame, col::Symbol, key::String)
   @ DataFrames ~/.julia/packages/DataFrames/kcA9R/src/other/metadata.jl:360
 [4] top-level scope
   @ ~/Documents/GitHub/ItsLivePlayground.jl/src/RiverTest.jl:41

```

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [November 27, 2024, 5:02pm UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/5 "2024-11-27T17:02:48Z")

</div>

> [@pdeffebach](#):
>
> Just as a side note, [TableMetaDataTools.jl](https://juliahub.com/ui/Packages/General/TableMetaDataTools) has utilities to save metadata as toml files and then re-load them and attach them to a data frame. So in theory you could save a .csv and a .toml together and load them both.

Oh, that’s good to know.

Do you know of a file format that is able to store the metadata in the same file as the table?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 27, 2024, 5:07pm UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/6 "2024-11-27T17:07:54Z")

</div>

No, no one has made that file format.

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [November 27, 2024, 5:23pm UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134/7 "2024-11-27T17:23:37Z")

</div>

I found [this thread here](https://discourse.julialang.org/t/reading-and-writing-arrow-column-metadata/102427/2) on how to append metadata to an Arrow file