# Writing dataframe to arrow format with column metadata

**URL:** <https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407>\
**Category:** Data\
**Created:** [September 29, 2023, 6:26pm UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407 "2023-09-29T18:26:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![alicepiller](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@alicepiller](https://discourse.julialang.org/u/alicepiller)\
**Post date:** [September 29, 2023, 6:26pm UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/1 "2023-09-29T18:26:40Z")

</div>

I’m trying to write a dataframe to an arrow file and include column metadata. The code below runs, but example\_metadata has none of the metadata.

using Arrow  
using DataFrames

df = DataFrame(x = [1, 2, 3], y = [4, 5, 6], z = [7, 8, 9])  
colmeta\_dict = Dict(:x =\> [“id” =\> “ID1”, “comment” =\> “Comment1”],  
:y =\> [“id” =\> “ID2”, “comment” =\> “Comment2”],  
:z =\> [“id” =\> “ID1”, “comment” =\> “Comment1”])  
Arrow.write(“example.arrow”, df, colmetadata = colmeta\_dict)  
arrow\_tbl = Arrow.Table(“example.arrow”)  
example\_metadata = Arrow.getmetadata(arrow\_tbl)

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 29, 2023, 7:49pm UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/2 "2023-09-29T19:49:06Z")

</div>

It has column metadata (not table metadata):

```julia
julia> example_metadata = Arrow.getmetadata(arrow_tbl.x)
Base.ImmutableDict{String, String} with 2 entries:
  "comment" => "Comment1"
  "id" => "ID1"

julia> example_metadata = Arrow.getmetadata(arrow_tbl.y)
Base.ImmutableDict{String, String} with 2 entries:
  "comment" => "Comment2"
  "id" => "ID2"

julia> example_metadata = Arrow.getmetadata(arrow_tbl.z)
Base.ImmutableDict{String, String} with 2 entries:
  "comment" => "Comment1"
  "id" => "ID1"

```

See also [Add reading metadata from Arrow.Table by bkamins · Pull Request #481 · apache/arrow-julia · GitHub](https://github.com/apache/arrow-julia/pull/481) in the future.

CC @quinnj

---

<div class="post-metadata">

**Author:** ![alicepiller](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@alicepiller](https://discourse.julialang.org/u/alicepiller)\
**Post date:** [October 1, 2023, 4:13pm UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/3 "2023-10-01T16:13:40Z")

</div>

Thank you! That was an easy solution! I have another problem. When reading the file back to a data frame, the column metadata is lost. How can one retain the column metadata such that it is callable with a function like colmetadata(df, :var, “label”)

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 1, 2023, 5:48pm UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/4 "2023-10-01T17:48:13Z")

</div>

Check out the [Add reading metadata from Arrow.Table by bkamins · Pull Request #481 · apache/arrow-julia · GitHub](https://github.com/apache/arrow-julia/pull/481) branch of Arrow.jl and this will be provided. The PR is ready so it should be safe to use it- it just waits for @quinnj to approve and a release.

---

<div class="post-metadata">

**Author:** ![alicepiller](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@alicepiller](https://discourse.julialang.org/u/alicepiller)\
**Post date:** [October 3, 2023, 9:28am UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/5 "2023-10-03T09:28:08Z")

</div>

Thanks for directing me to the resource. I still don’t see a way to read an arrow table as a dataframe and retain the metadata. Do you know if it’s possible?  
I’m looking to create a dataframe that has retained the metadata and column metadata from an arrow table with metadata and column metadata.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 3, 2023, 2:30pm UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/6 "2023-10-03T14:30:41Z")

</div>

> Do you know if it’s possible?

It is possible. As I have commented. You have to install the [GitHub - apache/arrow-julia at bk/metadata](https://github.com/apache/arrow-julia/tree/bk/metadata) branch of Tables.jl instead of released branch of this package and things will work.

---

<div class="post-metadata">

**Author:** ![alicepiller](https://avatars.discourse-cdn.com/v4/letter/a/e9bcb4/32.png) [@alicepiller](https://discourse.julialang.org/u/alicepiller)\
**Post date:** [October 6, 2023, 11:33am UTC](https://discourse.julialang.org/t/writing-dataframe-to-arrow-format-with-column-metadata/104407/7 "2023-10-06T11:33:08Z")

</div>

Thank you very much. I installed the branch and it works perfectly.