# Saving a DataFrame containing vectors

**URL:** <https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477>\
**Category:** Data\
**Tags:** question\
**Created:** [May 27, 2023, 10:33am UTC](https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477 "2023-05-27T10:33:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![krish-rmsh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krish-rmsh/32/50248_2.png) [@krish-rmsh](https://discourse.julialang.org/u/krish-rmsh)\
**Post date:** [May 27, 2023, 10:33am UTC](https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477/1 "2023-05-27T10:33:25Z")

</div>

I am trying to save a DataFrame, where one of the columns (Say “e\_list”) contains vectors of size 40.  
I tried the following packages, and none of them seem to work.

- CSV  
When opening the .csv file stored using the CSV package, the vectors are converted into a string.
- JLD  
JLD takes forever to open the saved file. (It eventually never opened)
- HDF5  
Will not let me save the DataFrame.
- Parquet2  
Saves the data frame, but then when opening it, the vectors are stored as `Union{AbstractDict{String, Any}, Vector{Any}}`. This makes resaving this data frame impossible, as `Union{AbstractDict{String, Any}, Vector{Any}}` is not a standard datatype for Parquet2.

What would you think would be the right way to do it?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [May 27, 2023, 10:46am UTC](https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477/2 "2023-05-27T10:46:59Z")

</div>

Did you try just serializing/deserializing them?

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [May 27, 2023, 11:24am UTC](https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477/3 "2023-05-27T11:24:27Z")

</div>

> [@krish-rmsh](#):
>
> JLD  
> JLD takes forever to open the saved file. (It eventually never opened)

Did you try JLD2? To be honest, I didn’t realize JLD was still maintained.

> [@krish-rmsh](#):
>
> CSV  
> When opening the .csv file stored using the CSV package, the vectors are converted into a string.

Yeah, CSV is not a particularly flexible format. That said, you could try calling `Meta.parse()` on the strings after they’re loaded 🤷

The option I would probably use if your Vectors are all the same length is to stack / unstack therm. That is, if you have 5 vectors of length 10, expand that to 50 rows, adding a column for `vector_num` or sometime

> [@krish-rmsh](#):
>
> Parquet2

You could open an issue for this - I know ExpandingMan is trying hard to make this package functional.

You might also try Arrow.jl, though I don’t know if it’s support for in-cell vector types is better.

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [May 27, 2023, 12:17pm UTC](https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477/4 "2023-05-27T12:17:30Z")

</div>

> [@kevbonham](#):
>
> You might also try Arrow.jl, though I don’t know if it’s support for in-cell vector types is better

It does support arbitrarily nested vectors (at least all the variants I have tried).

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [May 27, 2023, 3:50pm UTC](https://discourse.julialang.org/t/saving-a-dataframe-containing-vectors/99477/5 "2023-05-27T15:50:06Z")

</div>

I am [patching Parquet2](https://gitlab.com/ExpandingMan/Parquet2.jl/-/merge_requests/22) so that it will try bson/json for arrays with eltype `Union{AbstractDict,AbstractVector}`. However, Parquet2.jl does not truly support nested data types, and defaulting to json/bson is a bit of an undesirable “hack”. In fact, at some point I’m going to have to change it so that this doesn’t happen by default (though there will be an option to restore the current behavior). In the meantime, I agree that it makes no sense for it to refuse to write its own output.

I strongly suggest using Arrow.jl if possible, as arrow supports true arbitrarily nested data types with its own binary format and will allow for type-stable reading and writing, which Parquet2.jl can’t do unless or until support for nested data structures is added.