# Convert DataFrames 1.3 DataFrame to DataFrames 1.4

**URL:** <https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267>\
**Category:** General Usage\
**Tags:** dataframes\
**Created:** [October 5, 2022, 2:46am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267 "2022-10-05T02:46:43Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [October 5, 2022, 2:46am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/1 "2022-10-05T02:46:43Z")

</div>

I serialized a DataFrame from DataFrames.jl@1.3. Can I convert it to be compatible with DataFrames.jl@1.4 so I can load it in an environment with the new version?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 5, 2022, 3:15am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/2 "2022-10-05T03:15:21Z")

</div>

One option is to read it in and write it back out as an arrow file, then read it with newer DataFrames.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [October 5, 2022, 3:21am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/3 "2022-10-05T03:21:53Z")

</div>

This dataframe has complicated column types that aren’t natively supported by Arrow, so I’d have to write a lot of extra code to make that work.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 5, 2022, 3:51am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/4 "2022-10-05T03:51:48Z")

</div>

> [@jar1](#):
>
> complicated column types

Arrow should support self-descriptive structs, they have a metadata thingy in it, as long as your struct definition can be loaded again, you can load the same arrow file again

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 5, 2022, 7:44am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/5 "2022-10-05T07:44:18Z")

</div>

> [@jar1](#):
>
> I serialized a DataFrame from DataFrames.jl@1.3. Can I convert it to be compatible with DataFrames.jl@1.4 so I can load it in an environment with the new version?

Do the following:

- convert data frame using `Tables.columntable`
- serialize produced `NamedTuple`
- update to DataFrames.jl 1.4
- deserialize produced `NamedTuple`
- convert `NamedTuple` to a `DataFrame`

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [November 2, 2022, 9:19am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/6 "2022-11-02T09:19:08Z")

</div>

@bkamins is there a programmatic way to do this, if I have a couple of dozen JLD files based on DataFrames 1.3?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 2, 2022, 9:34am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/7 "2022-11-02T09:34:40Z")

</div>

An example.

Under DataFrames.jl 1.3 for all files you have

1. read `filename.jld` JLD file to `df` data frame
2. run `serialize("filename.bin", Tables.columntable(df))`

Now update to DataFrames.jl 1.4 and for all files you have:

1. run `df = DataFrame(deserialize("filename.bin"))`
2. write `df` as JLD file

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [November 2, 2022, 10:58am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/8 "2022-11-02T10:58:35Z")

</div>

> [@bkamins](#):
>
> Now update to DataFrames.jl 1.4 and for all files you have:

Thanks. Do I need to restart Julia after/before updating the DataFrames pkg? I think so.  
EDIT: I note that I disabled Revise, as it gave me ton’s of complaints.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 2, 2022, 11:27am UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/9 "2022-11-02T11:27:05Z")

</div>

Yes, you need to restart Julia.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [November 2, 2022, 12:33pm UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/10 "2022-11-02T12:33:44Z")

</div>

While this is your solution, that one (or converting to Arrow) seem very inconvenient, even with the programmatic way. Can’t it be done in DataFrame.jl 1.4 (or 1.5?) to do it for you when you open older (by default, or if problematic, as opt-in)? What’s done in other languages, e.g. pandas (or for pickle) do the just support saved data from older versions?

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [November 2, 2022, 12:42pm UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/11 "2022-11-02T12:42:51Z")

</div>

Along your idea: I keep thinking, that a smart person would be able to start up multiple julia sessions that talk to each other (most primitive variant would be through binary files on disk). One using DataFrames 1.3 and the other using DataFrames 1.4

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 2, 2022, 1:39pm UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/12 "2022-11-02T13:39:49Z")

</div>

> [@Palli](#):
>
> Can’t it be done in DataFrame.jl 1.4 (or 1.5?) to do it for you when you open older (by default, or if problematic, as opt-in)?

What you ask for cannot be done in DataFrames.jl as it is not DataFrames.jl that “opens” the stored data. It is JLD.jl in this case that would need to add such an extension (which is not very likely I assume).

Indeed the process is a bit cumbersome, but it is a one-time action. Here are scripts that do all what is needed. I assume that you want to convert `"old.jld"` to `"new.jld"` (not tested - I have written if from my head)

main.sh

```julia
mkdir old
mkdir new
cd old
julia old.jl
julia new.jl

```

old.jl

```julia
using Pkg
Pkg.activate("old")
Pkg.add("DataFrames", version="1.3.6")
Pkg.add("JLD", version="XXXX") # JLD version you used to save data
using DataFrames
using JLD
using Serialization
df = load("old.jld")
serialize("df.bin", Tables.columntable(df))

```

new.jl

```julia
using Pkg
Pkg.activate("new")
Pkg.add("DataFrames", version="1.4.2")
Pkg.add("JLD", version="XXXX") # JLD version you want to use to save data with
using DataFrames
using JLD
using Serialization
df = DataFrame(deserialize("df.bin")) # or whatever name you want to use
save("new.jld", "df")

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [November 2, 2022, 1:47pm UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/13 "2022-11-02T13:47:38Z")

</div>

> [@jar1](#):
>
> I serialized a DataFrame from DataFrames.jl@1.3.

you really should be using Arrow.jl and just get back DataFrame every time

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 2, 2022, 1:57pm UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/14 "2022-11-02T13:57:58Z")

</div>

> [@jling](#):
>
> you really should be using Arrow.jl and just get back DataFrame every time

I assume @bernhard uses JLD because of custom columns that Arrow.jl cannot represent properly.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [November 2, 2022, 2:10pm UTC](https://discourse.julialang.org/t/convert-dataframes-1-3-dataframe-to-dataframes-1-4/88267/15 "2022-11-02T14:10:59Z")

</div>

> [@bkamins](#):
>
> What you ask for cannot be done in DataFrames.jl as it is not DataFrames.jl that “opens” the stored data. It is JLD.jl in this case that would need to add such an extension (which is not very likely I assume).

OK, I see it wasn’t strictly a problem with DF, as I assumed from “I serialized a DataFrame from DataFrames.jl@1.3”, that some DF function was used. It’s a one-time thing now, until maybe a later version. And even more important if it’s a problem with Julia’s (or JLD) serialization, to know if something better can be done with it, or to avoid the problem by not using it, and then what else?

I recall from Pandas, that Pandas has a function to store dataframes. I’m still curious if a problem in other languages. Should DF have such a function that would just work?