Issues reading CSV file with array elements

General Usage
,
1

The problem is as follows:
I have a dataframe with a column that has Vector{Float64} elements. When I store the dataframe in a CSV file and then retrieve it, the type of the column changed from Vector{Float64} to String. Which is really annoying since I need the values from within the Vector. A MWE is shown below:

using DataFrames, CSV

testDF = DataFrame([:columnA => Vector{Vector{Float64}}(undef,1)]) #Column type is Vector{Vector{Float64}}
testDF[1,:columnA] = [1.5,3.2,9.0]
CSV.write("C:\\Users\\Public\\testCSVfile",testDF)
testDF = DataFrame(CSV.File("C:\\Users\\Public\\testCSVfile")) #Column type is String?

How can I retain the type of my column?

2

normally, for types you would want to store in a format that remembers the type like arrow, parquet, or even JDF.jl.

but aren’t you storing an array at row 1 column :columnA? that’ seem to be the issue here

3

That is the whole point. I want to store an array in the dataframe as an element. JDF seems like a solution, but would it be possible to retrieve the data in the Python programming language? The main reason I use a CSV format is because it is also usable to read the file in Python.

4

ok then the data will be written out as "[10, 2, 20]" you need some code to parse that back as array.

So you want it to be read by python? not sure what’s a good, I think maybe arrow. The parquet writer in julia doesn’t do that yet.

so you just write some python code on the other end to parse that back into an array.

1 Like
5

I have asked more or less the same question a while ago here. Maybe some answers are helpful for you.

1 Like