Save data frame with special characters in csv

General Usage
,
1

The column names of my dataframe has special character like "€. Before updating to v1.3, I was able to write my df in a csv file and keeping the different symbols. However, after updating the csv file doesn’t recognize these symbols and I got a result like that:

image

I can reproduce this problem easily as follow:

using DataFrames, CSV

df = DataFrame([:TypeProduct => [1,2,3], Symbol("Coût - €/l") => [4,5,6]])

CSV.write("df.csv", df;delim=";")

Anyone knows how I can handle this problem?
Thanks.

2

Works fine for me in both 1.3 and 1.4:

julia> using DataFrames, CSV

julia> df = DataFrame([:TypeProduct => [1,2,3], Symbol("Coût - €/l") => [4,5,6]])
3×2 DataFrame
│ Row │ TypeProduct │ Coût - €/l │
│     │ Int64       │ Int64      │
├─────┼─────────────┼────────────┤
│ 1   │ 1           │ 4          │
│ 2   │ 2           │ 5          │
│ 3   │ 3           │ 6          │

julia> CSV.write("/tmp/df.csv", df;delim=";")
"/tmp/df.csv"

shell> cat /tmp/df.csv
TypeProduct;Coût - €/l
1;4
2;5
3;6

(@v1) pkg> st CSV DataFrames
Status `~/.julia/environments/v1/Project.toml`
  [336ed68f] CSV v0.5.22
  [a93c6f00] DataFrames v0.20.0

Are you sure that the software you are reading it with handles UTF8?

3

OP, it looks like you are using excel to view the files. Excel has pretty notorious problems with displaying unicode characters (and saving them wrong!).

Best to use a text editor to view the .csv file if it’s not too big.

4

The BOM Header is missing in the .csv file.
Try the following:

using DataFrames, CSV
df = DataFrame([:TypeProduct => [1,2,3], Symbol("Coût - €/l") => [4,5,6]])
CSV.write("df.csv", df;delim=";")

inStream = open("df.csv","r")
outStream = open("df_bom.csv","w")
print(outStream,"\xEF\xBB\xBF")  #UTF-8 BOM header
print(outStream, read(inStream,String))
close(outStream)
close(inStream)
1 Like
5

Opened an issue:
https://github.com/JuliaData/CSV.jl/issues/567

1 Like
6

Thanks a lot