Hi, I’m trying to read a file ( here ) that contains the following (as it appears in Excel): [image] Reading the data file, reads the special character incorrectly: MP_names = CSV.read("All MPs 23.07.csv", DataFrame) println(MP_names[130, [:Constituency]]) println("test ", lowercase("Ynys Môn")) p…

The ISO-8859-1 example from the docs works fine for me? [image]

That’s probably a non-UTF 8 encoding, check the CSV docs for how to combine it with StringEncodings here: https://csv.juliadata.org/stable/examples.html#stringencodings

How can I find out what encoding to use? I’m suddenly out of my depth here! I can save the file from Excel to a Unicode txt file but CSV.read won’t open that at all. MP_names = CSV.read("All MPs 23.07.txt", DataFrame) ERROR: LoadError: ArgumentError: Symbol name may not contain \0 Stacktrace: …

Could you try saving a file (the minimum possible) containing these particular characters and then applying the following functions to it to see what it contains? buf=Vector{UInt8}(undef, 100) fnz=readbytes!(open("are-there-non-utf-code.txt","r"),buf ) Char.(buf[1:fnz]) String(buf[1:fnz]) I tried …

[image] rocco_sprmnt21: buf=Vector{UInt8}(undef, 100) fnz=readbytes!(open("are-there-non-utf-code.txt","r"),buf ) Char.(buf[1:fnz]) String(buf[1:fnz]) [‘T’, ‘h’, ‘é’, ‘r’, ‘è’, ‘s’, ‘e’, ‘,’, ‘S’, ‘i’, ‘n’, ‘n’, ’ ', ‘F’, ‘é’, ‘i’, ‘n’, ‘,’, ‘Y’, ‘n’, ‘y’, ‘s’, ’ ', ‘M’, ‘ô’, ‘n’, ‘\r’, ‘\n’] …

Excel shows the CSV file like this: [image] If I try (in VScode): file = CSV.File(open(“All MPs 23.07.csv”)) println(file) I get CSV.File(“<IOStream: 2980782706580401464>”): Size: 0 x 3 Tables.Schema: ERROR: LoadError: Base.InvalidCharError{Char}(‘\xe9’) and a very long stacktrace: S…

[image] nilshg: ISO-8859-1 Yes, this works for me, too! Thanks! How would I know a priori that my file is ISO-8859-1 encoded? I’ve looked it up and it seems to be a bit of an out of data scheme.

Another possibility here is to coax Excel into saving the file in UTF-8, which must be possible somehow. That’s pretty much the encoding everyone uses today.

[image] StefanKarpinski: which must be possible somehow [Unbenannt]

[image] TimG: How would I know a priori that my file is ISO-8859-1 encoded? I’ve looked it up and it seems to be a bit of an out of data scheme. It’s probably Windows-1252 , actually, which is almost the same as ISO-8859-1. It’s simple — Windows-1252 seems to be the only non-UTF8 extension of…

[image] TimG: How would I know a priori that my file is ISO-8859-1 encoded? I’ve looked it up and it seems to be a bit of an out of data scheme. I don’t know the topic, but, making sense, I would say that a logic of the type here mentioned is used. I used the first 3 bytes of Char.(buf[1:fn…

Read special characters using CSV.read

New to Julia

nilshg October 5, 2023, 12:31pm 7

The ISO-8859-1 example from the docs works fine for me?

Topic		Replies	Views
Unicode related error when reading a .csv General Usage	21	6280	November 25, 2017
Encoding, CSV.read / write General Usage package	5	2345	August 17, 2018
CSV won't read tab separated file General Usage csv	23	790	March 4, 2024
Julia CSV.read stopped working Data csv	32	3108	April 30, 2022
Find string with special characters in data frame read with the CSV package Data strings , dataframes , csv	14	2535	August 22, 2023

Read special characters using CSV.read

Related topics