Hi everyone, I get the following error message when trying to run readcsv(): at row 2, column 5 : UnicodeError: invalid character index) in readcsv at base\datafmt.jl:619 in #readcsv#15 at base\datafmt.jl:619 <inlined> in readdlm at base\datafmt.jl:54 in #readdlm#4 at base\datafmt.jl:54 <inlined> …

I also got error at 104000-th line. If I simplify it then problem is next: #this is good! julia> CSV.readsplitline(IOBuffer("104652,\"Thanks \\ \",a")) 3-element Array{CSV.RawField,1}: CSV.RawField("104652", false) CSV.RawField("Thanks \\ ", true) CSV.RawField("a", false) #this is suspi…

I tried also CSV.read(), but it gave me the following error message: CSV.CSVError("Unexpected start of quote (34), use \"9234\" to type \"34\"") in read at CSV\src\Source.jl:294 in #read#29 at CSV\src\Source.jl:294 in #Source#11 at CSV\src\Source.jl:25 in at base\<missing> in #Source#12 at CSV\sr…

You could also try https://github.com/davidanthoff/CSVFiles.jl . It uses GitHub - queryverse/TextParse.jl: A bunch of fast text parsing tools under the hood, and I have no idea whether it will be able to deal with that file, but probably worth a try.

Sadly, something similar there: UnicodeError: invalid character index in DataFrames.DataFrame at IterableTables\src\integrations\dataframes-dataarray.jl:127 in _DataFrame at IterableTables\src\integrations\dataframes-dataarray.jl:98 in getiterator at CSVFiles\src\CSVFiles.jl:31 in open at base\iost…

Hm, maybe this just isn’t encoded as UTF-8?

Is there a way I can check that or change how the file is encoded? I saved the file on my own computer using Python and want to do some more manipulations with Julia. I would be thankful for any information! Edit: I found a way to encode my CSV in unicode in Python. I will see if Julia can work wit…

Instead of cutting and pasting (which might do some transformations), could you upload the first few lines of the file, maybe as a gist? That way it would be possible to see the raw data

It is likely it’s the fancy quotes that are being used, that are causing problems, i.e.: ‘“’: Unicode U+201c (category Pi: Punctuation, initial quote) ‘”’: Unicode U+201d (category Pf: Punctuation, final quote) Those are not really normal to see in a plain CSV file (maybe something that Excel d…

I’ve run into this a few times: Readdlm: ignore_invalid_chars option does not exist anymore . There used to be the very helpful option ignore_invalid_chars in dlmread: https://docs.julialang.org/en/release-0.4/stdlib/io-network/ which magically handled my issues. I in my case CSV.jl did work though…

uCSV.jl seems to work well when I get rid of the text and only use the entries with the date and the numbers, but it seems to do nothing, when I try to load the big dataset with all variables. I’ll let it run for some time now and look if something happens.

[image] IljaK91: I saved the file on my own computer using Python and want to do some more manipulations with Julia. Can’t you use some other format to transfer the data? HDF5 maybe?

Unicode related error when reading a .csv

General Usage

vchuravy Split this topic November 24, 2017, 3:13pm 21

4 posts were split to a new topic: Changes to the representation of Char

Topic		Replies	Views
Read special characters using CSV.read New to Julia csv	22	1241	October 11, 2023
Readcsv julia 0.6.0 not working properly General Usage question	3	1061	March 28, 2017
Encoding, CSV.read / write General Usage package	5	2334	August 17, 2018
CSV won't read tab separated file General Usage csv	23	778	March 4, 2024
Julia CSV.read stopped working Data csv	32	3070	April 30, 2022

Unicode related error when reading a .csv

Related topics