### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 5, 2023, 9:30am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/1 "2023-10-05T09:30:24Z")
Hi, I’m trying to read a file ([here](https://www.parliament.uk/site-information/freedom-of-information/information-we-already-publish/house-of-commons-publication-scheme/members-and-members-staff/parliamentary-contact-details-for-mps/)) that contains the following (as it appears in Excel):
Reading the data file, reads the special character incorrectly:
> ```
> MP_names = CSV.read("All MPs 23.07.csv", DataFrame)
> println(MP_names[130, [:Constituency]])
> println("test ", lowercase("Ynys Môn"))
> println("data", lowercase(MP_names[130, :Constituency]))
>
> ```
> DataFrameRow
> Row │ Constituency
> │ String
> ─────┼──────────────
> 130 │ Ynys M\xf4n
> test ynys môn
> ERROR: LoadError: Base.InvalidCharError{Char}(‘\xf4’)
> Stacktrace:
> [1] throw\_invalid\_char(c::Char)
> @ Base .\char.jl:86
> [2] UInt32
> @ .\char.jl:133 [inlined]
> [3] convert
> @ .\char.jl:185 [inlined]
> [4] cconvert
> @ .\essentials.jl:492 [inlined]
> [5] lowercase(c::Char)
> @ Base.Unicode .\strings\unicode.jl:289
> [6] map(f::typeof(lowercase), s::String)
> @ Base .\strings\basic.jl:622
> [7] lowercase(s::String)
> @ Base.Unicode .\strings\unicode.jl:622
> [8] macro expansion
> @ c:\Users.…\MPs and Constituencies.jl:24 [inlined]
> [9] top-level scope
> @ .\timing.jl:273
> in expression starting at c:\Users.…\MPs and Constituencies.jl:16
How can I read this character properly?
Thanks
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [October 5, 2023, 9:50am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/2 "2023-10-05T09:50:28Z")
That’s probably a non-UTF 8 encoding, check the CSV docs for how to combine it with `StringEncodings` here:
[https://csv.juliadata.org/stable/examples.html#stringencodings](https://csv.juliadata.org/stable/examples.html#stringencodings)
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 5, 2023, 10:13am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/3 "2023-10-05T10:13:56Z")
How can I find out what encoding to use? I’m suddenly out of my depth here!
I can save the file from Excel to a Unicode txt file but CSV.read won’t open that at all.
> ```
> MP_names = CSV.read("All MPs 23.07.txt", DataFrame)
>
> ```
> ERROR: LoadError: ArgumentError: Symbol name may not contain \0
> Stacktrace:
> [1] \_Symbol
> @ .\boot.jl:509 [inlined]
> [2] Symbol
> @ .\boot.jl:515 [inlined]
> [3] #10
> @ .\none:0 [inlined]
> [4] iterate
> @ .\generator.jl:47 [inlined]
> [5] collect(itr::Base.Generator{Vector{String}, CSV.var"#10#13"{Bool}})
> @ Base .\array.jl:782
> [6] detectcolumnnames(buf::Vector{UInt8}, headerpos::Int64, datapos::Int64, len::Int64, options::Parsers.Options, header::Any, normalizenames::Bool, oq::UInt8, eq::UInt8, cq::UInt8, cmt::Nothing, ignoreemptyrows::Bool)
> @ CSV C:\Users\TGebbels.julia\packages\CSV\OnldF\src\detection.jl:185
> [7] CSV.Context(source::CSV.Arg, header::CSV.Arg, normalizenames::CSV.Arg, datarow::CSV.Arg, skipto::CSV.Arg, footerskip::CSV.Arg, transpose::CSV.Arg, comment::CSV.Arg, ignoreemptyrows::CSV.Arg, ignoreemptylines::CSV.Arg, select::CSV.Arg, drop::CSV.Arg, limit::CSV.Arg, buffer\_in\_memory::CSV.Arg, threaded::CSV.Arg, ntasks::CSV.Arg, tasks::CSV.Arg, rows\_to\_check::CSV.Arg, lines\_to\_check::CSV.Arg, missingstrings::CSV.Arg, missingstring::CSV.Arg, delim::CSV.Arg, ignorerepeated::CSV.Arg, quoted::CSV.Arg, quotechar::CSV.Arg, openquotechar::CSV.Arg, closequotechar::CSV.Arg, escapechar::CSV.Arg, dateformat::CSV.Arg, dateformats::CSV.Arg, decimal::CSV.Arg, groupmark::CSV.Arg, truestrings::CSV.Arg, falsestrings::CSV.Arg, stripwhitespace::CSV.Arg, type::CSV.Arg, types::CSV.Arg, typemap::CSV.Arg, pool::CSV.Arg, downcast::CSV.Arg, lazystrings::CSV.Arg, stringtype::CSV.Arg, strict::CSV.Arg, silencewarnings::CSV.Arg, maxwarnings::CSV.Arg, debug::CSV.Arg, parsingdebug::CSV.Arg, validate::CSV.Arg, streaming::CSV.Arg)
> @ CSV C:\Users\TGebbels.julia\packages\CSV\OnldF\src\context.jl:470
> [8] #file#32
> @ C:\Users\TGebbels.julia\packages\CSV\OnldF\src\file.jl:222 [inlined]
> [9] CSV.File(source::String)
> @ CSV C:\Users\TGebbels.julia\packages\CSV\OnldF\src\file.jl:162
> [10] #read#118
> @ C:\Users\TGebbels.julia\packages\CSV\OnldF\src\CSV.jl:117 [inlined]
> [11] read
> @ C:\Users\TGebbels.julia\packages\CSV\OnldF\src\CSV.jl:113 [inlined]
> [12] macro expansion
> @ c:\Users.…\MPs and Constituencies.jl:22 [inlined]
> [13] top-level scope
> @ .\timing.jl:273
> in expression starting at c:\Users.…\MPs and Constituencies.jl:16
---
### Author:  [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [October 5, 2023, 11:07am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/4 "2023-10-05T11:07:46Z")
Could you try saving a file (the minimum possible) containing these particular characters and then applying the following functions to it to see what it contains?
```julia
buf=Vector{UInt8}(undef, 100)
fnz=readbytes!(open("are-there-non-utf-code.txt","r"),buf )
Char.(buf[1:fnz])
String(buf[1:fnz])
```
I tried saving the file containing only the following two lines using the UTF8 format [I used Word on Windows system]:
AEIÔU
aeiôu
getting this:
```julia
julia> fnz=readbytes!(open("non-utf-code.txt","r"),buf )
20
julia> Char.(buf[1:fnz])
20-element Vector{Char}:
'ï': Unicode U+00EF (category Ll: Letter, lowercase)
'»': Unicode U+00BB (category Pf: Punctuation, final quote)
'¿': Unicode U+00BF (category Po: Punctuation, other)
'A': ASCII/Unicode U+0041 (category Lu: Letter, uppercase)
'E': ASCII/Unicode U+0045 (category Lu: Letter, uppercase)
'I': ASCII/Unicode U+0049 (category Lu: Letter, uppercase)
'Ã': Unicode U+00C3 (category Lu: Letter, uppercase)
'\u94': Unicode U+0094 (category Cc: Other, control)
'U': ASCII/Unicode U+0055 (category Lu: Letter, uppercase)
' ': ASCII/Unicode U+0020 (category Zs: Separator, space)
'\r': ASCII/Unicode U+000D (category Cc: Other, control)
'\n': ASCII/Unicode U+000A (category Cc: Other, control)
'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
'e': ASCII/Unicode U+0065 (category Ll: Letter, lowercase)
'i': ASCII/Unicode U+0069 (category Ll: Letter, lowercase)
'Ã': Unicode U+00C3 (category Lu: Letter, uppercase)
'´': Unicode U+00B4 (category Sk: Symbol, modifier)
'u': ASCII/Unicode U+0075 (category Ll: Letter, lowercase)
'\r': ASCII/Unicode U+000D (category Cc: Other, control)
'\n': ASCII/Unicode U+000A (category Cc: Other, control)
julia> String(buf[1:fnz])
"\ufeffAEIÔU \r\naeiôu\r\n"
```
and this:
```julia
julia> using CSV, DataFrames
julia> file = CSV.File(open("are-there-non-utf-code.txt"))
1-element CSV.File:
CSV.Row: (AEIÔU = String7("aeiôu"),)
julia> file |> DataFrame
1×1 DataFrame
Row │ AEIÔU
│ String7
─────┼─────────
1 │ aeiôu
```
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 5, 2023, 11:30am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/5 "2023-10-05T11:30:42Z")
> [@rocco\_sprmnt21](#):
>
> ```julia
> buf=Vector{UInt8}(undef, 100)
> fnz=readbytes!(open("are-there-non-utf-code.txt","r"),buf )
> Char.(buf[1:fnz])
> String(buf[1:fnz])
>
> ```
[‘T’, ‘h’, ‘é’, ‘r’, ‘è’, ‘s’, ‘e’, ‘,’, ‘S’, ‘i’, ‘n’, ‘n’, ’ ', ‘F’, ‘é’, ‘i’, ‘n’, ‘,’, ‘Y’, ‘n’, ‘y’, ‘s’, ’ ', ‘M’, ‘ô’, ‘n’, ‘\r’, ‘\n’]
Th�r�se,Sinn F�in,Ynys M�n
I did this in VSCode:
> buf = Vector{UInt8}(undef, 100)
> fnz = readbytes!(open(“All MPs 23.07.csv”, “r”), buf)
> println(Char.(buf[1:fnz]))
> println(String(buf[1:fnz]))
I cut down the csv file in Excel by deleting surrounding cells (not touching the offending cells at all). In notepad I can see:
> Thérèse,Sinn Féin,Ynys Môn
but if I open the whole (original) CSV file in Notepad, the same cells show as:
> Th r se
> Sinn F in
> Ynys M n
The apparently empty spaces are filled (in notepad) with a square with a ? inside. In Excel they display correctly.
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 5, 2023, 11:44am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/6 "2023-10-05T11:44:36Z")
Excel shows the CSV file like this:
If I try (in VScode):
> file = CSV.File(open(“All MPs 23.07.csv”))
> println(file)
I get
> CSV.File(“\”):
> Size: 0 x 3
> Tables.Schema:
> ERROR: LoadError: Base.InvalidCharError{Char}(‘\xe9’)
and a very long stacktrace:
> Stacktrace:
> [1] throw\_invalid\_char(c::Char)
> @ Base .\char.jl:86
> [2] UInt32
> @ .\char.jl:133 [inlined]
> [3] convert
> @ .\char.jl:185 [inlined]
> [4] cconvert
> @ .\essentials.jl:492 [inlined]
> [5] is\_id\_char(c::Char)
> @ Base .\show.jl:1414
> [6] \_all(f::typeof(Base.is\_id\_char), itr::Base.Iterators.Rest{String, Int64}, #unused#::Colon)
> @ Base .\reduce.jl:1283
> [7] all
> @ .\reduce.jl:1278 [inlined]
> [8] isidentifier(s::String)
> @ Base .\show.jl:1442
> [9] isidentifier
> @ .\show.jl:1444 [inlined]
> [10] show\_unquoted\_quote\_expr(io::IOContext{IOBuffer}, value::Any, indent::Int64, prec::Int64, quote\_level::Int64)
> @ Base .\show.jl:1757
> [11] show(io::IOContext{IOBuffer}, s::Symbol)
> @ Base .\show.jl:1346
> [12] sprint(f::Function, args::Symbol; context::IOContext{Base.TTY}, sizehint::Int64)
> @ Base .\strings\io.jl:112
> [13] sprint
> @ .\strings\io.jl:107 [inlined]
> [14] alignment\_from\_show
> @ .\show.jl:2817 [inlined]
> [15] alignment(io::Base.TTY, x::Symbol)
> @ Base .\show.jl:2836
> [16] alignment(io::Base.TTY, X::AbstractVecOrMat, rows::Vector{Int64}, cols::Vector{Int64}, cols\_if\_complete::Int64, cols\_otherwise::Int64, sep::Int64, ncols::Int64)
> @ Base .\arrayshow.jl:69
> [17] \_print\_matrix(io::Base.TTY, X::AbstractVecOrMat, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64, rowsA::UnitRange{Int64}, colsA::UnitRange{Int64})
> @ Base .\arrayshow.jl:207
> [18] print\_matrix(io::Base.TTY, X::Matrix{Any}, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64)
> @ Base .\arrayshow.jl:171
> [19] print\_matrix
> @ .\arrayshow.jl:171 [inlined]
> [20] show(io::Base.TTY, sch::ERROR: Base.InvalidCharError{Char}(‘\xe9’)
> Stacktrace:
> [1] throw\_invalid\_char(c::Char)
> @ Base .\char.jl:86
> [2] UInt32
> @ .\char.jl:133 [inlined]
> [3] convert
> @ .\char.jl:185 [inlined]
> [4] cconvert
> @ .\essentials.jl:492 [inlined]
> [5] is\_id\_char(c::Char)
> @ Base .\show.jl:1414
> [6] \_all(f::typeof(Base.is\_id\_char), itr::Base.Iterators.Rest{String, Int64}, #unused#::Colon)
> @ Base .\reduce.jl:1283
> [7] all
> @ .\reduce.jl:1278 [inlined]
> [8] isidentifier(s::String)
> @ Base .\show.jl:1442
> [9] isidentifier
> @ .\show.jl:1444 [inlined]
> [10] show\_unquoted\_quote\_expr(io::IOContext{IOBuffer}, value::Any, indent::Int64, prec::Int64, quote\_level::Int64)
> @ Base .\show.jl:1757
> [11] show
> @ .\show.jl:1346 [inlined]
> [12] show\_delim\_array(io::IOContext{IOBuffer}, itr::Tuple{Symbol, Symbol, Symbol}, op::Char, delim::Char, cl::Char, delim\_one::Bool, i1::Int64, n::Int64)
> @ Base .\show.jl:1325
> [13] show\_delim\_array
> @ .\show.jl:1310 [inlined]
> [14] show(io::IOContext{IOBuffer}, t::Tuple{Symbol, Symbol, Symbol})
> @ Base .\show.jl:1343
> [15] show\_typeparams(io::IOContext{IOBuffer}, env::Core.SimpleVector, orig::Core.SimpleVector, wheres::Vector{TypeVar})
> @ Base .\show.jl:707
> [16] show\_datatype(io::IOContext{IOBuffer}, x::DataType, wheres::Vector{TypeVar})
> @ Base .\show.jl:1092
> [17] show\_datatype
> @ .\show.jl:1058 [inlined]
> [18] \_show\_type(io::IOContext{IOBuffer}, x::Type)
> @ Base .\show.jl:958
> [19] show(io::IOContext{IOBuffer}, x::Type)
> @ Base .\show.jl:950
> [20] sprint(f::Function, args::Type; context::IOContext{Base.TTY}, sizehint::Int64)
> @ Base .\strings\io.jl:112
> [21] sprint
> @ .\strings\io.jl:107 [inlined]
> [22] #print\_type\_bicolor#540
> @ .\show.jl:2491 [inlined]
> [23] show\_tuple\_as\_call(io::IOContext{Base.TTY}, name::Symbol, sig::Type; demangle::Bool, kwargs::Nothing, argnames::Vector{Symbol}, qualified::Bool, hasfirst::Bool)
> @ Base .\show.jl:2472
> [24] show\_tuple\_as\_call
> @ .\show.jl:2441 [inlined]
> [25] show\_spec\_linfo(io::IOContext{Base.TTY}, frame::Base.StackTraces.StackFrame)
> @ Base.StackTraces .\stacktraces.jl:244
> [26] print\_stackframe(io::IOContext{Base.TTY}, i::Int64, frame::Base.StackTraces.StackFrame, n::Int64, ndigits\_max::Int64, modulecolor::Symbol)
> @ Base .\errorshow.jl:730
> [27] print\_stackframe(io::IOContext{Base.TTY}, i::Int64, frame::Base.StackTraces.StackFrame, n::Int64, ndigits\_max::Int64, modulecolordict::IdDict{Module, Symbol}, modulecolorcycler::Base.Iterators.Stateful{Base.Iterators.Cycle{Vector{Symbol}}, Union{Nothing, Tuple{Symbol, Int64}}, Int64})
> @ Base .\errorshow.jl:695
> [28] show\_full\_backtrace(io::IOContext{Base.TTY}, trace::Vector{Any}; print\_linebreaks::Bool)
> @ Base .\errorshow.jl:594
> [29] show\_full\_backtrace
> @ .\errorshow.jl:587 [inlined]
> [30] show\_backtrace(io::IOContext{Base.TTY}, t::Vector{Base.StackTraces.StackFrame})
> @ Base .\errorshow.jl:791
> [31] showerror(io::IOContext{Base.TTY}, ex::Base.InvalidCharError{Char}, bt::Vector{Base.StackTraces.StackFrame}; backtrace::Bool)
> @ Base .\errorshow.jl:90
> [32] showerror(io::IOContext{Base.TTY}, ex::LoadError, bt::Vector{Base.StackTraces.StackFrame}; backtrace::Bool)
> @ Base .\errorshow.jl:96
> [33] show\_exception\_stack(io::IOContext{Base.TTY}, stack::Base.ExceptionStack)
> @ Base .\errorshow.jl:895
> [34] display\_error(io::Base.TTY, stack::Base.ExceptionStack)
> @ Base .\client.jl:111
> [35] display\_error(stack::Base.ExceptionStack)
> @ Base .\client.jl:114
> [36] #invokelatest#2
> @ .\essentials.jl:819 [inlined]
> [37] invokelatest
> @ .\essentials.jl:816 [inlined]
> [38] exec\_options(opts::Base.JLOptions)
> @ Base .\client.jl:310
> [39] \_start()
> @ Base .\client.jl:522
>
> caused by: LoadError: Base.InvalidCharError{Char}(‘\xe9’)
> Stacktrace:
> [1] throw\_invalid\_char(c::Char)
> @ Base .\char.jl:86
> [2] UInt32
> @ .\char.jl:133 [inlined]
> [3] convert
> @ .\char.jl:185 [inlined]
> [4] cconvert
> @ .\essentials.jl:492 [inlined]
> [5] is\_id\_char(c::Char)
> @ Base .\show.jl:1414
> [6] \_all(f::typeof(Base.is\_id\_char), itr::Base.Iterators.Rest{String, Int64}, #unused#::Colon)
> @ Base .\reduce.jl:1283
> [7] all
> @ .\reduce.jl:1278 [inlined]
> [8] isidentifier(s::String)
> @ Base .\show.jl:1442
> [9] isidentifier
> @ .\show.jl:1444 [inlined]
> [10] show\_unquoted\_quote\_expr(io::IOContext{IOBuffer}, value::Any, indent::Int64, prec::Int64, quote\_level::Int64)
> @ Base .\show.jl:1757
> [11] show(io::IOContext{IOBuffer}, s::Symbol)
> @ Base .\show.jl:1346
> [12] sprint(f::Function, args::Symbol; context::IOContext{Base.TTY}, sizehint::Int64)
> @ Base .\strings\io.jl:112
> [13] sprint
> @ .\strings\io.jl:107 [inlined]
> [14] alignment\_from\_show
> @ .\show.jl:2817 [inlined]
> [15] alignment(io::Base.TTY, x::Symbol)
> @ Base .\show.jl:2836
> [16] alignment(io::Base.TTY, X::AbstractVecOrMat, rows::Vector{Int64}, cols::Vector{Int64}, cols\_if\_complete::Int64, cols\_otherwise::Int64, sep::Int64, ncols::Int64)
> @ Base .\arrayshow.jl:69
> [17] \_print\_matrix(io::Base.TTY, X::AbstractVecOrMat, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64, rowsA::UnitRange{Int64}, colsA::UnitRange{Int64})
> @ Base .\arrayshow.jl:207
> [18] print\_matrix(io::Base.TTY, X::Matrix{Any}, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64)
> @ Base .\arrayshow.jl:171
> [19] print\_matrix
> @ .\arrayshow.jl:171 [inlined]
> [20] show(io::Base.TTY, sch::fatal: error thrown and no exception handler available.
> Base.InvalidCharError{Char}(char=Char(0xe9000000))
> throw\_invalid\_char at .\char.jl:86
> UInt32 at .\char.jl:133 [inlined]
> convert at .\char.jl:185 [inlined]
> cconvert at .\essentials.jl:492 [inlined]
> is\_id\_char at .\show.jl:1414
> \_all at .\reduce.jl:1283
> all at .\reduce.jl:1278 [inlined]
> isidentifier at .\show.jl:1442
> isidentifier at .\show.jl:1444 [inlined]
> show\_unquoted\_quote\_expr at .\show.jl:1757
> show at .\show.jl:1346 [inlined]
> show\_delim\_array at .\show.jl:1325
> show\_delim\_array at .\show.jl:1310 [inlined]
> show at .\show.jl:1343
> unknown function (ip: 000002202952bb5a)
> show\_typeparams at .\show.jl:707
> show\_datatype at .\show.jl:1092
> show\_datatype at .\show.jl:1058 [inlined]
> \_show\_type at .\show.jl:958
> show at .\show.jl:950
> jfptr\_show\_49738.clone\_1 at C:\Users\TGebbels\AppData\Local\Programs\Julia-1.9.3\lib\julia\sys.dll (unknown line)
> #sprint#484 at .\strings\io.jl:112
> sprint at .\strings\io.jl:107 [inlined]
> #print\_type\_bicolor#540 at .\show.jl:2491 [inlined]
> print\_type\_bicolor at .\show.jl:2490
> jfptr\_print\_type\_bicolor\_25682.clone\_1 at C:\Users\TGebbels\AppData\Local\Programs\Julia-1.9.3\lib\julia\sys.dll (unknown line)
> #show\_tuple\_as\_call#539 at .\show.jl:2472
> show\_tuple\_as\_call at .\show.jl:2441 [inlined]
> show\_spec\_linfo at .\stacktraces.jl:244
> print\_stackframe at .\errorshow.jl:730
> print\_stackframe at .\errorshow.jl:695
> #show\_full\_backtrace#921 at .\errorshow.jl:594
> show\_full\_backtrace at .\errorshow.jl:587 [inlined]
> show\_backtrace at .\errorshow.jl:791
> #showerror#898 at .\errorshow.jl:90
> showerror at .\errorshow.jl:86
> unknown function (ip: 000002202952ae26)
> #showerror#899 at .\errorshow.jl:96
> showerror at .\errorshow.jl:94
> unknown function (ip: 00000220295286a6)
> show\_exception\_stack at .\errorshow.jl:895
> display\_error at .\client.jl:111
> unknown function (ip: 000002202952807a)
> display\_error at .\client.jl:114
> jfptr\_display\_error\_32250.clone\_1 at C:\Users\TGebbels\AppData\Local\Programs\Julia-1.9.3\lib\julia\sys.dll (unknown line)
> jl\_apply at C:/workdir/src\julia.h:1880 [inlined]
> jl\_f\_\_call\_latest at C:/workdir/src\builtins.c:774
> #invokelatest#2 at .\essentials.jl:819 [inlined]
> invokelatest at .\essentials.jl:816 [inlined]
> \_start at .\client.jl:524
> jfptr\_\_start\_29544.clone\_1 at C:\Users\TGebbels\AppData\Local\Programs\Julia-1.9.3\lib\julia\sys.dll (unknown line)
> jl\_apply at C:/workdir/src\julia.h:1880 [inlined]
> true\_main at C:/workdir/src\jlapi.c:573
> jl\_repl\_entrypoint at C:/workdir/src\jlapi.c:717
> mainCRTStartup at C:/workdir/cli\loader\_exe.c:59
> BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
> RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [October 5, 2023, 12:31pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/7 "2023-10-05T12:31:17Z")
The ISO-8859-1 example from the docs works fine for me?
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 5, 2023, 1:30pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/9 "2023-10-05T13:30:47Z")
> [@nilshg](#):
>
> ISO-8859-1
Yes, this works for me, too! Thanks!
How would I know _a priori_ that my file is ISO-8859-1 encoded? I’ve looked it up and it seems to be a bit of an out of data scheme.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [October 5, 2023, 2:21pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/10 "2023-10-05T14:21:44Z")
Another possibility here is to coax Excel into saving the file in UTF-8, which must be possible somehow. That’s pretty much the encoding everyone uses today.
---
### Author:  [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [October 5, 2023, 2:35pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/11 "2023-10-05T14:35:02Z")
> [@StefanKarpinski](#):
>
> which must be possible somehow
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [October 5, 2023, 2:47pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/12 "2023-10-05T14:47:28Z")
> [@TimG](#):
>
> How would I know _a priori_ that my file is ISO-8859-1 encoded? I’ve looked it up and it seems to be a bit of an out of data scheme.
It’s probably [Windows-1252](https://en.wikipedia.org/wiki/Windows-1252), actually, which is _almost_ the same as ISO-8859-1.
It’s simple — Windows-1252 seems to be the only non-UTF8 extension of ASCII still in widespread use, at least in Western countries. If you see text that looks _mostly_ okay in UTF-8 (because the ASCII characters are fine), but non-ASCII characters are garbled [mojibake](https://en.wikipedia.org/wiki/Mojibake), then 99 times out of 100 it’s Windows 1252.
(Thanks to Microsoft for keeping this precious historical artifact alive.)
More generally, there are various heuristics for [charset detection](https://en.wikipedia.org/wiki/Charset_detection). These days, however, you mostly only need to check for UTF-8, Windows-1252, and UTF-16LE (which will look like complete garbage in UTF-8 because it’s not an ASCII superset).
---
### Author:  [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [October 5, 2023, 2:48pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/13 "2023-10-05T14:48:39Z")
> [@TimG](#):
>
> How would I know _a priori_ that my file is ISO-8859-1 encoded? I’ve looked it up and it seems to be a bit of an out of data scheme.
I don’t know the topic, but, making sense, I would say that a logic of the type [here](https://stackoverflow.com/questions/44098326/ef-bb-bf-at-the-beginning-of-json-files-created-in-visual-studio) mentioned is used.
I used the first 3 bytes of
```julia
Char.(buf[1:fnz])
```
to do the search
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 5, 2023, 3:23pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/14 "2023-10-05T15:23:55Z")
Interesting, @oheil …
I saved the file to a new name using the Unicode (utf8) option you highlighted.
I assumed the more conventional approach to CSV.read would then just work (as usual):
> ```
> MP_names = CSV.read("All MPs 23.07 - utf-8.csv", DataFrame)
> println(MP_names[130, :Constituency])
> println(lowercase(MP_names[130, :Constituency]))
>
> ```
But no!
> Ynys M�n
> ERROR: LoadError: Base.InvalidCharError{Char}(‘\xf4’)
> Stacktrace:…
Even though I’m reasonably sure I did save it using utf-8 encoding (I did it several times, and the option persists if I reopen the file), reading it using ISO-8859-1 is still necessary.
Is it me or is it Microsoft?
---
### Author:  [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [October 5, 2023, 4:00pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/15 "2023-10-05T16:00:20Z")
> [@TimG](#):
>
> Is it me or is it Microsoft?
I have no idea, I just wanted to show in an explicit way, how Excel provides the possibility. I didn’t test it with Julia to be sure. Perhaps it’s even just an option which is meant for something else because “Weboption” isn’t really what we would expect or what we want to do by just saving the file.
All I know is, character en-/decoding is quite complex and tricky and always source of surprises.
By the way, LibreOffice Calc asks explicitly for encoding when “Saving as…” and .CSV file format. Perhaps this a good option for you. Open Excel sheet with LO Calc, “Save as…”, xxx.csv, choose encoding, and, with some luck, default CSV.read may work (didn’t tried 🙂 ).
---
### Author:  [@Nathan\_Boyer](https://discourse.julialang.org/u/Nathan_Boyer)
#### Post date: [October 5, 2023, 4:52pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/16 "2023-10-05T16:52:24Z")
I think it’s mostly working for me with UTF-8?
```julia-repl
julia> file = CSV.File(open("Test.csv"), header=false)
1-element CSV.File:
CSV.Row: (Column1 = String15("Thérèse"), Column2 = String7("Si’nn"), Column3 = String7("Féin"), Column4 = String7("Ynys"), Column5 = String7("Môn"))
```
---
### Author:  [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [October 5, 2023, 4:54pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/17 "2023-10-05T16:54:57Z")
My excel doesn’t show this file type, “CSV UTF-8”. It’s just “CSV” here. But I would like to have this too.
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 6, 2023, 6:44am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/18 "2023-10-06T06:44:27Z")
Yes @Nathan_Boyer This works for me, too. Thanks.
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 6, 2023, 7:03am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/19 "2023-10-06T07:03:16Z")
I’m just a beginner but I’d like to restate the problem here:
CSV.read fails to read special characters in some circumstances but does not throw an error. The error is only generated when subsequent code tries to process the incorrect data (in this instance, lowercase())
CSV.read obviously tries to read the file encoded as ISO-8859-1 (or Windows-1252) and it almost succeeds, but it fails on some valid special characters.
This seems to me to be an issue with CSV.read. It should either succeed or throw an error.
Further, since Excel is globally ubiquitous and therefore likely to be a very common source of csv files but does not use UTF-8 by default, I think CSV.read should read these files without _any_ problem.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [October 10, 2023, 3:39pm UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/20 "2023-10-10T15:39:17Z")
This is by design. Even when they are encoded as UTF-8, CSV files can and often do include invalid string data. If `CSV.read` refused to load such files, there’d be no way to work with them short of editing them in some external tool, which would really not be great. Moreover, there’s no reason (in Julia at least) to refuse to work with such files: you can read and work with invalid string data so long a you don’t try to do something with it that isn’t well-defined for that data. In this case, you’ve asked to change the case of a byte that isn’t valid UTF-8, which doesn’t have a meaningful answer, so it throws an error. (Although, if we wanted to be _really_ permissive, we could just leave invalid data alone while changing the case of a string.) A potential design change would be to add an option to `CSV.read` to validate strings as UTF-8 and turn that on by default, only letting you read invalid data if you disable that option.
---
### Author:  [@TimG](https://discourse.julialang.org/u/TimG)
#### Post date: [October 11, 2023, 7:20am UTC](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604/21 "2023-10-11T07:20:51Z")
From my point of view, your suggested design change seems like an excellent idea.
The characters I was trying to read were invalid under UTF8 but not under all encodings. Any error message might mention this, too:
> Error - invalid _enc\_scheme\_used_ characters found. Consider specifying a different encoding using enc"_scheme\_name_". Alternatively, to force reading of invalid characters, use allow\_invalid\_chars = true
where _enc\_scheme\_used_ states the encoding scheme used for this attempted CSV.read operation.
[Next page](https://discourse.julialang.org/t/read-special-characters-using-csv-read/104604.md?page=2)