# Julia 0.6 Unicode Parsing Problem

**URL:** <https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649>\
**Category:** Data\
**Tags:** strings\
**Created:** [May 11, 2017, 2:31pm UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649 "2017-05-11T14:31:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Daneel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daneel/32/175_2.png) [@Daneel](https://discourse.julialang.org/u/Daneel)\
**Post date:** [May 11, 2017, 2:31pm UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649/1 "2017-05-11T14:31:39Z")

</div>

Julia 0.6.0-rc1.0 can’t properly load a Unicode character in a settings file I have, specifically “³” (U+00b3). Julia doesn’t have a problem if it’s a UTF-8 Unicode file but the settings files we have are generated in ISO-8859 format.

Julia 0.6 does load the line but the character that should be a ³ shows up as a symbol with a question mark in it (see below). Using `readline` to load the line returns a `30-byte String of invalid UTF-8 data`.

Specific Problem Line

> CO2, 10 , 40 , mmol/m³ , 0 , E

Error Message:

> CO2, 10 , 40 , mmol/m� , 0 , E  
> ERROR: LoadError: at row 1, column 3 : UnicodeError: invalid character index)  
> Stacktrace:  
> [1] dlm\_parse(::String, ::Char, ::Char, ::Char, ::Char, ::Bool, ::Bool, ::Bool, ::Int64, ::Bool, ::Base.DataFmt.DLMOffsets) at ./datafmt.jl:610  
> [2] readdlm\_string(::String, ::Char, ::Type, ::Char, ::Bool, ::Dict{Symbol,Union{Char, Integer, Tuple{Integer,Integer}}}) at ./datafmt.jl:343  
> [3] readdlm\_auto(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Char, ::Type{T} where T, ::Char, ::Bool) at ./datafmt.jl:119  
> [4] #readdlm#7 at ./datafmt.jl:81 [inlined]  
> [5] readdlm(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Char, ::Char) at ./datafmt.jl:81  
> [6] #readdlm#6 at ./datafmt.jl:73 [inlined]  
> [7] readdlm(::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Char) at ./datafmt.jl:73  
> [8] #readcsv#14 at ./datafmt.jl:618 [inlined]  
> [9] readcsv(::Base.AbstractIOBuffer{Array{UInt8,1}}) at ./datafmt.jl:618  
> [10] slt\_configload(::Array{String,1}) at /home/user/.julia/v0.6/MyModule/src/slt\_configload.jl:67  
> [11] slt\_config(::Array{String,1}) at /home/user/.julia/v0.6/MyModule/src/slt\_config.jl:28  
> [12] #slt\_load#21(::Bool, ::Bool, ::Float64, ::Function, ::String, ::DateTime, ::DateTime) at /home/user/.julia/v0.6/MyModule/src/slt\_load.jl:115  
> [13] (::MyModule.#kw##slt\_load)(::Array{Any,1}, ::MyModule.#slt\_load, ::String, ::DateTime, ::DateTime) at ./:0  
> [14] include\_from\_node1(::String) at ./loading.jl:552  
> [15] include(::String) at ./sysimg.jl:14  
> [16] process\_options(::Base.JLOptions) at ./client.jl:305  
> [17] \_start() at ./client.jl:371  
> while loading /home/user/.julia/v0.6/MyModule/test/runtests.jl, in expression starting on line 11

Julia 0.5 didn’t have this problem.

Is there a setting I’ve missed that will allow Julia to parse it properly?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [May 11, 2017, 2:43pm UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649/2 "2017-05-11T14:43:02Z")

</div>

I can’t reproduce your problem.

```julia
julia> readcsv(IOBuffer("CO2, 10 , 40 , mmol/m³ , 0 , E"))
1×6 Array{Any,2}:
 "CO2" 10 40 " mmol/m³ " 0 " E"

```

works fine for me in Julia 0.6.

Are you sure that your file is UTF-8 encoded? You didn’t accidentally save it in some other encoding? Could it be a terminal issue on Windows?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [May 11, 2017, 2:48pm UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649/3 "2017-05-11T14:48:20Z")

</div>

Oh, I didn’t notice your comment _but the settings files we have are generated in ISO-8859 format_.

If your data is not valid UTF-8, it’s not to surprising that we throw a Unicode error when trying to parse it as a string. I’m surprised that Julia 0.5 didn’t complain, though.

(You can probably use [https://github.com/nalimilan/StringEncodings.jl](https://github.com/nalimilan/StringEncodings.jl) convert ISO-8859 streams into UTF-8 streams.)

---

<div class="post-metadata">

**Author:** ![Daneel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daneel/32/175_2.png) [@Daneel](https://discourse.julialang.org/u/Daneel)\
**Post date:** [May 12, 2017, 7:21am UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649/4 "2017-05-12T07:21:36Z")

</div>

It worked in Julia 0.5 so I figured it would work in Julia 0.6. I tried looking through the Julia repo for relevant changes but my Git-fu is lacking.

Is there a good way of programmatically determining the file encoding (preferably without adding new packages)?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [May 12, 2017, 7:55am UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649/5 "2017-05-12T07:55:24Z")

</div>

> [@Daneel](#):
>
> It worked in Julia 0.5 so I figured it would work in Julia 0.6. I tried looking through the Julia repo for relevant changes but my Git-fu is lacking.

Some things have changed in Julia 0.6 regarding string parsing, which could have made this more visible. See for example [this closed issue](https://github.com/JuliaLang/julia/issues/21180). Anyway when the input is invalid it is not guaranteed that you will get an error AFAIK (we could provide a way to enable input validation, but that’s a different issue).

> [@Daneel](#):
>
> Is there a good way of programmatically determining the file encoding (preferably without adding new packages)?

No. Even with external libraries, this is at best a fragile operation. See the `detect` function of [JuliaStrings/ICU.jl](https://github.com/JuliaString/ICU.jl) (_not_ the registered ICU.jl).