# SubString doesn't work with unicode

**URL:** <https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976>\
**Category:** New to Julia\
**Tags:** question, unicode\
**Created:** [June 8, 2020, 8:16am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976 "2020-06-08T08:16:50Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicolas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nicolas/32/26439_2.png) [@nicolas](https://discourse.julialang.org/u/nicolas)\
**Post date:** [June 8, 2020, 8:16am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/1 "2020-06-08T08:16:50Z")

</div>

It seems SubString works well with standard characters, but not for unicode characters. Below is an example in Julia 1.4.2:

```julia
julia> SubString("aaaa",1,2)
"aa"

julia> SubString("αααα",1,2)
ERROR: StringIndexError("αααα", 2)
Stacktrace:
 [1] string_index_err(::String, ::Int64) at .\strings\string.jl:12
 [2] SubString at .\strings\substring.jl:32 [inlined]
 [3] SubString(::String, ::Int64, ::Int64) at .\strings\substring.jl:38
 [4] top-level scope at REPL[79]:1

```

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [June 8, 2020, 8:55am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/2 "2020-06-08T08:55:14Z")

</div>

Julia indexes into strings by code unit, not by character. Take a look at the relevant [docs section](https://docs.julialang.org/en/v1/manual/strings/#Unicode-and-UTF-8-1) for how to correctly handle Unicode.

---

<div class="post-metadata">

**Author:** ![nicolas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nicolas/32/26439_2.png) [@nicolas](https://discourse.julialang.org/u/nicolas)\
**Post date:** [June 9, 2020, 8:10am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/3 "2020-06-09T08:10:42Z")

</div>

Thanks a lot. I didn’t think of that since variable names with unicode worked for me thus far. But seeing how delicate character encoding can be, it makes complete sense to have a sound basis within Julia to incorporate unicode into strings.

---

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [June 13, 2022, 4:13pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/4 "2022-06-13T16:13:09Z")

</div>

Hello,

I regularly stumble into this. There seem to be many `String` supporting packages, but I didn’t quickly find one that simply provided unicode-point indexing.

For now, a quick hack could be (no bounds checking)

```julia
function substring(s::AbstractString, i::Int, j::Int)
  chars = collect(c for c in s)
  return join(chars[i : j])
end

```

but this isn’t very efficient for repeated operation on the same string.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 13, 2022, 4:17pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/5 "2022-06-13T16:17:27Z")

</div>

> **[Subsetting strings in Julia using character indexing](https://bkamins.github.io/julialang/2020/08/29/charindex.html)**
>
> Introduction

---

<div class="post-metadata">

**Author:** ![StevenSiew](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevensiew/32/218393_2.png) [@StevenSiew](https://discourse.julialang.org/u/StevenSiew)\
**Post date:** [June 13, 2022, 11:16pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/6 "2022-06-13T23:16:15Z")

</div>

Is this in the Julia Forem?

---

<div class="post-metadata">

**Author:** ![davidavdav](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidavdav/32/1065_2.png) [@davidavdav](https://discourse.julialang.org/u/davidavdav)\
**Post date:** [June 17, 2022, 7:37am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/7 "2022-06-17T07:37:08Z")

</div>

Thanks, the macro looks impressive!

However I can’t imagine this is the kind of thing a novice Julia user is supposed to conjure up if she wants to substring based on character index rather than byte-in-utf8-encoding index.

Are there String implementations that simply work with 32bit Unicode characters in memory? It might not always be that memory efficient, but my guess is that there are lots of applications where memory is less important and ease of substring manipulation is more important.

Long ago I thought of a string type with an additional index `collect(eachindex(s))`, but the additional memory needed for that might as well be spent directly on the wider character representations.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 17, 2022, 8:05am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/8 "2022-06-17T08:05:11Z")

</div>

This blog post is meant to exactly provide the opposite. I do not think any Julia beginner is able to write such code. I made it public so that any Julia beginner can use it (even without understanding the details of it).

For UTF32 encoded strings you can use LegacyStrings.jl:

```julia
julia> using LegacyStrings

julia> x = convert(UTF32String, "aą∀∃ę")
"aą∀∃ę"

julia> x[1]
'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)

julia> x[2]
'ą': Unicode U+0105 (category Ll: Letter, lowercase)

julia> x[3]
'∀': Unicode U+2200 (category Sm: Symbol, math)

julia> x[4]
'∃': Unicode U+2203 (category Sm: Symbol, math)

julia> x[5]
'ę': Unicode U+0119 (category Ll: Letter, lowercase)

```

@StefanKarpinski - do you know what is the current level of maintenance guarantees for LegacyStrings.jl?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 17, 2022, 8:18am UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/9 "2022-06-17T08:18:21Z")

</div>

@davidavdav - see also [Consider adding kwargs to chop · Issue #37397 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/37397) as this suggestion was made to the Julia Core team to improve the situation you describe. Maybe you would have some thoughts to share there. Thank you!

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [June 17, 2022, 7:43pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/10 "2022-06-17T19:43:18Z")

</div>

I hope I am not mistaken, but there is [this PR](https://github.com/JuliaLang/julia/pull/44266) by @stevengj that was supposed to address a related question via:

```julia
using Unicode
graphemes("αααα", 1:2)

```

However, this doesn’t seem to work in Julia 1.7.3.

From the linked PR, how can we know where/when it is ready to be used? Thank you.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 17, 2022, 9:02pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/11 "2022-06-17T21:02:28Z")

</div>

> [@rafael.guerra](#):
>
> From the linked PR, how can we know where/when it is ready to be used? Thank you.

Look in the `NEWS.md` patch in that PR—it’s slated for Julia 1.9.

See also this discussion and my answers therein: [Substring function?](https://discourse.julialang.org/t/substring-function/76675)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [June 17, 2022, 9:20pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/12 "2022-06-17T21:20:19Z")

</div>

> [@stevengj](#):
>
> Look in the `NEWS.md` patch in that PR—it’s slated for Julia 1.9.

Thanks for the advice. I went for a look and it wasn’t next door: I had to go to the “Files changed” tab and then search for “1.9” to finally see on line 177:

 ![Graphemes_indexing_PR_MD.news](https://global.discourse-cdn.com/julialang/original/3X/3/0/30905f86c326783a20713f2e5e51b76a89c7f367.png)

Is this the right way?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 17, 2022, 10:02pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/13 "2022-06-17T22:02:05Z")

</div>

> [@rafael.guerra](#):
>
> Thanks for the advice. I went for a look and it wasn’t next door: I had to go to the “Files changed” tab and then search for “1.9” to finally see on line 177:

That’s fine, since many PRs will have a “compatibility” section like this in the docstring.

You can also go to “Files changed” and click on “View file” in the “…” for `NEWS.md`:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/b/e/bed4c012edc20ad2ab65909bcbb1c7ed575f5936.png)  
Scrolling to the top, you’ll see it is the NEWS for Julia 1.9.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 17, 2022, 10:07pm UTC](https://discourse.julialang.org/t/substring-doesnt-work-with-unicode/40976/14 "2022-06-17T22:07:33Z")

</div>

> [@davidavdav](#):
>
> However I can’t imagine this is the kind of thing a novice Julia user is supposed to conjure up if she wants to substring based on character index rather than byte-in-utf8-encoding index.

As I commented [in the other thread](https://discourse.julialang.org/t/substring-function/76675/10), anyone seeking to index strings this way is probably making a mistake, because [Unicode is too complicated for character indexing to be useful](https://discourse.julialang.org/t/substring-function/76675/34).