# Indexing strings by Unicode code point instead of code unit?

**URL:** <https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248>\
**Category:** General Usage\
**Tags:** strings\
**Created:** [February 14, 2021, 9:01am UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248 "2021-02-14T09:01:08Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![johann-petrak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann-petrak/32/15554_2.png) [@johann-petrak](https://discourse.julialang.org/u/johann-petrak)\
**Post date:** [February 14, 2021, 9:01am UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/1 "2021-02-14T09:01:08Z")

</div>

I am very new to Julia and would like to find out if I want to use it for implementing a library.  
For this, I need to deal with Unicode strings and especially offset ranges of substrings a lot. In this context, I would need/want to represent ranges of Unicode characters, i.e. ranges of Unicode code points. However, as I understand, the default indexing of strings in Julia accesses UTF-8 code units, not code points.

This is different from Python, where indexing actually accesses the code points.

My question is: is there some kind of library or alternate string implementation that already deals with Strings properly as sequences of Unicode code points, irrespective of the encoding used? I do not care what representation is used internally, I do care about a string/text being a sequence of code points (even if that may make some string operations a little bit slower or, more like, use more memory for mapping arrays or the like).

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [February 14, 2021, 9:15am UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/2 "2021-02-14T09:15:32Z")

</div>

UTF-32 strings from [GitHub - JuliaStrings/LegacyStrings.jl: Legacy Unicode string types](https://github.com/JuliaStrings/LegacyStrings.jl) should give you what you want. (Python stores strings in UTF32)

---

<div class="post-metadata">

**Author:** ![johann-petrak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann-petrak/32/15554_2.png) [@johann-petrak](https://discourse.julialang.org/u/johann-petrak)\
**Post date:** [February 14, 2021, 10:23am UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/3 "2021-02-14T10:23:04Z")

</div>

Interesting – do you happen to know if it is possible to not just convert between the UTF32 representation from this library and the default Julia strings back and forth, but more importantly also convert the offsets?

Actually, even more importantly: does this library actually literally convert the string representation? Because I would not actually want that – All I need is s “view” of a Julia string (no matter what the representation) which allows me to index Unicode codepoints.

My issue with the julia strings is not the representation - I do not care about it really, but the way how one indexes/retrieves the elemets of the string: the Julia design decision apparently was to use indexing to retrieve the elements of representation, not the actual elements a string has: characters.  
But it should be possible to at least ADD a way to do this by providing a “view”.

So codepoint(juliastring, 3) or similar should give me the actual third codepoint, which could be the 5th code unit in Julia. That should be achievable by simply creating an index map from code point indices to code units that are the beginning of code points (and another one for the other way).

I guess I would be prepared to actually implement this though it looks like something so basic that it feels like it should exist 🙂  
(on the other hand it does not exist in Java either where strings are represented using UTF-16 code units with surrogate characters).

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [February 14, 2021, 11:20am UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/4 "2021-02-14T11:20:18Z")

</div>

You can create this mapping with `points = collect(eachindex(str))`. This has a big cost of course so in most cases you’d avoid that and use `eachindex` directly…

---

<div class="post-metadata">

**Author:** ![johann-petrak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann-petrak/32/15554_2.png) [@johann-petrak](https://discourse.julialang.org/u/johann-petrak)\
**Post date:** [February 14, 2021, 1:44pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/5 "2021-02-14T13:44:29Z")

</div>

Thanks, this actually looks exactly like what I need! You say it has a big cost, can you be more specific? My assumption would be that the cost (time and memory) is linear in the number of code units of the string, is that true? Is there anything else to be worried about (e.g. handling slow exceptions internally or other things that could put a high constant factor on that linear cost?)  
If the cost is basically what going through all the code units of the string in some other way would be, that is fine, the same is needed to map offsets e.g. between Java UTF16 code units and Python unicode code points.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 14, 2021, 2:10pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/6 "2021-02-14T14:10:35Z")

</div>

> [@johann-petrak](#):
>
> I am very new to Julia and would like to find out if I want to use it for implementing a library.  
> For this, I need to deal with Unicode strings and especially offset ranges of substrings a lot. In this context, I would need/want to represent ranges of Unicode characters, i.e. ranges of Unicode code points. However, as I understand, the default indexing of strings in Julia accesses UTF-8 code units, not code points.

Where are you getting the ranges of Unicode characters from? In most kinds of string-processing operations (e.g. search/replace), you find ranges of characters by _first_ iterating over the string, in which case you also get the indices (regardless of the indexing scheme).

It’s pretty rare to have an application in which someone says “give me the 527th character of this string” where the number 527 just falls down out of the sky. (This is a key reason why a variable-width encoding like UTF-8 can be so popular.)

> [@johann-petrak](#):
>
> is there some kind of library or alternate string implementation that already deals with Strings properly as sequences of Unicode code points, irrespective of the encoding used?

When you iterate over a Julia string (for any `AbstractString`), you indeed get a sequence of Unicode codepoints, regardless of the encoding. If you loop over `pairs(somestring)` then you get both the codepoints and the corresponding indices. There are also `nextind` and `prevind` for more low-level iteration.

Note that you _can_ also get the index of the n-th character in a string `s` via `nextind(s, 0, n)`, although this is an O(n) operation in a variable-width encoding like UTF-8.

In the unlikely event that you really _need_ a fixed-width encoding, you can use `UTF32String` from [LegacyStrings.jl](https://github.com/JuliaStrings/LegacyStrings.jl).

---

<div class="post-metadata">

**Author:** ![johann-petrak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann-petrak/32/15554_2.png) [@johann-petrak](https://discourse.julialang.org/u/johann-petrak)\
**Post date:** [February 14, 2021, 2:40pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/7 "2021-02-14T14:40:55Z")

</div>

> [@stevengj](#):
>
> Where are you getting the ranges of Unicode characters from? In most kinds of string-processing operations (e.g. search/replace), you find ranges of characters by _first_ iterating over the string, in which case you also get the indices (regardless of the indexing scheme).
> 
> It’s pretty rare to have an application in which someone says “give me the 527th character of this string” where the number 527 just falls down out of the sky. (This is a key reason why a variable-width encoding like UTF-8 can be so popular.)

For the problem I want to solve it is actually the other way round: I do have a Unicode string (in some encoding originally, but ultimately it will be a “normal” Julia string) and a large list of offset ranges, where the offsets are 0-based Unicode code point indices.  
Now in order to access the string subsequences that correspond to those offset ranges, I need to first convert these offset ranges into the proper offset ranges for Julia strings, and I think the suggestion of creating a map using `collect(eachindex(str))` is probably the solution to this.

> [@stevengj](#):
>
> When you iterate over a Julia string (for any `AbstractString`), you indeed get a sequence of Unicode codepoints, regardless of the encoding. If you loop over `pairs(somestring)` then you get both the codepoints and the corresponding indices. There are also `nextind` and `prevind` for more low-level iteration.
> 
> Note that you can also get the index of the n-th character in a string `s` via `nextind(s, 0, n)`, but this is an O(n) operation in a variable-width encoding like UTF-8.

Thanks for pointing this out! I guess Julia’s compromise of treating strings somewhat like sparse arrays (I assume), is a good compromise between proper handling of Unicode and still being able to efficiently access code points.

Thank you @sijo and @stevengj I think my problem is solved and I have a rough idea of how this could work! Looking forward to learn Julia properly now 😃

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [February 14, 2021, 2:48pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/8 "2021-02-14T14:48:00Z")

</div>

> [@johann-petrak](#):
>
> You say it has a big cost, can you be more specific? My assumption would be that the cost (time and memory) is linear in the number of code units of the string, is that true? Is there anything else to be worried about (e.g. handling slow exceptions internally or other things that could put a high constant factor on that linear cost?)

Yes I just meant the cost of allocating the memory and processing the whole string to populate the mapping. It’s a large cost relative to looping directly on the string (so you process it only once and without allocation).

> [@johann-petrak](#):
>
> I do have a Unicode string (in some encoding originally, but ultimately it will be a “normal” Julia string) and a large list of offset ranges, where the offsets are 0-based Unicode code point indices.

Assuming performance is a concern in your application, why not keep the original encoding? If it’s not UTF-8 you can ~~probably~~ maybe¹ use direct indexing in the original string without a mapping…

¹ I forgot about UTF-16 surrogates, and as mentioned by @stevengj UTF-16 is a common case of non-UTF-8 text. (I don’t know if there’s another likely source of text with non-constant codepoint size?)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 14, 2021, 2:49pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/9 "2021-02-14T14:49:16Z")

</div>

> [@johann-petrak](#):
>
> a large list of offset ranges, where the offsets are 0-based Unicode code point indices.

Out of curiosity, where are these offset ranges coming from in your application?

(Beware that if they are coming from another program that uses the UTF-16 encoding, e.g. in JavaScript or Windows C++, they might be indices of UTF-16 code units, which are _almost_ codepoint indices but differ for relatively rare cases—unfortunately this just makes bugs harder to find but no less severe. This came up for me as a [Jupyter bug](https://github.com/jupyter/jupyter_client/issues/259) once, for example.)

---

<div class="post-metadata">

**Author:** ![johann-petrak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann-petrak/32/15554_2.png) [@johann-petrak](https://discourse.julialang.org/u/johann-petrak)\
**Post date:** [February 14, 2021, 2:54pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/10 "2021-02-14T14:54:08Z")

</div>

> [@stevengj](#):
>
> Out of curiosity, where are these offset ranges coming from in your application?

These are representations of texts with stand-off annotations for natural language processing and can ultimately come from many different sources. I have already worked for a long time with these in both Java (where the offsets refer to UTF16 code units with surrogate pairs representing what cannot be represented in 16 bits) and Python (where string indices directly access Unicode code points).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 14, 2021, 3:54pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/11 "2021-02-14T15:54:03Z")

</div>

> [@johann-petrak](#):
>
> These are representations of texts with stand-off annotations for natural language processing and can ultimately

If you have a text with a sequence of ranges that are ordered monotonically by codepoint, then you can convert them to UTF-8 (Julia `String`) indices (or whatever data structure is desired) by a single pass over the text and indices, without allocating an explicit index map. (In any case, I would recommend doing the conversion once when they are read in, then for subsequent processing use native indices.)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 14, 2021, 4:01pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/12 "2021-02-14T16:01:24Z")

</div>

> [@johann-petrak](#):
>
> the Julia design decision apparently was to use indexing to retrieve the elements of representation, not the actual elements a string has: characters.

This is **not correct**. Julia string indices retrieve characters, not code units, it’s just that the indices are not consecutive:

```julia
julia> s = "αβγ"
"αβγ"

julia> collect(eachindex(s))
3-element Array{Int64,1}:
 1
 3
 5

julia> s[1]
'α': Unicode U+03B1 (category Ll: Letter, lowercase)

julia> s[3]
'β': Unicode U+03B2 (category Ll: Letter, lowercase)

julia> s[5]
'γ': Unicode U+03B3 (category Ll: Letter, lowercase)

```

(Note that the return value of `s[i]` is a character represented by `Char`, a 4-byte object corresponding to a Unicode codepoint.) String iteration is also over characters:

```julia
julia> for c in s
           display(c) # pretty-print c
       end
'α': Unicode U+03B1 (category Ll: Letter, lowercase)
'β': Unicode U+03B2 (category Ll: Letter, lowercase)
'γ': Unicode U+03B3 (category Ll: Letter, lowercase)

julia> for (i,c) in pairs(s)
           @show i, c, s[i]
       end
(i, c, s[i]) = (1, 'α', 'α')
(i, c, s[i]) = (3, 'β', 'β')
(i, c, s[i]) = (5, 'γ', 'γ')

```

In contrast, the code units (bytes for UTF-8) are retrieved by the `codeunit` function:

```julia
julia> codeunit(s, 3)
0xce

julia> codeunits(s)
6-element Base.CodeUnits{UInt8,String}:
 0xce
 0xb1
 0xce
 0xb2
 0xce
 0xb3

```

Substrings work similarly: you give them code-unit indices, but they still give you the whole string of Unicode characters:

```julia
julia> s[3:5] # a copy
"βγ"

julia> SubString(s, 3:5) # a view
"βγ"

```

So, the point is, once you convert your character offsets to Julia `String` indices, you are still working with characters.

---

<div class="post-metadata">

**Author:** ![johann-petrak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann-petrak/32/15554_2.png) [@johann-petrak](https://discourse.julialang.org/u/johann-petrak)\
**Post date:** [February 14, 2021, 5:29pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/13 "2021-02-14T17:29:23Z")

</div>

@stevengj yes, sorry, I noticed this already from earlier explanations but did not correct myself, thanks for the clarification. I have learned that this is really more like a partial function or sparse array of actual proper Unicode characters.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 18, 2022, 2:13pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/14 "2022-02-18T14:13:54Z")

</div>

> [@stevengj](#):
>
> might be indices of UTF-16 code units, which are _almost_ codepoint indices but differ for relatively rare cases

It’s increasingly not rare, as I think you may know, e.g. all emojis use surrogates. Such as 🙂 not to be confused with emoticons like `:)`, which if written that way _and displayed that way_ is only ASCII (however many editors, like on Discourse change emoticons to emojis. So in practice e.g. otherwise English-only text, needs full variable length UTF-16 (as opposed to UCS-2). Such as SMS/text messaging. According to spec it uses UCS-2, but it’s known to use UTF-16 for that reason. Also non-BMP, i.e surrogates in UTF-16 used for “some modern scripts like [Osage](https://en.wikipedia.org/wiki/Osage_alphabet), [Warang Citi](https://en.wikipedia.org/wiki/Warang_Citi), [Adlam](https://en.wikipedia.org/wiki/Adlam_alphabet), [Wancho](https://en.wikipedia.org/wiki/Wancho_language) and [Toto](https://en.wikipedia.org/wiki/Toto_language).”

---

<div class="post-metadata">

**Author:** ![zot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zot/32/5817_2.png) [@zot](https://discourse.julialang.org/u/zot)\
**Post date:** [January 12, 2024, 5:54pm UTC](https://discourse.julialang.org/t/indexing-strings-by-unicode-code-point-instead-of-code-unit/55248/15 "2024-01-12T17:54:44Z")

</div>

`Vector{Char}(string)` sounds like it would work for this problem. Use that to convert a string into an array of Chars and mash them together with `String(char_array)` or `char1 * char2`.