# Changes to the representation of Char

**URL:** <https://discourse.julialang.org/t/changes-to-the-representation-of-char/7291>\
**Category:** Internals & Design\
**Created:** [November 23, 2017, 12:25pm UTC](https://discourse.julialang.org/t/changes-to-the-representation-of-char/7291 "2017-11-23T12:25:11Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 11, 2017, 8:33pm UTC](https://discourse.julialang.org/t/changes-to-the-representation-of-char/7291/5 "2017-12-11T20:33:22Z")

</div>

Here’s the new replacement PR for the one linked above:

[https://github.com/JuliaLang/julia/pull/24999](https://github.com/JuliaLang/julia/pull/24999)

This turned into a bigger overhaul than expected – there were a lot of conceptual inconsistencies that had crept into the string code over years of various people with slightly different mental models working on it. Changing the character representation itself wasn’t such a big deal, but rewriting all the low-level functions and having everything work required some rethinking of what the fundamental API that a string type has to provide should be. The core methods that a string type must provide are now:

- `ncodeunits(s::AbstractString)` – the number of code units
- `codeunit(s::AbstractString)` – the code unit type of a string
- `codeunit(s::AbstractString, i::Integer)` – extracting a single code unit
- `isvalid(s::AbstractString, i::Integer)` – is an index the start of a character
- `next(s::AbstractString, i::Integer)` – string iteration

Everything else has pretty efficient generic definitions in terms of these. (Except for string reversal, which we don’t currently have an efficient way to express generically, but that can be added later.)

I changed the representation of `Char` and it now holds bytes of data from an underlying `String` object and iterates `String` whether they contain valid UTF-8 data or not, by breaking the data into chunks of 1-4 bytes in a 32-bit `Char` value, which are either:

- well-formed UTF-8 characters
- bytes corresponding to a replacement character sequence in Unicode 10

For the latter case, the `Char` value captures the underlying bytes instead of replacing them, which allows code to operate on malformed values, choosing to ignore them, replace them, or passing them through as is. You only get an error for malformed UTF-8 in a `String` if you try to convert a malformed `Char` value to an integer code point. At that point there’s no other choice since there is no correct value to return since the character data is malformed.

The phrase “well-formed UTF-8” is more lenient than what Unicode defines as “valid UTF-8”. It includes anything with the basic structure of UTF-8: ASCII values or a lead byte (with 2-4 leading one bits) followed by enough continuation bytes (with 1 leading one bit). Any such sequence can be decoded to a code point value, and that’s what is returned when you do `Int(c)`. This allows decoding non-standard UTF-8 schemes like Modified UTF-8 and WTF-8, which commonly occur in practice. So if you want to check if a character is valid UTF-8, you could check that is a well-formed, which guarantees that converting it to an integer will not fail, and then check that converting it back to a character gives you the same character value. Of course, there’s an `isvalid(c)` predicate that checks this more efficiently, but you get the point.

The base string functions no longer try to combine CESU-8 and instead will yield `Char` values for each surrogate pair. You can detect this and combine them if easily, but combining them automatically would violate the “give the programmer what’s actually there” maxim that the new approach takes. Higher-level functionality can easily be provided for normalizing and “fixing” strange/broken UTF-8 data.

---

_[View the full topic](https://discourse.julialang.org/t/changes-to-the-representation-of-char/7291)._