StringIndex idea (Julia 2.0)

[mnemnion]

@stevengj

I disagree with your premise that random codepoint indexing is “working with strings properly”. String algorithms that involve random access should be operating on codeunits (= bytes in UTF-8), not codepoints (Chars), and codeunits already provide O(1) access.

I’m not sure what you’re disagreeing with, given that I said this:

@mnemnion

I’d rather not see indexing into strings messed with. A String is fundamentally a byte array, and indexing it by bytes is therefore the correct default. Julia doesn’t guarantee that the encoding in a String is correct, for one thing, and low-level String code wants to work with the bytes and cope with the encoding directly, for performance reasons (I’m working on a package which does this, which colors my opinion here).

But this also means a lot of algorithms which “work on my machine”, because they’re tested against the ASCII range of UTF-8 only. So someone writes a naïve slice of a string for five characters, and someone uses it on Greek and gets an invalid string. That can’t really be considered working with strings properly, can it.

The lede you quoted was an intro to some thoughts on how to get good performance while working with codepoints as though they were randomly accessible. At no point did I suggest blowing strings up into UTF-32, and I agree with you that it’s a bad language default.