UTF8 space/difficulty tradeoffs

[ScottPJones]

@alkorang

For example, 한 is 0xc7d0 in EUC-KR , but U+d55c in Unicode and will be encoded as 0xed9f9c in UTF-8 .

This is something that has always concerned me about the “UTF-8 is always best” mentality, text from the languages of the vast majority of the world’s population generally takes up 50% to 200% more space encoded with UTF-8 compared to their older national character sets (compared to UTF-16, which is the same size or only 100% larger, not 200%).

[Tamas_Papp]

@ScottPJones

concerned me about the “UTF-8 is always best” mentality, text from the languages of the vast majority of the world’s population generally takes up 50% to 200% more space encoded with UTF-8

This is correct, but another perspective on this issue is that space is usually cheap compared to programmer time and the cost of bugs, so economizing on the latter by choosing a single representation is reasonable in many circumstances.

[ScottPJones]

@Tamas_Papp

but another perspective on this issue is that space is usually cheap compared to programmer time and the cost of bugs

Actually, that’s one big reason I think that what I’m working on would be much better for the Julia community.
There’s a constant stream of bugs related to indexing into UTF-8 code units.