# Syntax: Escape hatch for unicode haters

**URL:** <https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363>\
**Category:** Internals & Design\
**Tags:** syntax, unicode\
**Created:** [January 4, 2024, 11:55pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363 "2024-01-04T23:55:47Z")\
**Posts on this page:** 9\
**Page:** 7

<div class="post-metadata">

**Author:** ![danielwe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielwe/32/35657_2.png) [@danielwe](https://discourse.julialang.org/u/danielwe)\
**Post date:** [January 15, 2024, 7:53pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/129 "2024-01-15T19:53:42Z")

</div>

> [@foobar\_lv2](#):
>
> But for e.g. greek people who want to typeset text in their native language, they can of course also write unicode `π` in their text!

Note that this has only been supported by default since 2018. Before that, you had to add `\usepackage[utf8]{inputenc}` to your preamble, see [input encodings - What is the function \usepackage[utf8]{inputenc} used for and why we should add this in latex document? - TeX - LaTeX Stack Exchange](https://tex.stackexchange.com/a/641193). Either way, latex’s font selection scheme limits the possible input glyphs quite severely; for example, it won’t work with combining accents. You need xelatex or lualatex for full unicode support. This is just to say that the flexibility to choose between escape sequences and unicode glyphs isn’t inherent in (la)tex’s user experience design, but more of an afterthought that has evolved over time. (In case that matters for your comparison/argument.)

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [January 16, 2024, 1:22am UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/130 "2024-01-16T01:22:31Z")

</div>

> [@foobar\_lv2](#):
>
> they could clash in code that contains both `\alpha<TAB>` and alpha…  
> These command-line tools would need to be reversible / idempotent, i.e. `julia-format --ihateunicode` would be do nothing on source files that contain no non-ascii-encoded identifiers or operators, and `julia-format --iloveunicode` would do nothing on sourcefiles that contain none of the new escape sequences / trigraphs; and ` julia-format --ihateunicode | julia-format --iloveunicode` would be equivalent to `julia-format --iloveunicode`, and `julia-format --iloveunicode | julia-format --ihateunicode` would be equivalent to `julia-format --ihateunicode`.

This only affirms to me that this is a usability nightmare. Mixing the ASCII-only way of writing and the normal way should be avoided at all costs; this may be more forgiven in a typesetting format where everything will be rendered for readability, but not in source code that may be fed into macros, which would all need to recognize 2 different substrings or subexpressions\* as the same name. \*(I’m not actually certain if you intend the parser to expand the ASCII sequences so `Expr` contain the Unicode characters, I’d hope so but printing and copy-and-pasting would become issues for these writers. In either case, string macros all have a new difficulty.) Modifying or copying source files is risky when you scale up, especially when the file extension and source header lines provide no easy indication which way the text was written. A dedicated string macro is a lot smoother than these tools.

I should clarify that the string macro I had in mind would process a string containing the entire source file’s text, not each variable. A macro on each variable would easily allow mixing the 2 ways of writing e.g. `Unicode"\mu" === µ`, which again hurts readability and metaprogramming. Instead, a `Unicodify""" #= entire source file =# """` would be expanded before any macro calls inside the text, and while it wouldn’t need some command-line setting to be rendered or executed, `@macroexpand1` or something similar could be used to roughly expand that outer macro call for anyone who wants to read the Unicode directly.

A minor(?) concern for appropriating the `\u` and `\U` escape sequences in strings: those are allowed to vary the number of hexadecimal digits e.g. `"\u00b5m" === "\ub5m" === "µm"`. This poses an issue when writing trailing hexadecimal characters, e.g. `"µ1"` needs to be written with the full number of digits `"\u00b51"`, not `"\ub51"`. As long as people take care to write the full 4 or 6 digits in these cases, I don’t think it’ll be a problem for the `Unicodify` processing. Still, being able to write the same name with different sequences, even if avoiding direct Unicode, poses a metaprogramming problem again.

> [@foobar\_lv2](#):
>
> Can we make it easier for these groups to interact, to satisfy both their preferences?

Even given such a string macro to discourage mixing the 2 ways of writing within a source file, this is not a good goal. The fact remains that you are imposing an unprecedented standard on _everyone_, because the ASCII-sequence writers still cannot avoid Unicode files written by others, and vice versa, especially if they share a code base. Communication will be incredibly disharmonious, e.g. editing or copying from each other’s source files. Ironically in an attempt to address the evidently scarce complaint of writing with too many characters, this escape hatch idea introduces difficult and perhaps intractable problems by allowing people to inscribe a name in more than one way. Taking an earlier complaint as an example, this wouldn’t reduce the ways we can write for-loops, but add `for x \\in array` as fully equivalent to `for x ∈ array` in addition to the alternate versions with `in` and `=`. There’s merit to sticking to fundamental decisions, even if a change could technically not be breaking, because some accommodations add unnecessary complexity for everyone.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [January 16, 2024, 10:07am UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/131 "2024-01-16T10:07:30Z")

</div>

> [@Benny](#):
>
> that may be fed into macros

That’s why I am so insistent on this living in the tokenizer, and that ASTs must be equivalent between both ascii-only and unicode spellings, same as current unicode synonyms. Macros only see ASTs, so this is a complete non-issue.

Conversion between `uvar"\mu"` and `\mu<TAB>` is not code-formatting. It cannot be reasonably done automatically without human oversight, because macros can see it, and that could change semantics of the program.

> [@Benny](#):
>
> I should clarify that the string macro I had in mind would process a string containing the entire source file’s text, not each variable.

This breaks IDEs and also causes escaping trouble with code using backslash as division operator.

Something that could be done today, in a package, is `includeWithUnicodeEscapes("somefile.jl")`, where the modified include would replace unicode-escapes before invoking the standard runtime parser. But this again breaks existing IDEs and syntax highlighting, and would introduce an incompatible dialect of julia.

> [@Benny](#):
>
> Still, being able to write the same name with different sequences, even if avoiding direct Unicode, poses a metaprogramming problem again.

See above, no meta-programming problems. My proposal would use the exact same mechanism that avoids meta-programming problems with the difference between `'μ': Unicode U+03BC` and `'µ': Unicode U+00B5`: The tokenizer normalizes both to `'μ': Unicode U+03BC` before constructing the AST.

> [@Benny](#):
>
> Taking an earlier complaint as an example, this wouldn’t reduce the ways we can write for-loops, but add `for x \\in array` as fully equivalent to `for x ∈ array` in addition to the alternate versions with `in` and `=`.

The ship “only one way to write for-loops” has sailed long ago. I don’t see big issue with having one more way that the parser technically accepts, especially if this would not be expected to see widespread use. (the `for x \\in array` is still essential in my proposal: Otherwise round-tripping code-formatting from `for x ∈ array` could not end up at the same spelling! And round-tripping must end up at bitwise identical files, otherwise this messes up diffs)

> [@Benny](#):
>
> The fact remains that you are imposing an unprecedented standard on _everyone_,

I don’t see it that way. People who use “weird” (non US keyboard) unicode in their code impose their standard on everyone who wants to interact with them. Sometimes this makes sense: E.g. Chinese coders who mainly write for a Chinese audience.

“unprecedented standard”? Is U+00B5 vs U+03BC not a precedent in julia? Is `in` vs `∈` in for-loop headers not a precedent?

`in` vs `∈` outside of for-loop headers is _not_ a precedent: `:(a ∈ b)` and `:(a in b)` are different ASTs that only happen to have the same semantics in the default namespace, due to some `const ∈= in` in Base. This causes all the hypothetical macro havoc that you are so worried about. Outside of loop-headers where the tokenizer normalizes, the choice between `∈` and `in` is not mere code-formatting, because these are not synonyms.

And this “unprecedented standard” would be imposed on people only in the same way that reading and writing Chinese characters is imposed on them today.

(you would have a valid point if you complained that this annoying more complex standard gets imposed on IDE authors!)

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [January 16, 2024, 11:45am UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/132 "2024-01-16T11:45:28Z")

</div>

> [@foobar\_lv2](#):
>
> This would be paired with command-line tools that replace trigraph-style backslash representations with unicode representations, and vice versa.

Anybody can write a simple search-and-replace script in a few minutes that round-trips between unicode and some escaped representation (or even a representation that still parses as valid Julia). There is absolutely zero reason that anything like that would have to be incorporated into `julia` or the Julia parser.

P.S.: If you need some help: [ChatGPT](https://chat.openai.com/share/be61b5f1-d91b-4fa3-90f1-16601c9ecadc) 😉

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [January 16, 2024, 12:16pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/133 "2024-01-16T12:16:53Z")

</div>

> [@goerz](#):
>
> Anybody can write a simple search-and-replace script in a few minutes that round-trips between unicode and some escaped representation

I think the shortest way would be to fork JuliaSyntax.jl? Because we don’t want to touch string/character literals.

> [@goerz](#):
>
> (or even a representation that still parses as valid Julia).

That still parses as valid julia and generates the same AST (to avoid interactions with metaprogramming)? I don’t see that.

> [@goerz](#):
>
> There is absolutely zero reason that anything like that would have to be incorporated into `julia` or the Julia parser.

Well, one would also want REPL-support, IDE-support and syntax-highlighting. Hence many projects that would need to standardize on a single escaped representation. And maybe one would like to avoid headaches a la “which of these files need pre-processing to unescape unicode”, by simply asking the julia-parser to do that. After the more involved REPL / IDE / JuliaSyntax.jl changes, that would be a very tiny code change. And thus it would be incorporated into the language.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [January 16, 2024, 1:41pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/134 "2024-01-16T13:41:10Z")

</div>

> [@foobar\_lv2](#):
>
> After the more involved REPL / IDE / [JuliaSyntax.jl](https://juliahub.com/ui/Packages/JuliaSyntax) changes, that would be a very tiny code change. And thus it would be incorporated into the language.

It likely will not. I mean given the opposition, I have no say in this.

> [@foobar\_lv2](#):
>
> I think the shortest way would be to fork [JuliaSyntax.jl](https://juliahub.com/ui/Packages/JuliaSyntax)?

You could, or anyone, and it could be a non-default parser. Before it was merged, it could still be used. I’m still amazed how dynamic Julia is, that you can switch out the parser. It’s not like this is unheard of maybe, I guess possible in Lisp (Racket?), but given its “non-syntax” not sure done. I just think unheard of in (most other) languages, e.g. C/C++.

But would you want to (spend time on it) even if you could, and _assuming it would not be merged_? Maybe, maybe not, you will convince more people after a ready PR. But at least I don’t want the parser bloated. I want it as fast as possible. Maybe it’s not a huge risk it will become (much) larger, but I’m skeptical of the gain even. Maybe I didn’t read your proposal well enough…

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [January 16, 2024, 1:52pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/135 "2024-01-16T13:52:41Z")

</div>

My guidance here would be the same as another recent syntax idea:

> [@Proposed alias for union types](https://discourse.julialang.org/t/proposed-alias-for-union-types/108205/127):
>
> How to advocate for this really depends upon how much you care and how much you time and energy you want to put in. The best way to truly get a complete evaluation would indeed be with a comprehensive pull request that implements the definition you want, complete with tests, news, etc., as well as a concise enumeration of the tradeoffs everyone brought to this thread so it’s easy to weigh the pros and cons. But that takes not-insignificant work… and it’s not guaranteed to ever fully resolve. …

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [January 16, 2024, 1:58pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/136 "2024-01-16T13:58:19Z")

</div>

> [@Palli](#):
>
> Maybe, maybe not, you will convince more people after a ready PR.

^ This. Maybe I will submit a PoC, or maybe I’ll give up on that.

This thread was conclusive in the sense that significant opposition exists. That still surprises me, and I still cannot really understand it – “meh, too much work” is a point of opposition that I anticipated.

However, most of this thread appeared to focus on denying my lived experience of “unicode in source-code really sucks for me, and IDE + tab-completion is a different ballgame than bring-your-own-editor”.

I also think that I am not alone in that – but as others mentioned upthread, maybe many such people don’t use julia. I can second that first reaction from many programmer julia-outsiders to unicode operators is “are you batshit insane?”.

I think that’s a pity: Stuff like unicode, or begin-end syntax, or one-based indexing are surface syntax.

I don’t particularly like julia’s surface syntax, but I really adore the dynamic multi-dispatch and corresponding compilation + specialization model + type system, and that to me is the heart of the language. So I can tolerate some annoying surface syntax.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 16, 2024, 2:00pm UTC](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363/137 "2024-01-16T14:00:12Z")

</div>

Closed on January 16, 2024, 2:00pm UTC

This topic was automatically closed after 2 days. New replies are no longer allowed.

[Previous page](https://discourse.julialang.org/t/syntax-escape-hatch-for-unicode-haters/108363.md?page=6)
