Tab completion of \uXXXX in the REPL?

[devel-chm]

@foobar_lv2 This topic was split off from your recent discourse thread but it started as a suggestion addressing some of the issues in your posts in that discussion.

@foobar_lv2

my input keyboard should not constrain the set of ASTs I can generate.

My proposal is adding better, more uniform support for Unicode entry by extending
the REPL tab completion from the current ad hoc LaTeX-ish expansions by allowing
expansion of any Unicode codepoint in the REPL.

I see this as more a polishing of the existing REPL expansion capability to allow entry
of any Unicode character and not just the set of characters that are blessed by historic
or common usage, such as \alpha or \mu. This would not only allow any Unicode
character to be entered, but could also enable search/recognition of existing aliases.
If there were no existing alias, it could offer an immediate ability to add an entry to
complete that character.

While there was a generally dismissive attitude towards this capability, the
discussions clarified what is available in the existing Julia support that could be
leveraged to smooth some of the rough edges:

1. String macros could be implemented (here are examples for Unicode only support
   and for LaTeX only support). N.B. My proposal would essentially combine these 2
   into the existing REPL TAB completion capability.

   @stevengj

   You can do something now with a string macro, no parser changes needed:

   macro uvar_str(s::AbstractString)
      sym = Meta.parse(unescape_string(s))
      sym isa Symbol || throw(ArgumentError("expecting a single symbol"))
      return esc(sym)
   end
   

   Which lets you do e.g.:

   julia> uvar"\ub5" = 3
   3
   
   julia> µ
   3
   
   julia> uvar"\u2208"('b', "foobar") # 'b' ∈ "foobar"
   true
   

   and

   @Dan

   Based on the previous macro, this version might have more compassion towards source-code reader:

   using REPL
   macro lvar_str(s::AbstractString)
       if !haskey(REPL.REPLCompletions.latex_symbols, "\\"*s)
           throw(ArgumentError("expecting relevant latex"))
       end
       new_s = REPL.REPLCompletions.latex_symbols["\\"*s]
       sym = Meta.parse(unescape_string(new_s))
       sym isa Symbol || throw(ArgumentError("expecting a single symbol"))
       return esc(sym)
   end
   

   julia> lvar"mu"
   3
   
   julia> lvar"pi"
   π = 3.1415926535897...
   

2. Entering the glyph as a character will display the unicode codepoint value

   @stevengj

   I showed how to get the UTF-8 encoding bytes with codeunits above, but it sounds like you really want the codepoint value, which you can get easily by e.g.:

   julia> '🎅'  # display information about the character in the REPL
   '🎅' : Unicode U+1F385 (category So: Symbol, other)
   
   julia> UInt32('🎅' ) # codepoint as an integer value
   0x0001f385
   

   Note that a “visible glyph” might be more than one character, e.g. 'α̂ '(a single “grapheme”) is two characters:

   julia> collect("α̂")
   2-element Vector{Char}:
   'α': Unicode U+03B1 (category Ll: Letter, lowercase)
   '̂': Unicode U+0302 (category Mn: Mark, nonspacing)
   

   and you can get information about how to type it easily by pasting it at the help?> prompt:

   help?> α̂
   "α̂" can be typed by \alpha<tab>\hat<tab>
   

3. You can use string escape sequences to return a string with the desired unicode in it

   @stevengj

   You can also type codepoint values as \uXXXX escape sequences into a string and then copy-paste it:

   julia> "\u03B1\u0302"
   "α̂"
   

For additional information on Unicode support currently in Julia, see the documentation for
the Unicode standard library.

I hope this collected information is helpful for anyone having some of these Unicode pain points.
My thanks to everyone who participated in this discussion.

Also, I meant to include use Glyphy in the suggestion by @cormullion, thanks!