\mscr unicode character display in REPL with Windows

General Usage
1

Hi, I have been enjoying some of the many Unicode characters available in Julia (https://docs.julialang.org/en/stable/manual/unicode-input/), however, found that some of them do not display in the REPL in Windows. Specifically found the issue with some of the \mscr & \mit characters.

Note I have been using vscode as my editor with Julia 0.6.2.

Example:

  1. “\mscrI” displays as ℐ in both the editor and ℐ the REPL.
  2. “\mscrP” displays as 𝒫 in the editor but as � in the REPL
  3. “\mitP” displays as 𝑃 in the editor and as � in the REPL

It is curious that some work and others do not (example 1 is fine). These are just some that I came across.

In addition, the characters display fine in the terminal (if you paste from the editor) after installing and applying the DejaVu fonts family, but still fail to show correctly in the Julia REPL. I have also tried the same using the ConEmu console emulator and get the same results (characters display fine in terminal but show up as � in REPL).

The problem seems to be Windows related since I tried example 2 with Ubuntu and the character displayed fine in the REPL.

Is this a known issue? any thoughts would be appreciated, thanks.

2

Seems like you might need to switch to fonts that have these characters.

3

For the Julia and JuliaPro installations on windows, is there any way to create the shortcut to the console to use a font with better unicode support? This would make it a lot easier for users, since changing fonts is not something most windows users know how to do. Installing an appropriate font would also be very nice as well.

4

A longstanding issue has been the desire to ship Julia with a better console on Windows: https://github.com/JuliaLang/julia/issues/7267 … https://github.com/JuliaLang/julia/pull/12879 ran into some technical issues, but maybe the situation has improved?

1 Like
5

Minnty or Cmder or something would be nice, but low priority compared to other enhancements to the Julia environment. But just changing the font to a more unicode friendly one would be an improvement as a stop-gap.

1 Like
6

Thanks for the replies!

Very likely I am missing something but I did install the DejaVu font family and set the terminal to use that. In addition I tried the example with ConEmu with UTF-8 encoding with the same result. Because the character displays correctly in the terminal when pasted but not when pasted in Julia REPL led me to believe that it is not a problem with the font itself.

Please see the image below:

image

I get the same result with Powershell using the DejaVu font family.

I have not tried Minnty or Cmder, that will be my next test.

1 Like
7

I can replicate this is both Juno and VSCode. Both have DejaVu Sans Mono set as the font in both the editor and console. \mscrP displays fine in the editor, but not in the console, nor in the REPL. In all cases, Julia is running in PowerShell on Windows 10.

println("𝒜, ℬ, 𝒞, 𝒟, ℰ, ℱ, 𝒢, ℋ, ℐ, 𝒥, 𝒦, ℒ, ℳ, 𝒩, 𝒪, 𝒫, 𝒬, ℛ, 𝒮, 𝒯, 𝒰, 𝒱, 𝒲, 𝒳, 𝒴, 𝒵")

yields �, ℬ, �, �, ℰ, ℱ, �, ℋ, ℐ, �, �, ℒ, ℳ, �, �, �, �, ℛ, �, �, �, �, �, �, �, �

8

an update:

I have tried with Cmder & Mintty. Cmder also do not display all the fonts correctly (fonts based on Conemu I believe). The fonts do work by installing Cygwin & Mintty. However, it does not seem to be possible to use Mintty as an integrated terminal within VScode (only an external terminal).

image
image1555×810 49 KB

As has been motioned in previous posts, the font displays correctly in the editor (or if you paste it somewhere else e.g. notepad) but not in the Julia REPL. This is only a problem in Windows.

9

I think this is neither a Julia nor even a Windows problem. I think what Stefan said near the top is correct: these characters are simply missing in the font you’re using. My guess is that you only see them in your editor because it tries to be helpful and replaces missing characters with glyphs from some other font.

I think something similar is going on in the Markdown rendering as well. Look carefully at the println() statement above: the characters for B, E, F, H etc. are styled differently than A, C, D, G etc., and are clearly from a different font.

We can get a clue as to why these particular characters behave differently from a few lines of Julia:

julia> s = "𝒜, ℬ, 𝒞, 𝒟, ℰ, ℱ, 𝒢, ℋ, ℐ, 𝒥, 𝒦, ℒ, ℳ, 𝒩, 𝒪, 𝒫, 𝒬, ℛ, 𝒮, 𝒯, 𝒰, 𝒱, 𝒲, 𝒳, 𝒴, 𝒵";

julia> for i in eachindex(s)
         (s[i]==',' || s[i]==' ') && continue
         display(s[i])
       end
'𝒜': Unicode U+01d49c (category Lu: Letter, uppercase)
'ℬ': Unicode U+212c (category Lu: Letter, uppercase)
'𝒞': Unicode U+01d49e (category Lu: Letter, uppercase)
'𝒟': Unicode U+01d49f (category Lu: Letter, uppercase)
'ℰ': Unicode U+2130 (category Lu: Letter, uppercase)
'ℱ': Unicode U+2131 (category Lu: Letter, uppercase)
'𝒢': Unicode U+01d4a2 (category Lu: Letter, uppercase)
'ℋ': Unicode U+210b (category Lu: Letter, uppercase)
'ℐ': Unicode U+2110 (category Lu: Letter, uppercase)
'𝒥': Unicode U+01d4a5 (category Lu: Letter, uppercase)
'𝒦': Unicode U+01d4a6 (category Lu: Letter, uppercase)
'ℒ': Unicode U+2112 (category Lu: Letter, uppercase)
'ℳ': Unicode U+2133 (category Lu: Letter, uppercase)
'𝒩': Unicode U+01d4a9 (category Lu: Letter, uppercase)
'𝒪': Unicode U+01d4aa (category Lu: Letter, uppercase)
'𝒫': Unicode U+01d4ab (category Lu: Letter, uppercase)
'𝒬': Unicode U+01d4ac (category Lu: Letter, uppercase)
'ℛ': Unicode U+211b (category Lu: Letter, uppercase)
'𝒮': Unicode U+01d4ae (category Lu: Letter, uppercase)
'𝒯': Unicode U+01d4af (category Lu: Letter, uppercase)
'𝒰': Unicode U+01d4b0 (category Lu: Letter, uppercase)
'𝒱': Unicode U+01d4b1 (category Lu: Letter, uppercase)
'𝒲': Unicode U+01d4b2 (category Lu: Letter, uppercase)
'𝒳': Unicode U+01d4b3 (category Lu: Letter, uppercase)
'𝒴': Unicode U+01d4b4 (category Lu: Letter, uppercase)
'𝒵': Unicode U+01d4b5 (category Lu: Letter, uppercase)

The reason is that there are two completely different regions of Unicode involved here. I have no idea why related characters are so jumbled in Unicode.

EDIT: Also, Markdown’s syntax highlighting for Julia renders the characters in the lower Unicode region in red. So weird.

I suspect that the only way to solve this is to keep looking for a font that supports all these characters. Or maybe build your own chimera font by copying glyphs from fonts that support different regions?

2 Likes
10

There is a career at CSI waiting for you… :+1:

2 Likes
11

OK, back from holiday, so I had time to run a little test. Turns out it is a bit deeper. If I run Julia inside Git Bash, with the same font, all of \scrA to \scrZ actually display just fine. Running Julia inside PowerShell or the Windows console still displays placeholders instead, even though the characters display just fine in Atom. Git bash (mintty) is set to use UTF-8, which - to my knowledge - the Windows console does not handle properly.

1 Like
12

Bit late to the game, but this seems to basically be the explanation: utf 8 - Why are there holes in the Unicode table? - Stack Overflow (for whatever reason, probably popularity and demand, the U+21... characters were added before the U+01d4... set, and were then left out when the U+01d4... set was introduced to avoid duplicated characters and preserve backward compatibility).

The table is full of such reserved and “empty” spots, apparently: https://unicode-table.com/en/blocks/mathematical-alphanumeric-symbols/

13

Sorry to resurrect an old thread, but I wanted to note that there is a simple explanation for this puzzle: when a glyph is not present for a character in a given font, decent text-rendering software switches to a “fallback” font to find a glyph for that character. Mintty (used by git bash) supports fallback fonts, but the Windows console does not. So, probably your font does not have the \scrA character, but Mintty switches to a fallback font whereas the Windows console gives up. This has nothing to do with UTF-8.

1 Like
14

Thanks. I guess that’s all she wrote then, until the Windows console catches a wake-up.

15
16

Note though that currently julia (or rather the julia fork of libuv) does not play nicely with the new Windows Terminal. Details are here, but the short version is that julia currently is not using the right way to interact with the console on Windows to utilize the features now supported in Windows Terminal.

17

It works with git-bash! That’s enough to me

image
image862×371 16.2 KB

18

It also works with wsl in windows terminal.

19

Sorry to resurrect an old thread, but I found a way to automate running Julia in Mintty with one click in Windows 7. I hope this will be useful for other readers in this thread.
Basically I created a batch file with two lines and saved it to the desktop. This batch file does the following:
1 - It adds Julia and Mintty to the path local to this batch file (first line);
2 - It calls mintty passing julia as a command that should be executed by mintty (second line).

Example of a batch file created to open Julia 1.6.2 in Mintty (file julia_1_6_2.bat, saved in the desktop):

     path=%path%;C:\Users\User1\AppData\Local\Programs\Julia-1.6.2\bin;C:\Program Files\Git\usr\bin;
     mintty -h -e julia

By some strange reason, if mintty and julia are not in the path (first line in the batch file above), julia crashes.