Julia Syntax highlighting on websites

I recently encountered an issue when trying to do syntax highlighting Julia code on my blog.

An example:

A C++ example:

Both are done with Franklin.jl using highlightjs.

I recon that Documenter has the same problem. Is there anyone who has experience in adding better support to a service like highlightjs which is used in both cases?
Maybe something can be ported from services like Juno or the VSCode extension?

Have the feeling that the community of bloggers is growing where this would be quite beneficial. Documenter and maybe some other services would benefit from it as well.

Cc: @tlienart

2 Likes

@mortenpi @mike any thoughts on this? I’m also aware of https://github.com/JuliaDocs/Highlights.jl but haven’t tried it in a while

Documenter is using hightlightjs as well as far as I recall, so it’ll be giving the same results.

Incidentally, I’m currently looking into updating Highlights.jl to work by loading grammars and themes directly from vscode definitions, which, if it actually works, should give us native highlighting on par with vscode with very little work since we won’t have to port hundreds of grammars. It’s a little ways off though.

8 Likes

I am using Chroma (with Hugo) and find the results good (and it is fast too).

Highlights.jl is a pretty good solution for this, imho – we’re using it for the package readmes on JuliaHub and so far no one has complained :wink:

Very exciting! Having all TextMate/Atom/VSCode grammars available will be great.

1 Like

I agree that Highlights.jl looks quite nice:

One thing I noticed is that variable names and types have the same class and therefore get the same color.

Tangentially related question: has anyone looked into using https://github.com/tree-sitter/tree-sitter-julia for highlighting?

nice, I didn’t know chroma, it seems to use the same stuff as highlightjs though:

I think that the differences are the following: Chroma is

  1. written purely in Go, incorporating ideas from Pygments
  2. is actively maintained, while highlightjs is looking for a maintainer,
  3. insanely fast.

If there is an issue with highlighting Julia code, maybe it is worth reporting it.

Right so speed is irrelevant here, basically highlighting is easy and typically over max a few hundred lines of code which will be done in <1ms for any decently coded highlighter.

The key IMO is whether there’s an up to date Julia Lang specification for the highlighting; that’s why Mike is discussing having a system that ports the specs from vscode which makes a lot of sense so that we as a community only have to maintain the specs in some format and possibly write code that translates the specs in a format readable by chroma, prism, highlight or whatever people use.

I have been thinking of using the built-in Julia parser (written in FemtoLisp, not Julia) for parsing. That is to say, given a piece of source code (either from stdin, or a file) like

abstract type Animal end
struct Lizard <: Animal
    name::String
end
race(l::Lizard, r::Rabbit) = "$(l.name) wins in wall climbing"

emit a stream of strings annotated as S-expressions, eg something like

(keyword "abstract")
(space " ")
(keyword "type")
(space " ")
(typename "Animal")
(space " ")
(end) 
(newline)

that would allow reconstruction of the original verbatim, but with annotation information.

The advantage would be that

  1. it is always valid and up-to-date,
  2. all it would need is the Julia binary (provided this functionality, written in FemtoLisp, is added to Julia, otherwise an external file would be needed)
  3. just using the built-in FemtoLisp is fast, with very little startup time (or one could use it directly)

A trivial piece of shim code could then convert format to whatever an IDE prefers for lexed data.

The disadvantage is that someone would have to write it :wink:

2 Likes

I can’t help but think that the code in Kristoffer’s OhMyREPL will be a good basis for a syntax highlighter.

2 Likes

The heavy lifting for that is done by another excellent package by Kristoffer,

I am thinking of getting similar output from the Julia parser, directly. But using a precompiled image with this package would also be fast.

Please correct me if I’m wrong, but I don’t think something like this is possible with the current design of the parser. It does have a concept of tokens, which is probably what you are looking for, but they are read one by one from an IO stream and don’t have any context attached to them. It would be possible to check whether something is a keyword, for example, using the parser infrastructure, but I believe you would have to factor out a lot of the logic there to make it work for this use case, which kind of negates a lot of the benefits from interfacing Julia’s parser in the first place. A better approach might be to use CSTParser.jl which seems a lot closer to what you are asking for and has even been discussed as an eventual replacement for the Scheme parser.

2 Likes

I think they do (LineNumberNode, obtained with flisp’s input-port-line), and there is a similar facility for columns (input-port-column), but currently the latter is only used in some special cases (error messages etc). AFACT the stream keeps track of these so they could be attached to the token information if required.

Yes, CSTParser.jl could provide a viable approach, too, that’s another great alternative. The important decision may just be whether we want to implement syntax highlighting within the Julia ecosystem, with the associated benefits (will be done right, full controll, not having to work around limitations of lexers not designed for Julia, programming can be done mostly in Julia) and costs (possibly larger startup time, but this can be mitigated), then the actual library can be found from many alternatives.

In any case, I think it is worth pursuing this. Julia’s syntax is quite difficult to parse with general parsers — we run into this with the Emacs julia-mode all the time.

1 Like

What I meant by context was less the line number and column, but rather whether a token acts just as a variable name, as a keyword, an infix operator… Currently those get figured out during the construction of the s-expressions, so it’s not as straightforward to retrieve just this information alongside the tokens.

1 Like

tree-sitter may actually be the simplest, and most useful path to go down here. It gives you fast parsers that produce CSTs that give you all the token info you need to highlight stuff, plus an actual syntax tree that you can do other cool things with.

Even though Atom looks like it’ll eventually be abandoned (which it where tree-sitter originated from I believe) it looks as though it’s got a second life in neovim so it’s probably not going to be abandoned itself and should accumulate parsers for plenty of languages in the long term. Writing new parsers also looks relatively straightforward compared to the regex-nightmares of tmLanguage :smiley:

Regarding implementing something based off of textmate/vscode grammars that I discussed above: we’d need to wrap the oniguruma regex lib since there appears to be subtle differences in some regex syntax compared to Julia’s PCRE that can’t really be glossed over. I went down a deep rabbit hole getting oniguruma to compile and trying to wrap it – more effort than it’s worth.

So I’ll probably start to wrap tree-sitter grammars and integrate them with Highlight.jl in the near future since, as far as I can tell, basing our highlighting off of that seems reasonably future-proof and a good return on investment of time.

4 Likes