Pygments and Tokenize.jl

Community
1

Pygments seems to be pretty widely used for source code formatting (pythontex and JuliaBox uses it). There is a Pygments lexer for Julia, but it is based on regex and is not quite perfect. Has anyone looked into integrating Tokenize.jl into Pygments?

1 Like
2

There is https://github.com/ZacLN/DocumentFormat.jl, which is based on https://github.com/ZacLN/CSTParser.jl which is based on https://github.com/KristofferC/Tokenize.jl. The whole stack is mainly used to power the auto formatting in the julia VS Code extension, so Iā€™m not 100% how stable the API is that e.g. DocumentFormat.jl exposes, but the goal is certainly to have that as a really powerful native julia code formatting package that can be more widely used.

@ZacLN would know more about the stability of that package.

3 Likes