# Pygments and Tokenize.jl

**URL:** <https://discourse.julialang.org/t/pygments-and-tokenize-jl/8253>\
**Category:** Community\
**Created:** [January 9, 2018, 4:49pm UTC](https://discourse.julialang.org/t/pygments-and-tokenize-jl/8253 "2018-01-09T16:49:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mykelk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mykelk/32/879_2.png) [@mykelk](https://discourse.julialang.org/u/mykelk)\
**Post date:** [January 9, 2018, 4:49pm UTC](https://discourse.julialang.org/t/pygments-and-tokenize-jl/8253/1 "2018-01-09T16:49:54Z")

</div>

Pygments seems to be pretty widely used for source code formatting (pythontex and JuliaBox uses it). There is a [Pygments lexer for Julia](https://bitbucket.org/natashawatkins/pygments-main/src/a893dbff5bfc4359d07d222bb9c3b4a0dec062bc/pygments/lexers/julia.py?fileviewer=file-view-default), but it is based on regex and is not quite perfect. Has anyone looked into integrating [Tokenize.jl](https://github.com/KristofferC/Tokenize.jl) into Pygments?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 9, 2018, 5:04pm UTC](https://discourse.julialang.org/t/pygments-and-tokenize-jl/8253/2 "2018-01-09T17:04:54Z")

</div>

There is [https://github.com/ZacLN/DocumentFormat.jl](https://github.com/ZacLN/DocumentFormat.jl), which is based on [https://github.com/ZacLN/CSTParser.jl](https://github.com/ZacLN/CSTParser.jl) which is based on [https://github.com/KristofferC/Tokenize.jl](https://github.com/KristofferC/Tokenize.jl). The whole stack is mainly used to power the auto formatting in the [julia VS Code extension](https://github.com/JuliaEditorSupport/julia-vscode), so I’m not 100% how stable the API is that e.g. DocumentFormat.jl exposes, but the goal is certainly to have that as a really powerful native julia code formatting package that can be more widely used.

@ZacLN would know more about the stability of that package.
