# Writing a parser in Julia

**URL:** <https://discourse.julialang.org/t/writing-a-parser-in-julia/13769>\
**Category:** General Usage\
**Created:** [August 20, 2018, 3:57pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769 "2018-08-20T15:57:58Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Drgo](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Drgo](https://discourse.julialang.org/u/Drgo)\
**Post date:** [August 20, 2018, 3:57pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/1 "2018-08-20T15:57:58Z")

</div>

Hello,  
Has anyone written a parser in Julia for an arbitrary syntax (other than Julia’s)? Any packages available for lexing and parsing?

Thanks,

---

<div class="post-metadata">

**Author:** ![dawbarton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dawbarton/32/215461_2.png) [@dawbarton](https://discourse.julialang.org/u/dawbarton)\
**Post date:** [August 20, 2018, 4:15pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/2 "2018-08-20T16:15:03Z")

</div>

In terms of helper packages, there is [Tokenize.jl](https://github.com/KristofferC/Tokenize.jl) which powers [CSTParser.jl](https://github.com/ZacLN/CSTParser.jl).

---

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [August 20, 2018, 5:19pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/3 "2018-08-20T17:19:33Z")

</div>

I asked over at ANTLR if Julia generation might be of interest; seems it might, but we’d probably have to volunteer the generator code – or at least participate. I’m interested, but it’s not something I’m working on right now:

[https://github.com/antlr/antlr4/issues/2330](https://github.com/antlr/antlr4/issues/2330)

Less helpful, perhaps, but … I ended up hand-coding a tiny parser combinator framework for the parser I needed, and Julia turned out to be quite suited for the task 🙂 (The code isn’t quite release-worthy yet, I think, but I could send it to you if you want.)

---

<div class="post-metadata">

**Author:** ![Drgo](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Drgo](https://discourse.julialang.org/u/Drgo)\
**Post date:** [August 20, 2018, 7:38pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/4 "2018-08-20T19:38:07Z")

</div>

@mlhetland  
Thanks Magnus,  
Yes, agree. Porting Antlr is going to be a major exercise. I am still learning Julia, so I do not mind writing a small lexer/parser by hand. Seeing an example would really help. I would love to see the code you mentioned.

Best,

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [August 21, 2018, 6:33am UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/5 "2018-08-21T06:33:28Z")

</div>

This one is for the Reduce language

> **[GitHub - chakravala/Reduce.jl: Symbolic parser for Julia language term...](https://github.com/chakravala/Reduce.jl)**
>
> Symbolic parser for Julia language term rewriting using REDUCE algebra - GitHub - chakravala/Reduce.jl: Symbolic parser for Julia language term rewriting using REDUCE algebra

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [August 21, 2018, 8:18am UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/6 "2018-08-21T08:18:21Z")

</div>

CorpusLoaders.jl  
contain’s a parser for the SemCor format,

> <https://github.com/JuliaText/CorpusLoaders.jl/blob/master/src/SemCor.jl>

WordTokenizers.jl contains many tokenizers (lexers) for natural language.

> **[GitHub - JuliaText/WordTokenizers.jl: High performance tokenizers for natural...](https://github.com/JuliaText/WordTokenizers.jl)**
>
> High performance tokenizers for natural language processing and other related tasks - GitHub - JuliaText/WordTokenizers.jl: High performance tokenizers for natural language processing and other rel...

I’ld love to get ParserCombinators.jl working again.

> <https://github.com/andrewcooke/ParserCombinator.jl/pull/27>
>
> Ok this is building on #26 and #21 
> 
> I'm not saying this is the prettiest code…, but it passes all tests.
> 
> In a few places I got rid of inner constructors, and replaced them with outer constructors.
> They are much more sensibly behaved, being just functions.
> I'ld like to do that everywhere, but for now I am happy just to have it working.
> 
> I also drop all support for 0.5.
> And remove Compat.
> Its just easier to stop maintaining old versions (at least til 1.0) (In my opinion)
> Particularly given this package is stable so the version to version difference is likely just deprecation fixes

I think it might be better to start that again though with the new language features.  
as I don’t know that it was performant in 0.5,  
and just converting it isn’t going to solve that.

---

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [August 21, 2018, 8:25am UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/8 "2018-08-21T08:25:33Z")

</div>

This might be useful:

> **[GitHub - queryverse/TextParse.jl: A bunch of fast text parsing tools](https://github.com/queryverse/TextParse.jl)**
>
> A bunch of fast text parsing tools. Contribute to queryverse/TextParse.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![bicycle1885](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bicycle1885/32/107_2.png) [@bicycle1885](https://discourse.julialang.org/u/bicycle1885)\
**Post date:** [August 21, 2018, 9:18am UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/9 "2018-08-21T09:18:08Z")

</div>

For lexing, [Automa.jl](https://github.com/BioJulia/Automa.jl) may do good work.

---

<div class="post-metadata">

**Author:** ![Drgo](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Drgo](https://discourse.julialang.org/u/Drgo)\
**Post date:** [August 21, 2018, 3:01pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/10 "2018-08-21T15:01:03Z")

</div>

Thanks everyone for the valuable advice and to Magnus for the code sample. Much appreciated.

---

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [August 29, 2018, 5:03pm UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/11 "2018-08-29T17:03:55Z")

</div>

I see that Automa reads text bytewise; to what is Unicode (e.g., UTF-8) supported? It seems Unicode character classes (as in PCRE) are not supported; might it be possible to add that, at least to the extent that membership is easy to check (using things like `isletter` or `ispunct` or the like)?

---

<div class="post-metadata">

**Author:** ![bicycle1885](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bicycle1885/32/107_2.png) [@bicycle1885](https://discourse.julialang.org/u/bicycle1885)\
**Post date:** [August 30, 2018, 2:13am UTC](https://discourse.julialang.org/t/writing-a-parser-in-julia/13769/12 "2018-08-30T02:13:09Z")

</div>

Unicode character classes are not supported in Automa.jl. Technically speaking, it is possible to support encoded Unicode data like UTF-8, but it will need some fundamental changes to the code generation process of Automa.jl. Automa.jl changes the current state by reading a single byte from a data stream, and hence functions like `isletter` or `ispunct` are not workable here since these functions may need to consume multiple bytes from the stream.