Suggestions for Projects that add features to Julia

Hello Julia community!

My friend and I are final year undergraduate students. We have an interest in compilers/interpreters and we’ve done some work related to Clang/LLVM in the past.

For our final year undergraduate project, we have 6 months to work on a project of our choosing. Since I am a bit new to Julia, I’d appreciate it if I could get some feature requests of suitable difficulty that we can take a stab at. Ideally the project should add something to Julia/ or its tooling. It’s not compulsory that it be officially integrated into the mainline, so we’re open to trying out experimental things.

We just need a starting point. We’ll be doing all the required research. Thank you for your time!

Links to past work in compilers:

  1. https://github.com/0Xero7/Argon
    2.https://github.com/AshutoshPandey123456/micropy-convert
12 Likes

If you want a cool, but pretty hard project, a Julia parser written in Julia would be really cool, and would make it easier for us to improve error messages, speed of parsing, and fix a few longstanding issues in Julia.

9 Likes

@vchuravy?

2 Likes
4 Likes

That’s only part of the battle, right? You will have to make it able to parse Julia code. Also, you have to make it work without special bootstrapping.

1 Like

You may want to browse the following labels:

codegen issues

latency issues

optimizer issues

EDIT: fixes links, thanks @Palli

4 Likes

Sure, but my point was that “a Julia parser written in Julia” already exists and it probably isn’t worth redoing that.

6 Likes

I’m interested in this! What are some of the requirements?

1 Like

The real challenge to actually replace femtolisp with this thing and pass all regression tests?

4 Likes

CSTParser.jl is quite well tested at this point, so I think the main challenge in replacing the current parser would be solving bootstrapping and providing the right entrypoints for the rest of Julia’s C ABI. I believe @c42f already started a lot of this work in https://github.com/JuliaLang/julia/pull/35243 and https://github.com/JuliaLang/julia/pull/35844, but it will certainly still be fairly nontrivial to implement. This is also not enough to completely replace all femtolisp code in Julia, since lowering is also implemented in Lisp.

2 Likes

Yes, I rearranged the runtime a bit so that it should be quite easy to plug in an alternative parser at runtime.

Actually with that background infrastructure done, there’s some relatively low-hanging fruit available here which would be an excellent starting point. For example, @Ashutosh_Pandey if you wanted to pick up https://github.com/JuliaLang/FancyDiagnostics.jl/pull/4 and finish it off it could be pretty useful.

Note that you don’t need to solve the bootstrapping problem for this to be useful, because the parser can be replaced at runtime instead.

Replacing all the flisp code in the frontend is indeed a big job. There’s roughly 2x more code in the lowering passes than the parser, and some of it is fairly tricky.

8 Likes

I assumed is was only in the parser (and that’s it not that speed-critical). Is it only additionally in the “lowering passes” and do you have an idea if performance could improve a lot by replacing?

And on a related note, Python 3.9 changed to a PEG parser (from LL(1) parser):

I’m not up-to-speed on CSTParser.jl I just know https://github.com/abeschneider/PEGParser.jl is not its dependency. Nor are https://github.com/wdebeaum/PEG.jl and GitHub - vonDonnerstein/StringParserPEG.jl: A Library for Parsing Strings in Julia via Parsing Expression Grammars that are supported for Julia 1.0.

Would changing the Julia parser, to CSTParser.jl and/or otherwise, have some advantages, unrelated to speed, as with the Python change?

1 Like

Hi all, thank you for all the responses! I am the other guy working with @Ashutosh_Pandey on this project.

From what we understand, CSTParser.jl is already in very good shape, and the bootstrapping problem seems to be fairly complex for a first try. We do like @c42f’s comment about FancyDiagnostics.jl, it will probably be a great starting point to familiarize ourselves with the inner workings of the parser.

Do you have any resources which would help us get started with FancyDiagnostics.jl?

3 Likes

There’s not a lot of resources or documentation, to be honest. But do feel free to ask specific questions here or elsewhere.

CSTParser is in very good shape performance- and functionality-wise, and it’s quite well battle tested due to being part of the VSCode extension, among other things. CSTParser is also a lot faster than the flisp parser — 20x or so when I measured it. Another great thing about CSTParser is that it produces precise source code location info, due to its use of a CST rather than an AST.

As of the start of this year, I thought CSTParser could improve in several areas:

  • Documentation - there’s almost none right now and I found the structure of the code a bit of a mystery.
  • Error states - CSTParser knows fairly precisely where an error occurred, but the reason for the error can be hard to extract.

@zacln or @davidanthoff might be able to provide guidance or explain where I’m mistaken about the state of things :slight_smile:

I think the advantages of rewriting the compiler frontend (parsing+lowering) would be to:

  • Preserve more source information to provide precise compiler diagnostics. For a start, syntax errors could be more precise — this is the low-hanging fruit.
  • Expose some of the source code transformations and analyses from lowering as APIs. This would enable certain types of advanced metaprogramming which currently require macros to re-implement parts of lowering.
  • Make the compiler implementation more accessible to the julia community.
  • Possibly performance in some circumstances. I don’t think the compiler frontend is a huge bottleneck compared to inference and code generation in the default configuration. However in certain circumstances (eg, low optimization levels) the difference could be substantial. Largely this is just my gut feeling — should be measured.
7 Likes

Something that I think would make a fun project would be to write a Lisp-to-Julia string macro. So, that, for example lisp"""(display "Hello, world!")""" would expand to print("Hello, world"), etc.

Not sure exactly how useful this would be. Julia already has most of the nice features of Lisp, but I think some people would enjoy embedding snippets of Lisp into their Julia code just for the fun of it.

Also, if you’d make it femtolisp compatible, then the julia parser would technically be written in Julia once you wrap it in that string macro.

1 Like

This idea is basically https://github.com/swadey/LispSyntax.jl. (LispSyntax.jl is Clojure flavoured. But it would be more poetic if it was flisp flavored :smiley: )

1 Like

Cool! It seems the Julia ecosystem is getting to the point where, if you can think of a cool idea for a package, then somebody has probably written it already!

1 Like