# Compiler work priorities **URL:** \ **Category:** Internals & Design\ **Created:** [November 16, 2018, 10:14pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623 "2018-11-16T22:14:46Z")\ **Posts on this page:** 20\ **Page:** 1

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\ **Post date:** [November 16, 2018, 10:14pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/1 "2018-11-16T22:14:46Z")

In response to > [@State of the Debugger](https://discourse.julialang.org/t/state-of-the-debugger/4308): > > Hi there, I just wanted to ask about the state of the debugger in Julia. I am coming from Matlab and the debugger there is quite important for me to see where something is going wrong. For example, using NLsolve is a bit of a black box for me. If it does not work, I don’t know where to start to look for the problem. In contrast, in the meantime I copied the code to Matlab and could stop during the the execution of fsolve and check what is going wrong. Now it works in Matlab, but still it does n… I thought I might write a little post about the rough priorities of the compiler team: 1. Correctness - finding and fixing compiler and inference bugs 2. Multithreading - [non-copying task stack switching](https://github.com/JuliaLang/julia/pull/13099) (done) - [new PARTR parallel runtime](https://github.com/JuliaLang/julia/pull/22631) - [locks for I/O operations](https://github.com/libuv/libuv/issues/1595) - other thread safety 3. Compile-time latency, aka “the time-to-first-plot problem” - making compilation faster - caching more things 4. Compiler-related packages and tools - [PackageCompiler](https://github.com/JuliaLang/PackageCompiler.jl) - Debugger - [Cxx](https://github.com/Keno/Cxx.jl) - type checking/linting Julia 1.0 introduced a new compiler and there were inevitably various bugs and issues, so the first order of business is fixing those. Multithreading is the next highest priority because we want to merge the new parallel runtime as soon as possible and allow people to start writing and using threaded package code before things get too far along. Ecosystems for single-threaded languages tend to start to bake that assumption in very deep if they’re around for too long. Julia’s package ecosystem is small enough and young enough that if we introduce proper multithreading soon, it will adapt quickly. Compile-time latency is clearly a huge pain point for all Julia users, so that’s quite important. **Update:** all of these items are complete, see [this post](https://discourse.julialang.org/t/compiler-work-priorities/17623/106) for details. ---

**Author:** ![cdsousa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdsousa/32/215553_2.png) [@cdsousa](https://discourse.julialang.org/u/cdsousa)\ **Post date:** [November 16, 2018, 11:54pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/2 "2018-11-16T23:54:06Z")

Thank you for the info! I’m so happy that Cxx.jl made it to the list! ---

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\ **Post date:** [November 17, 2018, 7:46am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/3 "2018-11-17T07:46:44Z")

I am already excited for the next JuliaCon’s 1 second first plot live demo! No pressure! ---

**Author:** ![LaurentPlagne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurentplagne/32/10103_2.png) [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)\ **Post date:** [November 17, 2018, 10:14am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/4 "2018-11-17T10:14:10Z")

It is clear that this would be nice but not as revolutionary as efficient nested parallelism: if this is really achieved it would make Julia an unrivaled tool for hpc scientific computing. In my domain, the ability to provide compiled scientific libs written in Julia and easy to use from large simulation application written in other language would also make a great difference because early adopters can start to interact with others and demonstrate the Julia’s power and productivity. ---

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\ **Post date:** [November 17, 2018, 10:40am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/5 "2018-11-17T10:40:57Z")

It may be part of the compiler-time latency “caching more things” and “PackageCompiler,” but I would agree that “getting Julia to the point where it generates great `.so` files that can be used by other languages” would likely have a positive impact on Julia adoption. ---

**Author:** ![viralbshah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viralbshah/32/54_2.png) [@viralbshah](https://discourse.julialang.org/u/viralbshah)\ **Post date:** [November 17, 2018, 4:53pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/6 "2018-11-17T16:53:36Z")

In addition to what Stefan said, the way to accomplish more compiler projects is: 1. If you know of funding opportunities, please let me know and it will help us at Julia Computing grow our compiler team and execute more compiler projects. 2. The Julia Lab at MIT is also another place where this could be a research topic for the right person. 3. Influence tech companies to contribute engineering manpower, if they can’t contribute money. Many of the large tech firms do large open source contributions. The more vocal Julia users are, the more likely we will get contributions. 4. Discuss with people at National Labs. The more they adopt Julia, the more we will get contributions and capabilities from that community. Personally, I work on all the options above. I find that there are Julia users in every university, company, national lab, government, etc. but we need to be more vocal about asking for help - money or time. -viral ---

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\ **Post date:** [November 17, 2018, 5:23pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/7 "2018-11-17T17:23:36Z")

Thanks for laying it out. One thing I would like to ask though is, how can I help? I don’t have the time to really be a core contributor there (my efforts are probably best kept concentrated in DiffEq), but there’s gotta be something I can do? Fixing compile times is quite important to me so I would like to offer myself as an extra set of hands who knows packages which take a long time to compile. I opened [Understanding the compile times of DifferentialEquations.jl and Attempting to Help](https://discourse.julialang.org/t/understanding-the-compile-times-of-differentialequations-jl-and-attempting-to-help/17628) to get feedback but haven’t gotten any. Hopefully there’s a way to decrease the burden on the few compiler workers in order to better distribute the work and get it done! ---

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\ **Post date:** [November 17, 2018, 5:47pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/8 "2018-11-17T17:47:08Z")

Not sure whether this is a valid way of accomplishing more compiler projects, but I find the barrier to entry for the compiler somewhat higher than for Base. This is because the data structures and source for Base are pretty apparent, well-documented and you can always play around in the REPL. In contrast, the compiler is harder to hack for relative newcomers. Are there any guides for going into that? Guides for setting up a separate compile chain (at least the julia parts) for hacking, e.g. `using Revise, ExtraCompiler` such that I can make modifications that don’t break my session (beginning with insertion of `print`s until the debugger improves)? Descriptions of internal data-structures? Is there a sensible way of helping with that (given the constraint that I currently don’t get how everything fits together)? Re caching more things: If I finally passed the barrier-to-entry for compiler hacking, something I’d like to try is to create a persistent (cross session) cache. Idea would be that each compiled entity gets a collision-resistant hash of its inferred code, and we have a Merkle ~~tree~~ DAG where `hash(A)` incorporates `hash(B)` if `A` depends on `B`, i.e. if there is a backedge `B->A`. “compiled entity” means component of the dependency graph (need to collapse directed cycles). That way, much of llvm’s work and possibly some previous optimizer passes could theoretically be cached (last time I brought this up I was informed that linking is not ready for caching native code; and I still don’t get the world age system). ---

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\ **Post date:** [November 18, 2018, 7:14am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/9 "2018-11-18T07:14:47Z")

Thanks for this writeup. There is something I feel could belong to this list: more detailed documentation on what the compiler is expected to optimize, and better tools for inspecting cases it didn’t. I often run into cases where I wonder if a particular optimization or type inference didn’t happen because I didn’t code it the right way, or if the compiler is not handling that optimally (yet). The best strategy I know at the moment is asking here, and reading `Base` code for getting ideas, but the compiler is often a black box to me. `@code_warntype` is useful in finding out _what_ happened, but sometimes it is more difficult to say _why_ it happened and what I can do about it. ---

**Author:** ![TsurHerman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tsurherman/32/1234_2.png) [@TsurHerman](https://discourse.julialang.org/u/TsurHerman)\ **Post date:** [November 18, 2018, 8:05am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/10 "2018-11-18T08:05:17Z")

> [@tim.holy](#): > > It may be part of the compiler-time latency “caching more things” and “PackageCompiler,” but I would agree that “getting Julia to the point where it generates great `.so` files that can be used by other languages” would likely have a positive impact on Julia adoption. I agree wholeheartedly, and I think the main blocker is the way multiple dispatch is resolved on some base module (usually Base) instead of being resolved by the caller. the general outline , along with some overly emotional correspondences (mainly on my side) can be found in this thread: > [@MultiFunctions: Context dispatch and binary cache](https://discourse.julialang.org/t/multifunctions-context-dispatch-and-binary-cache/11565): > > I have been writing this draft for several days , and got a little mixed up verbalising these ideas , which are simpler to understand using code and examples… If you want to jump straight into code, there is a link at the end. @Jameson @mbauman tl;dr There is a way to make Julia completely binary cacheable(between sessions), so it can become an effective way of developing and distributing .so .dll Background Function = a module wide constant symbol ( e.g :sum) without any specifications re… I also wrote a small POC using macro that generate a generated function … the added indirection gave me enough flexibility to prove that this is possible. If there is interest I will “patch” the POC to work on julia version 1.0 ---

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\ **Post date:** [November 18, 2018, 2:44pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/11 "2018-11-18T14:44:10Z")

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\ **Post date:** [November 20, 2018, 8:40am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/12 "2018-11-20T08:40:00Z")

2 posts were split to a new topic: [Making it easier to contribute to Julia](https://discourse.julialang.org/t/making-it-easier-to-contribute-to-julia/17762) ---

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\ **Post date:** [November 22, 2018, 4:45am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/13 "2018-11-22T04:45:39Z")

I hear what you’re saying about multithreading, but threads have a bit of a bad reputation as tools of parallel computation (which I say from the position of someone who has used threads quite a bit, and successfully, even if I say so myself). What I wonder about is the support of the other implementation mechanisms for parallel computing (remote channels and remote references). How solid is the code at the moment, and are any changes/improvements planned? I suppose any work in this area would perhaps not fall under the “compiler” heading, so may be this is not the place to ask? ---

**Author:** ![phlavenk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phlavenk/32/1873_2.png) [@phlavenk](https://discourse.julialang.org/u/phlavenk)\ **Post date:** [November 23, 2018, 1:37pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/14 "2018-11-23T13:37:35Z")

I’d like also ask if the topic `PackageCompiler` will focus also on reducing the number of required dependencies - not linking libraries that are never called in the code, and full AOT with disabling JIT-runtime? This would support the use-case where a scientist develops the “bussiness-logic” dynamic library that will be deployed as part of a big C# / C++ library, without 500 MB dependencies (why ship FFTW when never used) and without unused stdlibs. ---

**Author:** ![zsz00](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zsz00/32/6314_2.png) [@zsz00](https://discourse.julialang.org/u/zsz00)\ **Post date:** [December 4, 2018, 3:20pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/15 "2018-12-04T15:20:32Z")

update Cxx.jl is very important, there are many repositories that rely on it. ---

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\ **Post date:** [December 4, 2018, 3:28pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/16 "2018-12-04T15:28:19Z")

> [@zsz00](#): > > update Cxx.jl is very important, there are many repositories that rely on it. Just noticed that there seems to be a [new commit](https://github.com/Keno/Cxx.jl/commit/c62cb578ebde43d040e3f175ea2365b292c6f487) from Keno himself. Very exciting! It would be really nice if more people get involved so that we are not relying on one very busy guy for absolutely everything. ---

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\ **Post date:** [December 4, 2018, 9:58pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/17 "2018-12-04T21:58:34Z")

Agreed. @zsz00 and others who mentioned Cxx, checking out the branch and doing obvious fixes for test failures might earn brownie points for showing you care enough to help out. While Cxx surely has a lot of difficult-to-update components, many packages require a certain amount of annoying but fairly routine work that’s pretty easy for most people to help out with. If someone can take that burden off Keno, it might clear more time for him to focus on the more difficult pieces. ---

**Author:** ![cdsousa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdsousa/32/215553_2.png) [@cdsousa](https://discourse.julialang.org/u/cdsousa)\ **Post date:** [December 4, 2018, 11:41pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/18 "2018-12-04T23:41:20Z")

Some developments on Cxx.jl are being reported in the comments of [Make Cxx.jl ready for production environments post Julia 1.0.0 · Issue #390 · JuliaInterop/Cxx.jl · GitHub](https://github.com/Keno/Cxx.jl/issues/390) ---

**Author:** ![Gnimuc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gnimuc/32/2194_2.png) [@Gnimuc](https://discourse.julialang.org/u/Gnimuc)\ **Post date:** [December 5, 2018, 1:16am UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/19 "2018-12-05T01:16:53Z")

> [@tim.holy](#): > > If someone can take that burden off Keno, it might clear more time for him to focus on the more difficult pieces. +1 for this, but > > > https://github.com/JuliaInterop 😆 😆 😆 ---

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\ **Post date:** [December 6, 2018, 6:48pm UTC](https://discourse.julialang.org/t/compiler-work-priorities/17623/20 "2018-12-06T18:48:30Z")

I’ll cross-post this here for more broad attention: > [@Julia LLVM projects?](https://discourse.julialang.org/t/julia-llvm-projects/1349/26): > > Wit lot’s of @keno help, I was able to put up instructions to build julia with webassembly: [https://nextjournal.com/sdanisch/julia-webassembly](https://nextjournal.com/sdanisch/julia-webassembly) You can edit this directly online or use the instructions to set it up locally. This basically just needs someone to go through lots of compilation issues and fix them one after the other wink @sdanisch has posted about how to compile Julia for wasm—if you’re looking to get involved or help with compiler work in a high-impact way that is fairly accessible, this is your chance! Also, think about how fun it will be when you get Julia running in a browser 😁 [Next page](https://discourse.julialang.org/t/compiler-work-priorities/17623.md?page=2)