I’m working on Makie.jl / MakieLayout.jl so I think a lot about compilation latency. I found the recent discussions about method invalidations quite interesting, as they point to an avenue for package developers to reduce compilation latency. As far as I understand, if you avoid introducing new return types when overloading Base functions, or avoid overloading all together, you can avoid a lot of recompilation.
So I thought about other ways in which the compiler’s work could be reduced. Isn’t it true that a completely self-contained package (doesn’t extend other packages’ or Base’s methods) could be separately compiled and then loaded into a REPL session in binary format? It should not invalidate any previous functions, so maybe it could just be “added” to the currently compiled code. I wonder if a plotting package like Makie.jl is a good package to try such an approach, because it doesn’t have any “algorithms” that need to be compiled through and through for custom types. Rather after the surface layer, all complex types will be converted to arrays of Floats or similarly basic stuff anyway.
So if we have a package that tries hard not to overload or extend existing functions, could we then not ship a completely precompiled (as in binary, not just type inferred) version of it which can be immediately loaded, without any more precompilation being necessary? We’d still have all the Julia magic at the surface level and be able to put any kind of types into plot!(), but all the backend code would be compiled.
Ins’t this what SnoopCompile does?
I think people are still using that, I think i even heard there was a github action for it now, but less sure of that.
You seem to be kind of mixing together things about static compilation, (redistrubuting) precompilation, normal JIT compilation, and compile cache invalidation.
Which is reasomable since they are related, but the conditions are not the same.
Hopefully someone can explain, with some detail
Especially the term precompilation seems to have a double meaning. I’m talking about anything that allows me to ship code that is already (partially) compiled to the user installing my package, so that they don’t have to wait when using it (instead of compiling a sysimage themselves, with PackageCompiler, e.g.). As I’ve read often that this is impractical because of the many interconnections of Julia packages, I was wondering if one could cut back on these interconnections. This is because even with all the advances that are being made in the compilation arena, Makie.jl will likely stay very slow to compile and that would seriously impact the amount of people we could reach. (No chance to use the package for quick plots, only lengthy REPL sessions).