What is "incremental compilation" and what does it mean for it to be "broken"?

Consider three packages:

module A
f() = 1
end
module B
using A
A.f() = 2
end
module C
using A
A.f() = 3
end

Now the behavior of your system depends on which order you load these three packages; the last to define A.f() “wins.” This is broken behavior because Julia tries hard to ensure that the order of loading packages and compilation does not matter to final outcome (this is why we have invalidation).

Even more simply, it’s really bad form to change the behavior of a method. “Here’s how package A works, unless you happen to load B” is not scalable for a large ecosystem.

To address the meaning of “incremental compilation”: it’s the compilation that can be done when only a subset of the entire codebase is available. When we precompile a package, we don’t know what other packages it will be loaded with, and again this is why we need invalidation. This is very different from how C (the programming language, not the module above), for example, compiles, where first you define the entire “universe” of code and then it compiles without ever having to worry about what else might happen. But even in C there is a similar issue: if you’re building a shared library, the linker will complain if you load two libraries that define the same symbol. It’s basically the same warning as what you’re seeing here.

18 Likes