# What is "incremental compilation" and what does it mean for it to be "broken"?

**URL:** <https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956>\
**Category:** General Usage\
**Tags:** precompilation\
**Created:** [June 29, 2023, 12:51am UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956 "2023-06-29T00:51:42Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [June 29, 2023, 12:51am UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/1 "2023-06-29T00:51:43Z")

</div>

I’ve seen posts talking about a warning “incremental compilation may be fatally broken for this module”. Sometimes it’s not clear why, sometimes it happens because they overwrote a method in a package. What does this mean exactly? I tried reading about incremental compilers that only recompile the edited parts of source code, but the details are pretty complicated and different across compilers, so I figured I would ask about Julia specifically. The broad concept reminds me of method invalidation and Revise.jl, but I don’t see how that could possibly break.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [June 29, 2023, 1:51pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/2 "2023-06-29T13:51:00Z")

</div>

Consider three packages:

```julia
module A
f() = 1
end

```

```julia
module B
using A
A.f() = 2
end

```

```julia
module C
using A
A.f() = 3
end

```

Now the behavior of your system depends on which _order_ you load these three packages; the last to define `A.f()` “wins.” This is broken behavior because Julia tries hard to ensure that the order of loading packages and compilation does not matter to final outcome (this is why we have invalidation).

Even more simply, it’s really bad form to change the behavior of a method. “Here’s how package A works, _unless_ you happen to load B” is not scalable for a large ecosystem.

To address the meaning of “incremental compilation”: it’s the compilation that can be done when only a subset of the entire codebase is available. When we precompile a package, we don’t know what other packages it will be loaded with, and again this is why we need invalidation. This is very different from how C (the programming language, not the module above), for example, compiles, where first you define the entire “universe” of code and then it compiles without ever having to worry about what else might happen. But even in C there is a similar issue: if you’re building a shared library, the linker will complain if you load two libraries that define the same symbol. It’s basically the same warning as what you’re seeing here.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [June 29, 2023, 2:19pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/3 "2023-06-29T14:19:55Z")

</div>

Would it be good to add a short version of your explanation to that warning, to reduce confusion and give people something actionable to do when they encounter this? As is, the warning isn’t really helpful if you’re not already deep in the weeds on how julia compilation works under the hood.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [June 29, 2023, 9:22pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/4 "2023-06-29T21:22:09Z")

</div>

Maybe a precompilation section or page in the Pkg docs or something? The warnings already write a couple sentences, and I don’t think that little text could ever really get the point across.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [June 29, 2023, 9:28pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/5 "2023-06-29T21:28:08Z")

</div>

Some points for clarification:

1. Is it correct to say incremental compilation means precompilation of each package in isolation?

2. Even if it’s broken, it would never stop our program from running, right? For example, loading (and invalidating) A,B,C in any order results in a functional program, even if the behavior differs by order.

3. What does happen to precompilation when this is broken? Does loading ignore or delete cached precompiled code?

> [@tim.holy](#):
>
> Even more simply, it’s really bad form to change the behavior of a method. “Here’s how package A works, _unless_ you happen to load B” is not scalable for a large ecosystem.

1. Am I correct to think this is limited to type piracy altering the dispatch behavior? IIRC from a previous thread, extending a function without type piracy should not cause method invalidations, so shouldn’t package X precompile fine regardless of whether package Y has added `X.foo(::Y.WhyType)` yet?

2. So far this has been about separate packages competing to determine a method’s behavior, but this warning has shown up for [a constructor duplicated within a few lines of StringDistances.jl](https://github.com/matthieugomez/StringDistances.jl/issues/56). Why would that be a problem, shouldn’t the method dispatch have been sorted out before the package precompiles? There’s also an odder example of the overwriting happening [in the same _line_ in Flux.jl](https://github.com/FluxML/Flux.jl/issues/1370), but that went away so I assume it was a patched Julia bug.

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 30, 2023, 12:35am UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/6 "2023-06-30T00:35:11Z")

</div>

> [@Benny](#):
>
> but this warning has shown up for [a constructor duplicated within a few lines of StringDistances.jl](https://github.com/matthieugomez/StringDistances.jl/issues/56)

Note that it’s possible to enable warnings for overwritten methods with the `--warn-overwrite=yes` command line flag.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [June 30, 2023, 2:16am UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/7 "2023-06-30T02:16:43Z")

</div>

I think a bunch of messages like this could do with ending up in this form (rust-inspired):

```julia
Warning: something is dodgy
  for more information see `explaincode(:J123)`

```

Or if `isinteractive()` is false

```julia
Warning: something is dodgy
  for more information see `julia --explain J123`

```

---

<div class="post-metadata">

**Author:** ![roflmaostc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roflmaostc/32/30123_2.png) [@roflmaostc](https://discourse.julialang.org/u/roflmaostc)\
**Post date:** [June 30, 2023, 2:45pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/8 "2023-06-30T14:45:00Z")

</div>

Is that bad form too?

Isn’t that the whole point of multiple dispatch to _change_ or extend behavior of methods of different packages?

```julia
julia> module A
         f(x) = 1
       end
Main.A

julia> module B
         using Main.A
         A.f(x::Int) = 2
       end
Main.B

julia> using Main.B

julia> Main.A.f(1)
2

julia> Main.A.f(1.0)
1

```

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [June 30, 2023, 3:46pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/9 "2023-06-30T15:46:48Z")

</div>

> [@Benny](#):
>
> 1. Is it correct to say incremental compilation means precompilation of each package in isolation?

Yes

> [@Benny](#):
>
> 1. Even if it’s broken, it would never stop our program from running, right? For example, loading (and invalidating) A,B,C in any order results in a functional program, even if the behavior differs by order.

Correct. But order in which you load packages is not something that we want mattering (it can introduce deadlocks in package dependency graphs, where two simultaneously-loaded packages need A and B in opposite orders), hence the scary warning.

> [@Benny](#):
>
> 1. What does happen to precompilation when this is broken? Does loading ignore or delete cached precompiled code?

invalidation, which means ignore

> [@Benny](#):
>
> 1. Am I correct to think this is limited to type piracy altering the dispatch behavior? IIRC from a previous thread, extending a function without type piracy should not cause method invalidations, so shouldn’t package X precompile fine regardless of whether package Y has added `X.foo(::Y.WhyType)` yet?

You can only overwrite a method if you’re committing type piracy (all the types must be defined in the first package to load, otherwise they can’t have the same signature). Hence this warning only happens if you’re committing type piracy.

> [@Benny](#):
>
> 1. So far this has been about separate packages competing to determine a method’s behavior, but this warning has shown up for [a constructor duplicated within a few lines of StringDistances.jl](https://github.com/matthieugomez/StringDistances.jl/issues/56). Why would that be a problem, shouldn’t the method dispatch have been sorted out before the package precompiles?

Not necessarily; compilation might happen any time you execute code at top level, e.g.,

```julia
module MyPackage
foo() = 1
const v = vcat(foo(), 2)
foo() = 2
end

```

compiles `foo` and then overwrites it during precompilation. At the end of the process `v` is not consistent with what you’d expect given the (current) definition of `foo`.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [June 30, 2023, 3:48pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/10 "2023-06-30T15:48:47Z")

</div>

Your example is type-piracy: `B` owns neither `A.f` nor `Int`. That method needs to be in `A`, not `B`. If `B` defines _new_ types, then it can extend `A.f` to operate on those types; there’s no way `A` could know about those types.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [June 30, 2023, 8:31pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/11 "2023-06-30T20:31:01Z")

</div>

> [@tim.holy](#):
>
> You can only overwrite a method if you’re committing type piracy

But, you can commit type piracy and invalidate a method without overwriting it or its callees. Or by “overwrite” did you mean invalidation rather than definition?

```julia
julia> foo(::Integer) = 0; # not overwritten
julia> bar(x) = foo(x); # not overwritten
julia> bar(1)
0
julia> foo(::Int) = 1; # define new method
julia> bar(1) # bar(::Int) was invalidated
1

```

If broken incremental compilation means invalidation of precompiled code, then I imagine `foo(::Int) = 1` invalidating `bar(1)` would do that just like `foo() = 2` invalidating `vcat(foo(), 2)`?

> [@tim.holy](#):
>
> Not necessarily; compilation might happen any time you execute code at top level

Didn’t seem like the method was executed between definitions in the linked example. Here’s the patch that fixed broken incremental compilation:

```julia
struct Normalized{T <: Union{StringSemiMetric, StringMetric}} <: StringSemiMetric
    dist::T
end
#- Normalized(dist::Union{StringSemiMetric, StringMetric}) = Normalized{typeof(dist)}(dist)
Normalized(dist::Normalized) = dist

```

The package was small enough to read over, and it doesn’t seem like there was a repeated include somewhere, either. Do type constructors compile the default constructor upon definition? (MethodAnalysis.methodinstances doesn’t seem to work on type constructors or functor instances). Whichever the case, does this count as type piracy if it’s all in 1 package?

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 30, 2023, 8:46pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/12 "2023-06-30T20:46:11Z")

</div>

> [@Benny](#):
>
> But, you can commit type piracy and invalidate a method without overwriting it or its callees. Or by “overwrite” did you mean invalidation rather than definition?

I’ll note that your first sentence is consistent with what Tim Holy is saying, so I’m not sure what’s your point.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [June 30, 2023, 8:49pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/13 "2023-06-30T20:49:42Z")

</div>

When tim.holy said “change the behavior of” or “overwrite” a method, I interpreted it as something like this:

```julia
foo(::Int) = 0
foo(x::Int) = x # foo(::Int) method gets replaced

```

as opposed to this

```julia
foo(::Integer) = 0
foo(x::Int) = x # foo(::Integer) method is unchanged

```

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 30, 2023, 8:54pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/14 "2023-06-30T20:54:22Z")

</div>

Yes, that’s how “overwrite” is used.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [June 30, 2023, 8:58pm UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/15 "2023-06-30T20:58:23Z")

</div>

And my point was that type piracy can occur without overwriting any existing method, at least given my understanding of what overwriting means, so I was asking for clarification. That’s almost verbatim to what I said before so I don’t understand the disconnect here.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 2, 2023, 9:02am UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/16 "2023-07-02T09:02:11Z")

</div>

> [@Benny](#):
>
> But, you can commit type piracy and invalidate a method without overwriting it or its callees

But none of those make behavior dependent on the order of package loading: if B and C do not overwrite `A.f` or each other, then behavior is consistent no matter what order you load packages in. In other words, this is not a warning about invalidation or type-piracy: it’s a warning that you may have a broken system. The cache files contain method definitions, not just native code, and it’s the state of the method definitions that is most strongly at risk here.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [July 2, 2023, 9:30am UTC](https://discourse.julialang.org/t/what-is-incremental-compilation-and-what-does-it-mean-for-it-to-be-broken/100956/17 "2023-07-02T09:30:29Z")

</div>

Ah, I was under the impression any invalidations cause the warning, but it’s only a particular subset. So to recap, invalidations toss out package-wise precompiled code, which is called incremental compilation, but only the subset of invalidations with direct method replacement will also break it, which means behavior varies by package loading order.