# Julia's stability guarantee (e.g. precompilation and "incremental compilation may be fatally broken for this module")

**URL:** <https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635>\
**Category:** General Usage\
**Created:** [October 12, 2022, 7:39pm UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635 "2022-10-12T19:39:17Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 12, 2022, 7:39pm UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/1 "2022-10-12T19:39:17Z")

</div>

As far as I know Julia has 1) a _syntax_ guarantee, and 2) an API guarantee (for documented non-experimental stuff)

But that doesn’t seem to be enough in practice, for packages. I see some precompilation issues, and workaround in one package:

> <https://github.com/JuliaPlots/Plots.jl/pull/4406>
>
> Workaround for https://github.com/JuliaLang/julia/issues/46989.

I’m not too worried, about that package, it’s in good hands, and fixed already, I’m just curious about in general, since there was an issue for that package, it could happen to any package?

Precompilation is the default (once wasn’t), can be turned off. So what’s the problem with it, and could at least potential issues always be worked around by turning it off?

Why didn’t PkgEval catch this? I’m guessing it doesn’t run on Windows, so is Linux a bit better supported platform? Was this simply a bug, and precompilation and packages on all platforms supposed to be guaranteed?

At least part of the reason was a dependency of Plots (or older version of) StatsPlots and:

```julia
  [9] _show(io::IOStream, #unused#::MIME{Symbol("image/png")}, plt::WARNING: importing deprecated binding Colors.RGB1 into PlotUtils.
WARNING: importing deprecated binding Colors.RGB1 into Plots.
WARNING: importing deprecated binding Colors.RGB4 into PlotUtils.
WARNING: importing deprecated binding Colors.RGB4 into Plots.

```

I also see in logs: [DUMMY: investigating Windows precompilation failure in 1.8.2 by BioTurboNick · Pull Request #4445 · JuliaPlots/Plots.jl · GitHub](https://github.com/JuliaPlots/Plots.jl/pull/4445)

```julia
 WARNING: Method definition uv_readcb(Ptr{Nothing}, Int64, Ptr{Nothing}) in module Base at stream.jl:656 overwritten in module Plots at D:\a\Plots.jl\Plots.jl\src\backends\gr.jl:707.
  **incremental compilation may be fatally broken for this module**

WARNING: Method definition create_expr_cache(Base.PkgId, String, String, Array{Pair{Base.PkgId, UInt64}, 1}) in module Base at loading.jl:1661 overwritten in module Plots at D:\a\Plots.jl\Plots.jl\src\backends\gr.jl:758.
  **incremental compilation may be fatally broken for this module**

WARNING: Method definition create_expr_cache(Base.PkgId, String, String, Array{Pair{Base.PkgId, UInt64}, 1}, IO) in module Base at loading.jl:1661 overwritten in module Plots at D:\a\Plots.jl\Plots.jl\src\backends\gr.jl:758.
  **incremental compilation may be fatally broken for this module**

WARNING: Method definition create_expr_cache(Base.PkgId, String, String, Array{Pair{Base.PkgId, UInt64}, 1}, IO, IO) in module Base at loading.jl:1661 overwritten in module Plots at D:\a\Plots.jl\Plots.jl\src\backends\gr.jl:758.
  **incremental compilation may be fatally broken for this module**

```

The possibility of that warning “incremental compilation may be fatally broken for this module” [on 29 Mar 2019](https://github.com/JuliaLang/julia/pull/31191#event-2239733978), so I believe in Julia 1.2.

I don’t want to be misinforming people (nor too positive). I think I know pretty much why/when Julia packages are stable, and the few exceptions (did didn’t wok on newer Julia), e.g. Cxx.jl.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 12, 2022, 9:26pm UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/2 "2022-10-12T21:26:15Z")

</div>

What is the question? There seems to be a regression in 1.8 on Windows, yes. There is an issue open to track it. When it gets fixed the fix can be backported into 1.8.x.

This has nothing to do with any stability guarantee since that is about intentional changes.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 13, 2022, 11:33am UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/3 "2022-10-13T11:33:46Z")

</div>

Ok, it was a simple bug (“regression”), and no guarantee guards against that, so the stability guarantee more of a commitment to fix bugs, and try to prevent them (and is of course meant for package code too). Why you got me confused with “that is about intentional changes”.

Are you saying the changes where in Plots.jl or some of its dependencies (e.g. StatsPlots)? Or some “intentional changes” in Julia, since the workaround in the package is for the still open issue in Julia (note Plots.jl has though be fixed):

> <https://github.com/JuliaLang/julia/issues/46989>
>
> I have no other explanation, the error was first spotted in https://github.com/J…uliaPlots/Plots.jl/pull/4402, and is constantly reproducible.
> 
> \`windows\` - \`julia 1.8.1\`: https://github.com/JuliaPlots/Plots.jl/actions/runs/3158230006/jobs/5140026800 ✔
> \`windows\` - \`julia 1.8.2\`: https://github.com/JuliaPlots/Plots.jl/actions/runs/3158234450/jobs/5140035915 ✘
> 
> PR pinning \`1.8.1\` in \`ci\`: https://github.com/JuliaPlots/Plots.jl/pull/4404 ✔
> PR pinning \`1.8.2\` in \`ci\`: https://github.com/JuliaPlots/Plots.jl/pull/4403 ✘
> 
> \<details\>
> 
> \`\`\`julia
> ✗ Plots
> ✓ ImageQualityIndexes
> ✓ VisualRegressionTests
> ✓ Images
> ✗ StatsPlots
> 189 dependencies successfully precompiled in 311 seconds. 117 already precompiled.
> 2 dependencies errored. To see a full report either run \`import Pkg; Pkg.precompile()\` or load the packages
> Testing Running tests...
> ERROR: LoadError: SystemError: opening file "C:\\\\Users\\\\RUNNER~1\\\\AppData\\\\Local\\\\Temp\\\\jl\_0Tex0PafwR.png": No such file or directory
> Stacktrace:
> \[1\] systemerror(p::String, errno::Int32; extrainfo::Nothing)
> @ Base .\\error.jl:176
> \[2\] #systemerror#80
> @ .\\error.jl:175 \[inlined\]
> \[3\] systemerror
> @ .\\error.jl:175 \[inlined\]
> \[4\] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
> @ Base .\\iostream.jl:293
> \[5\] open
> @ .\\iostream.jl:275 \[inlined\]
> \[6\] open(f::Base.var"#387#388"{String}, args::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
> @ Base .\\io.jl:382
> \[7\] open
> @ .\\io.jl:381 \[inlined\]
> \[8\] read
> @ .\\io.jl:462 \[inlined\]
> \[9\] \_show(io::IOStream, #unused#::MIME{Symbol("image/png")}, plt::WARNING: importing deprecated binding Colors.RGB1 into PlotUtils.
> WARNING: importing deprecated binding Colors.RGB1 into Plots.
> WARNING: importing deprecated binding Colors.RGB4 into PlotUtils.
> WARNING: importing deprecated binding Colors.RGB4 into Plots.
> Plots.Plot{Plots.GRBackend})
> @ Plots D:\\a\\Plots.jl\\Plots.jl\\src\\backends\\gr.jl:2180
> \[10\] #invokelatest#2
> @ .\\essentials.jl:729 \[inlined\]
> \[11\] invokelatest
> @ .\\essentials.jl:726 \[inlined\]
> \[12\] show
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\output.jl:227 \[inlined\]
> \[13\] #281
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\output.jl:6 \[inlined\]
> \[14\] open(::Plots.var"#281#282"{Plots.Plot{Plots.GRBackend}}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
> @ Base .\\io.jl:384
> \[15\] open
> @ .\\io.jl:381 \[inlined\]
> \[16\] png(plt::Plots.Plot{Plots.GRBackend}, fn::String)
> @ Plots D:\\a\\Plots.jl\\Plots.jl\\src\\output.jl:4
> \[17\] savefig(plt::Plots.Plot{Plots.GRBackend}, fn::String)
> @ Plots D:\\a\\Plots.jl\\Plots.jl\\src\\output.jl:139
> \[18\] var"##1#313"()
> @ Plots D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:20
> \[19\] top-level scope
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:25
> \[20\] eval
> @ .\\boot.jl:368 \[inlined\]
> \[21\] eval
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\Plots.jl:1 \[inlined\]
> \[22\] \_broadcast\_getindex\_evalf
> @ .\\broadcast.jl:670 \[inlined\]
> \[23\] \_broadcast\_getindex
> @ .\\broadcast.jl:643 \[inlined\]
> \[24\] getindex
> @ .\\broadcast.jl:597 \[inlined\]
> \[25\] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, typeof(Plots.eval), Tuple{Vector{Expr}}})
> @ Base.Broadcast .\\broadcast.jl:899
> \[26\] materialize
> @ .\\broadcast.jl:860 \[inlined\]
> \[27\] macro expansion
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:32 \[inlined\]
> \[28\] macro expansion
> @ C:\\Users\\runneradmin\\.julia\\packages\\SnoopPrecompile\\UWvXF\\src\\SnoopPrecompile.jl:51 \[inlined\]
> \[29\] (::Plots.var"#574#575")()
> @ Plots D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:29
> \[30\] withenv(f::Plots.var"#574#575", keyvals::Pair{String, String})
> @ Base .\\env.jl:172
> \[31\] macro expansion
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:28 \[inlined\]
> \[32\] macro expansion
> @ C:\\Users\\runneradmin\\.julia\\packages\\SnoopPrecompile\\UWvXF\\src\\SnoopPrecompile.jl:107 \[inlined\]
> \[33\] top-level scope
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:4
> \[34\] include(mod::Module, \_path::String)
> @ Base .\\Base.jl:419
> \[35\] include(x::String)
> @ Plots D:\\a\\Plots.jl\\Plots.jl\\src\\Plots.jl:1
> \[36\] top-level scope
> @ D:\\a\\Plots.jl\\Plots.jl\\src\\Plots.jl:203
> \[37\] include
> @ .\\Base.jl:419 \[inlined\]
> \[38\] include\_package\_for\_output(pkg::Base.PkgId, input::String, depot\_path::Vector{String}, dl\_load\_path::Vector{String}, load\_path::Vector{String}, concrete\_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
> @ Base .\\loading.jl:1554
> \[39\] top-level scope
> @ stdin:1
> in expression starting at D:\\a\\Plots.jl\\Plots.jl\\src\\precompilation.jl:3
> in expression starting at D:\\a\\Plots.jl\\Plots.jl\\src\\Plots.jl:1
> in expression starting at stdin:1
> ERROR: LoadError: Failed to precompile Plots \[91a5bcdd-55d7-5caf-9e0b-520d859cae80\] to C:\\Users\\runneradmin\\.julia\\compiled\\v1.8\\Plots\\jl\_91B7.tmp.
> Stacktrace:
> \[1\] error(s::String)
> @ Base .\\error.jl:35
> \[2\] compilecache(pkg::Base.PkgId, path::String, internal\_stderr::IO, internal\_stdout::IO, keep\_loaded\_modules::Bool)
> @ Base .\\loading.jl:1707
> \[3\] compilecache
> @ .\\loading.jl:1651 \[inlined\]
> \[4\] \_require(pkg::Base.PkgId)
> @ Base .\\loading.jl:1337
> \[5\] \_require\_prelocked(uuidkey::Base.PkgId)
> @ Base .\\loading.jl:1200
> \[6\] macro expansion
> @ .\\loading.jl:1180 \[inlined\]
> \[7\] macro expansion
> @ .\\lock.jl:223 \[inlined\]
> \[8\] require(into::Module, mod::Symbol)
> @ Base .\\loading.jl:1144
> \[9\] include(fname::String)
> @ Base.MainInclude .\\client.jl:476
> \[10\] top-level scope
> @ none:6
> in expression starting at D:\\a\\Plots.jl\\Plots.jl\\test\\runtests.jl:2
> ERROR: LoadError: Package Plots errored during testing
> Stacktrace:
> \[1\] pkgerror(msg::String)
> @ Pkg.Types C:\\hostedtoolcache\\windows\\julia\\1.8.2\\x64\\share\\julia\\stdlib\\v1.8\\Pkg\\src\\Types.jl:67
> \[2\] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia\_args::Cmd, test\_args::Cmd, test\_fn::Nothing, force\_latest\_compatible\_version::Bool, allow\_earlier\_backwards\_compatible\_versions::Bool, allow\_reresolve::Bool)
> @ Pkg.Operations C:\\hostedtoolcache\\windows\\julia\\1.8.2\\x64\\share\\julia\\stdlib\\v1.8\\Pkg\\src\\Operations.jl:1813
> \[3\] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test\_fn::Nothing, julia\_args::Vector{String}, test\_args::Cmd, force\_latest\_compatible\_version::Bool, allow\_earlier\_backwards\_compatible\_versions::Bool, allow\_reresolve::Bool, kwargs::Base.Pairs{Symbol, IOContext{Base.PipeEndpoint}, Tuple{Symbol}, NamedTuple{(:io,), Tuple{IOContext{Base.PipeEndpoint}}}})
> @ Pkg.API C:\\hostedtoolcache\\windows\\julia\\1.8.2\\x64\\share\\julia\\stdlib\\v1.8\\Pkg\\src\\API.jl:434
> \[4\] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{Base.PipeEndpoint}, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:coverage, :julia\_args, :force\_latest\_compatible\_version), Tuple{Bool, Vector{String}, Bool}}})
> @ Pkg.API C:\\hostedtoolcache\\windows\\julia\\1.8.2\\x64\\share\\julia\\stdlib\\v1.8\\Pkg\\src\\API.jl:156
> \[5\] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:coverage, :julia\_args, :force\_latest\_compatible\_version), Tuple{Bool, Vector{String}, Bool}}})
> @ Pkg.API C:\\hostedtoolcache\\windows\\julia\\1.8.2\\x64\\share\\julia\\stdlib\\v1.8\\Pkg\\src\\API.jl:171
> \[6\] top-level scope
> @ D:\\a\\\_actions\\julia-actions\\julia-runtest\\latest\\test\_harness.jl:15
> \[7\] include(fname::String)
> @ Base.MainInclude .\\client.jl:476
> \[8\] top-level scope
> @ none:1
> in expression starting at D:\\a\\\_actions\\julia-actions\\julia-runtest\\latest\\test\_harness.jl:7
> Error: Process completed with exit code 1.
> \`\`\`
> \</details\>
> 
> (I can only test this through \`ci\`: I'm on \`linux\` and do not have access to a \`windows\` machine).

So I repeat my (second) question:

> Why didn’t PkgEval catch this?

Since this can happen, it can happen to other (Windows), possibly non-registered packages. Then of course you should file a bug. I think the bug should be obvious (or not, since only “WARNING: […]  
\*\* incremental compilation may be fatally broken for this module \*\*”).

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 13, 2022, 11:35am UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/4 "2022-10-13T11:35:34Z")

</div>

The stability guarantee is as complete as I want it, one more exception is that the stream of RNG numbers isn’t guaranteed across major versions (and that’s documented, changed from 1.6 LTS to 1.7, and could change again).

And Julia can’t be more stable than IEEE allows. +, -, \*, and / are, and I believe at least ^ and sqrt too but not e.g. sin. Calculated values of sin have improved in more recent Julia versions, so you can’t expect bit-identical results for floats (I believe that’s the “table maker’s dillemma”), while you can for ints (except involving rand). I think that would also apply to e.g. Posit standard, that specifies:

```julia
sin(posit) returns sin(posit), rounded.
sinPi(posit) returns sin(𝜋 × posit), rounded

```

---

<div class="post-metadata">

**Author:** ![gbaraldi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gbaraldi/32/22101_2.png) [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)\
**Post date:** [October 13, 2022, 11:41am UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/5 "2022-10-13T11:41:56Z")

</div>

Expecting stability for floating point is already not correct. Different computers might give different results for the same operations due to SIMD and FMA differences.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 13, 2022, 11:54am UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/6 "2022-10-13T11:54:21Z")

</div>

> [@gbaraldi](#):
>
> Expecting stability for floating point is already not expected.

By some, not by me, I’m just clarifying for others reading, what I believe to be true, and as I wrote it “is as complete as I want it”.

Some people might disagree with not having stability of floating point (for the major operations), e.g. Kahan, why `strictfp` was added to Java (since it’s become a no-op):

[https://news.ycombinator.com/item?id=13739793](https://news.ycombinator.com/item?id=13739793)

> > If the author of the IEEE 754 standard says that the way you do floating-point arithmetic is obviously a bug, you’re probably doing it wrong.
> 
> That’s interesting, because it seems to contradict his attitude toward FLT\_EVAL\_METHOD.
> 
> Kahan is a strong supporter of FLT\_EVAL\_METHOD==2—in fact, he’s quite possibly the single person who’s most responsible for its existence—and FLT\_EVAL\_METHOD==2 has been resoundly rejected by compilers on newer architectures due to being totally unintuitive. (For those who don’t know, FLT\_EVAL\_METHOD==2 means that “x∗y+z” can have a different result from “double tmp = x∗y; tmp + z”, _even if x, y, and z are doubles_. It’s a very similar situation to the bug, just with expression temporaries vs. explicit storage locations instead of register-memory spills.)
> 
> In particular, Kahan wrote a widely-shared, hyperbolic essay called “Why Java Floating Point Hurts Everyone Everywhere” (which is still sometimes shared around today as a way to attack Java, despite being obsolete) castigating Java for having FLT\_EVAL\_METHOD==0. Java added the little-used “strictfp” in response, but history proved Java’s original behavior right—with the advent of SSE on x86-64, GCC switched to FLT\_EVAL\_METHOD==0 by default, matching Java.

At [FLT\_EVAL\_METHOD - cppreference.com](https://en.cppreference.com/w/cpp/types/climits/FLT_EVAL_METHOD)

> #define FLT\_EVAL\_METHOD /\* implementation defined \*/ (since C++11)

Does Julia have something similar to FLT\_EVAL\_METHOD?

---

<div class="post-metadata">

**Author:** ![gbaraldi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gbaraldi/32/22101_2.png) [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)\
**Post date:** [October 13, 2022, 12:03pm UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/7 "2022-10-13T12:03:27Z")

</div>

I imagine LLVM does support something like this. That maybe could be exposed like `@fastmath` is, but that’s just a guess.

---

<div class="post-metadata">

**Author:** ![mikmoore](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikmoore/32/31109_2.png) [@mikmoore](https://discourse.julialang.org/u/mikmoore)\
**Post date:** [October 13, 2022, 3:59pm UTC](https://discourse.julialang.org/t/julias-stability-guarantee-e-g-precompilation-and-incremental-compilation-may-be-fatally-broken-for-this-module/88635/8 "2022-10-13T15:59:58Z")

</div>

> [@gbaraldi](#):
>
> Expecting stability for floating point is already not correct. Different computers might give different results for the same operations due to SIMD and FMA differences.

This is partly true and partly false. To clarify for other readers, Julia does not apply such transformations unless expressly permitted (via `@fastmath`, `@simd`, or `muladd`, for example – although I recall `muladd` sometimes leaking to adjacent operations). Code like

```julia
% not inside a @fastmath block
w = x * y + z

```

will always give exactly the same result for the same x, y, and z on any IEEE754-compliant architecture. It will not apply `fma` without an appropriate annotation. The operations `+` , `-`, `*`, `/`, `sqrt`, `fma`, `rem` are always correctly rounded except when transformed by a flag. Note that functions like `sum` may apply such flags, although `foldl(+,...)` will not.

However, nonprimitive operations like `exp`, `sin`, etc, are not strictly defined by IEEE754. Julia makes no guarantee that they will produce the same values between versions, nor that they will match the values produced by any other library or language. If you discover a substantial error in one of these operations, you can submit an issue and someone smart will eventually take a look to see if it can be improved.