# Building an incremental sysimage in seconds instead of minutes

**URL:** <https://discourse.julialang.org/t/building-an-incremental-sysimage-in-seconds-instead-of-minutes/139352>\
**Category:** Internals & Design\
**Tags:** performance, compilation, precompilation, package-compiler\
**Created:** [September 10, 2026, 8:56pm UTC](https://discourse.julialang.org/t/building-an-incremental-sysimage-in-seconds-instead-of-minutes/139352 "2026-09-10T20:56:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![melevy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/melevy/32/209696_2.png) [@melevy](https://discourse.julialang.org/u/melevy)\
**Post date:** [September 10, 2026, 8:56pm UTC](https://discourse.julialang.org/t/building-an-incremental-sysimage-in-seconds-instead-of-minutes/139352/1 "2026-09-10T20:56:55Z")

</div>

**Disclaimer:** this post was written using AI, but I did my best to make this informative and useful for others. The post has been flagged for being an advertisement, it is not. I hope it passes this time.

This is a research project. I’m putting it out so that others can experiment with it. You have to build both branches from source.

Some background on why I did this. I’m writing a discrete event network simulator in Julia. People write their simulation models on top of it, and not everyone wants to work in the REPL. If someone prefers to run a compiled binary instead, I don’t want to prevent that and lose a customer. The problem was that `create_app` took two and a half minutes for every source change, which is way too slow to be a real workflow. So I spent some time making the compiler incremental.

The result is a patched Julia (branched off 1.13.0-rc4) and a patched PackageCompiler. Both branches are public and they work today.

## What it does

The build result is kept in a store, roughly the way git keeps its objects: every build is a snapshot. When you edit a source file and rebuild, it

1. boots a Julia process from the previous system image,
2. diffs the tracked sources and evaluates only the top level expressions that changed,
3. compiles only what Julia’s own invalidation killed,
4. links everything else (most of Base, the packages, the rest of your app) from the object code of the earlier builds,
5. writes a new image and replaces a single file in the bundle, `lib/julia/sys.so`. The executable, the libraries and the artifacts stay as they are.

No build ever runs your program. The compile roots come from a `--trace-compile` trace that is taken once, in a throwaway process, when the store is created.

Calls from newly compiled code into reused code are direct calls, resolved by symbol at link time, so there’s no dispatch penalty. A call that isn’t inlined costs 1.0 ns from new code into reused code, and 1.0 ns inside reused code. (The trampoline it replaced cost 43 ns.)

## The compiler server

By default the store keeps a compiler process running in the background. It’s started with `julia --reactive-server=<socket>`, it has the image of the last build loaded, and it stays there between your edits. A rebuild sends the changed expressions to it over a Unix socket, the server applies them and compiles, then forks a child that writes out the image. Because of the fork the parent keeps its heap, its ledger and everything it compiled so far, so your next edit starts from there again.

That’s most of the difference between 0.8 s and 12 s. Without the server every rebuild has to start a process, load the 167 MB image and redo the setup before it can even look at your edit.

The protocol is deliberately boring: a Unix socket, one request per connection, one line in each direction, no libuv. Any build tool can start the server and drive it, one of the gates does exactly that from a Python script that knows nothing but the socket path. `status` tells you whether the server runs, `stop` shuts it down, and `--no-server` does the rebuild in a fresh process instead.

## Numbers

Measured on the routing sample of my simulator: a 167 MB system image with 65067 functions, on an AMD Ryzen AI Max+ 395 running Linux, with 8 image threads.

```julia-auto
full build (create_app) 155 s 9.7 GB peak
rebuild after an edit 0.8 s 1.3 GB, 37 of 65067 functions recompiled
binary start after a rebuild 0.23 s same as after a full build

```

Applying the source diff is 1 to 3 ms of that 0.8 s. The rest is inference of the invalidated cone, precompiling the trace, writing the heap and linking.

The way the image gets written makes a difference. In one test of ten consecutive edits on the same app: overlay 2.1 to 2.6 s, dirty page write 3.1 to 3.3 s, whole relink 7.0 to 7.6 s. The 0.8 s above is overlay mode, measured after I moved the rebuild machinery into the system image itself. With `--no-server` it’s 12 to 13 s.

## How I verified it

The invariant I was after: for any sequence of edits that get applied, the incremental image should be semantically equal to the image you’d get from a full build of the final sources. Semantically equal means the same method tables (same signatures, same lowered code) for every tracked module, a valid code instance for every trace root, the same program output, and no leftover workload state in the heap.

Not byte equal though, and that’s on purpose. Inference recursion limits depend on which root inference started from, so a recomputed code instance can come out different and still be correct.

There’s an oracle that digests an image (method tables, roots, output, globals). The gate scripts create a store, run a chain of edits through it, and compare that digest against a full build of the same final sources. Together the gates cover 14 categories of source change plus the refusals, ten edits through the compiler server, all three image write modes, the trimmed binary, and the command line. One of them also checks that the image of a chain doesn’t grow, and that it keeps exactly one definition per live function.

Two functional checks on top of that. The mean hop count of a simulation goes from 2.308011 to 4.616022 after the edit (exactly double, which is what the edit does), and back when I revert it. And all 14 call shapes I could think of reach the edited code: keyword, varargs, `invoke`, `@cfunction`, finalizer, opaque closure, static parameter and so on.

If a change can’t be applied you get a refusal with a reason and a file:line, and the store is left alone. You never end up with a stale image, and the fallback is always a full build.

## Command line

```julia-auto
julia -m PackageCompiler build <app_dir> [--package=<dir>] [options]
julia -m PackageCompiler status <app_dir>
julia -m PackageCompiler stop <app_dir>
julia -m PackageCompiler watch <app_dir> [--package=<dir>] [options]

```

`build` creates the store if there isn’t one in `app_dir`, otherwise it rebuilds. `status` prints what’s in the store: when it was created, the snapshots, the overlay chain, whether the server runs, how much the image grew. `stop` stops the compiler server. `watch` builds once and then rebuilds every time you save a tracked file, until you hit Ctrl-C.

Options when the store gets created: `--package=<dir>` is the package to build, `--workload=<file>` is the script that gets traced for the compile roots, `--tracked=<file>=<Module>,...` overrides the list of tracked files (by default it’s whatever the package root file includes), `--executable=<name>=<main>`, and `--optimization`, `--debug-info`, `--cpu-target` for the image itself.

Options for a rebuild:

- `--image=overlay|pages|whole` picks how the image gets written. `overlay` is the default, it writes the new code as a separate shared object and patches the base image word by word. `pages` rewrites only the dirty pages. `whole` relinks the whole image.
- `--delta-opt=0..3` is the optimization level for the newly compiled code only. Level 1 is noticeably faster, and the reused code keeps whatever it was built with, so your hot paths stay at -O3.
- `--no-server` does the rebuild in a fresh process instead of the resident server. Slower, but nothing stays in memory between builds.
- `--found` forces a full build right now.
- `--compact=<n>,<f>` does a full build automatically after n saves, or after the image grew by f.
- `--trim` also writes a trimmed binary next to the bundle. Your entry point has to be `Base.@ccallable julia_main` for that.

Every option has an environment variable as its default. On the Julia side everything is a flag, `julia --help-hidden` lists the `--reactive-*` ones. From Julia code you can also call `create_app(...; reactive = :auto)`, which turns a build into a rebuild if a store is already there.

## How to try it

First build the patched Julia, this is a normal Julia source build:

```julia-auto
git clone https://github.com/levy/julia && cd julia
git checkout reactive-compiler
make -j$(nproc)

```

Then get the client:

```julia-auto
git clone https://github.com/levy/package-compiler && cd package-compiler
git checkout reactive

```

Then, from an environment that has this PackageCompiler:

```julia-auto
julia --project=<env> -m PackageCompiler build /tmp/myapp \
      --package=/path/to/MyApp --workload=/path/to/workload.jl
/tmp/myapp/bin/MyApp
# edit something in MyApp/src
julia --project=<env> -m PackageCompiler build /tmp/myapp # about a second

```

The design document, the plans with all the measurements and the gate scripts are in `contrib/reactive-compiler/` on the Julia branch. If you want to check for yourself that it works, run `tool/gate_c.sh`. It needs the branch built in `usr/` and the PackageCompiler checkout next to it, nothing else.

## Limitations

- Linux only, and one CPU target per store. The image format has no clone tables, and the link needs `--gc-sections`.
- A store is tied to the Julia build that created it. If you rebuild Julia, you have to create the store again.
- Some changes can’t be applied and need a full build: a new dependency, an `include` of a file that isn’t tracked, a changed module header, a module option, or a type that a method in an untracked file uses.
- Only your app’s own sources are diffed. If you change a package from the Manifest, that’s a full build.
- The image format is version 3, so you can’t rebuild from a stock Julia image.

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/julialang/original/3X/1/2/12829a7ba92b924d4ce81099cbf99785bee9b405.png) [@system](https://discourse.julialang.org/u/system)\
**Post date:** [September 10, 2026, 10:34pm UTC](https://discourse.julialang.org/t/building-an-incremental-sysimage-in-seconds-instead-of-minutes/139352/2 "2026-09-10T22:34:13Z")

</div>



---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/julialang/original/3X/1/2/12829a7ba92b924d4ce81099cbf99785bee9b405.png) [@system](https://discourse.julialang.org/u/system)\
**Post date:** [September 11, 2026, 6:58am UTC](https://discourse.julialang.org/t/building-an-incremental-sysimage-in-seconds-instead-of-minutes/139352/3 "2026-09-11T06:58:43Z")

</div>