# Documention around Julia build

**URL:** <https://discourse.julialang.org/t/documention-around-julia-build/91806>\
**Category:** General Usage\
**Created:** [December 18, 2022, 8:42am UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806 "2022-12-18T08:42:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [December 18, 2022, 8:42am UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/1 "2022-12-18T08:42:59Z")

</div>

Hi,  
I was wondering if there is documentation on how Julia is built. I am not referring on how to build Juliawhich is already done here:

> **[julia/doc/src/devdocs/build at master · JuliaLang/julia](https://github.com/JuliaLang/julia/tree/master/doc/src/devdocs/build)**
>
> master/doc/src/devdocs/build

I am thinking more about documenting what really happens when you run “make”  
I think documenting:  
-how the various part of Julia are built from the ground up when running “make”  
-how they interact together and what are their functions  
would attract more contributions and also educate people about Julia itself.  
Again, I would be happy to document the build process in more detail and do some kind of map of the various part of the language. But I need some help in creating such documentation.  
Thanks

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [December 18, 2022, 3:23pm UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/2 "2022-12-18T15:23:10Z")

</div>

> [@gitboy16](#):
>
> - how the various part of Julia are built from the ground up when running “make”

In general the build process is defined by a series of makefiles, so you may need to look at the source code to understand the details.

When building julia, most of the dependencies will use pre-compiled artifacts from BinaryBuilder, which only need to be downloaded.

> [@gitboy16](#):
>
> - how they interact together and what are their functions

For the purpose of some dependencies, please refer to THIRDPARTY.md

> <https://github.com/JuliaLang/julia/blob/master/THIRDPARTY.md>

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [December 18, 2022, 3:45pm UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/3 "2022-12-18T15:45:21Z")

</div>

# Overview of julia’s build process

```sh
# v1.8.3
make O=julia-wsl configure
make -C julia-wsl/ -j4 # build all things

```

We build julia in the folder `julia-wsl`.

## Build the order of targets

You can build these targets one by one, making it easy to observe each step of the action.

- `julia-deps`
- `julia-stdlib`
- `julia-base`
- `julia-cli-release`
- `julia-src-release`
- `julia-sysimg-release`
- `julia-base-cache`

## `make -C julia-wsl/ julia-deps`

Equivalent to `make -C julia-wsl/deps/`.

Most of the `deps\Makefile` file is making conditional judgments about whether to use the system’s, or julia’s compiled dependencies.  
There are a number of `include`s at the end of the makefile, and each individually imported `.mk` file is responsible for building a given dependency.

In general, most of the dependencies are pre-compiled in advance by `BinaryBuilder.jl`. When executing `make`, most of the dependencies are simply downloaded as pre-compiled artifacts, while the remaining few dependencies need to be compiled.

### Direct Download

BinaryBuilder pre-compiled dependencies are downloaded and unpacked directly to `julia-wsl\usr\`

The compilation script used for precompilation is found in the BB compilation configuration repo by the dependency name: [GitHub - JuliaPackaging/Yggdrasil: Collection of builder repositories for BinaryBuilder.jl](https://github.com/JuliaPackaging/Yggdrasil)

e.g.:

- libblastrampoline：[https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/libblastrampoline/build\_tarballs.jl](https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/libblastrampoline/build_tarballs.jl)
- curl：[Yggdrasil/L/LibCURL/common.jl at master · JuliaPackaging/Yggdrasil · GitHub](https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/LibCURL/common.jl)

### Source code compilation

Dependencies that need to be compiled from source code:

- `utf8proc`
- `libwhich`
- `patchelf`

## `make -C julia-wsl/ julia-stdlib`

Equivalent to `make -C julia-wsl/stdlib/`

Many of the julia standard libraries have been moved out of the main julia repo, so they need to be downloaded.  
There are also some standard libraries that have binary dependencies and need to be downloaded.

There are two types of julia packages

- The normal source-only packages, with `.jl` as a suffix
- thin wrappers of binary dependencies, usually automatically generated, with `_jll.jl` as a suffix.

e.g.: `LibCURL.jl`, `LibCURL_jll.jl`

### Standard libraries to be downloaded

> Print the list of standard libraries to be downloaded:  
> `make -C julia-wsl/stdlib/ print-STDLIBS_EXT`

`Pkg Statistics LibCURL Downloads ArgTools Tar NetworkOptions SuiteSparse SparseArrays SHA`

- The version number (commit) is specified by the corresponding `stdlib\*.version` file
- Download cache path: `stdlib\srccache`
- Installation path: `julia-wsl\stdlib`

### JLL binary dependencies to be downloaded

> Print all JLL packages:  
> `make -C julia-wsl/stdlib/ print-JLL_NAMES`

```julia-auto
MozillaCACerts_jll dSFMT_jll GMP_jll LibCURL_jll LibGit2_jll libLLVM_jll LibSSH2_jll LibUV_jll MbedTLS_jll MPFR_jll nghttp2_jll libblastrampoline_jll OpenBLAS_jll OpenLibm_jll p7zip_jll PCRE2_jll SuiteSparse_jll Zlib_jll LLVMLibUnwind_jll CompilerSupportLibraries_jll LibUnwind_jll

```

- The corresponding thin wrapper is already located in the `stdlib\` folder
- The version number is specified by the corresponding `stdlib\*_jll\Project.toml`
- Download and unpack and install it to `julia-wsl\usr\`

## `make -C julia-wsl/ julia-base`

Equivalent to `make -C julia-wsl/base/`

A number of symbolic links are created, generating a number of `.jl` files containing constants.

## `make -C julia-wsl/ julia-cli-release`

Equivalent to `make -C julia-wsl/cli/ release`

Build `cli` and `loader`.

## `make -C julia-wsl/ julia-src-release`

Equivalent to `make -C julia-wsl/src/ release`

Build the core of julia.

Folder build order:

- `julia/src`
- `julia/src/support`
- `julia/src/flisp`

## `make -C julia-wsl/ julia-sysimg-release`

> Warn: I don’t know much about this step, so I may not be describing it accurately.

Execute `sysimage.mk` to build the system image.  
Compile the `Base`+`stdlib` `.jl` file into a binary.

This step also generates the REPL pre-compilation statements and executes them.

## `make -C julia-wsl/ julia-base-cache`

> Write the sys source cache in format readable by Base.\_read\_dependency\_src  
> —— `etc\write_base_cache.jl`

## final

You can run `make -C julia-wsl/ -j4`. It will build two more test targets.  
After that you can execute `. /julia` to open the julia REPL.

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [December 18, 2022, 4:07pm UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/4 "2022-12-18T16:07:25Z")

</div>

> [@woclass](#):
>
> most of the dependencies are pre-compiled in advance by `BinaryBuilder.jl`.

You can decide whether to use BB pre-built binary dependencies by specifying the following flag:

e.g.: Compile `dsfmt` from source code

```julia
echo 'USE_BINARYBUILDER_DSFMT := 0' > julia-wsl/Make.user

```

> <https://github.com/JuliaLang/julia/blob/eb57a775ccbf05294584e203b10988d77d43da7a/Make.inc#L1163-L1180>

* * *

> [@woclass](#):
>
> Source code compilation

Some of julia’s dependencies have additional patches in `deps\patches`.  
Pre-built dependencies in Yggdrasil will also have the same patches.

---

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [December 18, 2022, 7:14pm UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/5 "2022-12-18T19:14:43Z")

</div>

Thank you @woclass , that looks amazing. I am going to study all this and might come with more questions. But that is more than anyone has done so far. Did you consider making a PR to document all this? I think that would be great for all the people trying to understand how Julia is built. I’ll mark your answer as solution for now even if I did not go through all the details you provided yet. THANK YOU again!

---

<div class="post-metadata">

**Author:** ![gitboy16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gitboy16/32/24906_2.png) [@gitboy16](https://discourse.julialang.org/u/gitboy16)\
**Post date:** [December 19, 2022, 11:10am UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/6 "2022-12-19T11:10:52Z")

</div>

@woclass , how were you able to figure out these steps? I can see that these are in the Makefile in the root directory but there are a lot of other steps as well:

- julia-symlink
- julia-libccalltest
- etc…

Is there a main procedure which is called when one runs `make`?  
I am asking because there is possibility in the future that these steps change and it would be nice to habe a way to find out what are the new steps if they change.  
Thank you

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [December 19, 2022, 12:05pm UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/7 "2022-12-19T12:05:02Z")

</div>

> [@gitboy16](#):
>
> Is there a main procedure which is called when one runs `make`?

`make` == `make default`  
This may be a feature of make.

- [makefile - How does “make” app know default target to build if no target is specified? - Stack Overflow](https://stackoverflow.com/questions/2057689/how-does-make-app-know-default-target-to-build-if-no-target-is-specified#:~:text=GNU%20Make%20also%20allows%20you%20to%20specify%20the,Thanks%20for%20contributing%20an%20answer%20to%20Stack%20Overflow%21)

> [@gitboy16](#):
>
> how were you able to figure out these steps?

**Dynamic debugging**  
I compile julia a lot and have contributed to the makefile in `deps`.  
So, I had a rough idea that the build process could be divided into several stages.  
I chose to add the _`-n` flag_ and debug the make process directly, which prints out the current command to be executed but does nothing.  
Then you can observe the behaviour of the makefile.

- [‘-n’ - GNU make](https://www.gnu.org/software/make/manual/make.html#Instead-of-Executing-Recipes)

You could search for “Debugging Makefile” to find some flags and methods for debugging a makefile.

**Reading Code**  
I mentioned above that `make default` is the first target, and then you can look at its dependencies, and the dependencies of the dependencies.  
This will eventually form a tree. (I guess there are ready-made tools to visualize the tree)

Combining the two methods above, we can identify the relationship between the targets.  
There are a lot of targets in the makefile, and you need to pick out some important points from them, so it’s up to you to make your own choices.  
I chose the above targets to divide the compilation phase, and ignored the targets you listed because they are simple.

* * *

There are also some targets that will not be in the `make default` dependencies.

- `docs`
- `check-whitespace`
- `win-extras`
- …

I think this might be a good place to start.  
We can filter out the targets not covered by `make default` and add comments to them one by one in place.

Perhaps we could also organize a small table and put it at the beginning of the makefile, or in a separate devdoc.

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [December 19, 2022, 12:26pm UTC](https://discourse.julialang.org/t/documention-around-julia-build/91806/8 "2022-12-19T12:26:02Z")

</div>

> [@gitboy16](#):
>
> Did you consider making a PR to document all this?

Not before, because people usually don’t care about the compilation process, or just want to know how to compile.

> [@gitboy16](#):
>
> in the future that these steps change

Yes, these makefiles are not part of the API, and they change quickly, so it is difficult to maintain a development document over time, except by adding some short comments in place.

> [@gitboy16](#):
>
> and might come with more questions.

I’d be happy to answer those questions.  
My main concern at the moment is the compilation and testing process of julia on windows, as well as the CI build.

If there are more in-depth questions that I may not be able to answer, you can go to [slack](https://julialang.org/slack/)’s #build-system, #ci-dev channel for help.