Documention around Julia build

General Usage
1

Hi,
I was wondering if there is documentation on how Julia is built. I am not referring on how to build Juliawhich is already done here:

I am thinking more about documenting what really happens when you run “make”
I think documenting:
-how the various part of Julia are built from the ground up when running “make”
-how they interact together and what are their functions
would attract more contributions and also educate people about Julia itself.
Again, I would be happy to document the build process in more detail and do some kind of map of the various part of the language. But I need some help in creating such documentation.
Thanks

2

In general the build process is defined by a series of makefiles, so you may need to look at the source code to understand the details.

When building julia, most of the dependencies will use pre-compiled artifacts from BinaryBuilder, which only need to be downloaded.

For the purpose of some dependencies, please refer to THIRDPARTY.md

1 Like
3

Overview of julia’s build process

# v1.8.3
make O=julia-wsl configure
make -C julia-wsl/ -j4  # build all things

We build julia in the folder julia-wsl.

Build the order of targets

You can build these targets one by one, making it easy to observe each step of the action.

  • julia-deps
  • julia-stdlib
  • julia-base
  • julia-cli-release
  • julia-src-release
  • julia-sysimg-release
  • julia-base-cache

make -C julia-wsl/ julia-deps

Equivalent to make -C julia-wsl/deps/.

Most of the deps\Makefile file is making conditional judgments about whether to use the system’s, or julia’s compiled dependencies.
There are a number of includes at the end of the makefile, and each individually imported .mk file is responsible for building a given dependency.

In general, most of the dependencies are pre-compiled in advance by BinaryBuilder.jl. When executing make, most of the dependencies are simply downloaded as pre-compiled artifacts, while the remaining few dependencies need to be compiled.

Direct Download

BinaryBuilder pre-compiled dependencies are downloaded and unpacked directly to julia-wsl\usr\

The compilation script used for precompilation is found in the BB compilation configuration repo by the dependency name: GitHub - JuliaPackaging/Yggdrasil: Collection of builder repositories for BinaryBuilder.jl

e.g.:

Source code compilation

Dependencies that need to be compiled from source code:

  • utf8proc
  • libwhich
  • patchelf

make -C julia-wsl/ julia-stdlib

Equivalent to make -C julia-wsl/stdlib/

Many of the julia standard libraries have been moved out of the main julia repo, so they need to be downloaded.
There are also some standard libraries that have binary dependencies and need to be downloaded.

There are two types of julia packages

  • The normal source-only packages, with .jl as a suffix
  • thin wrappers of binary dependencies, usually automatically generated, with _jll.jl as a suffix.

e.g.: LibCURL.jl, LibCURL_jll.jl

Standard libraries to be downloaded

Print the list of standard libraries to be downloaded:
make -C julia-wsl/stdlib/ print-STDLIBS_EXT

Pkg Statistics LibCURL Downloads ArgTools Tar NetworkOptions SuiteSparse SparseArrays SHA

  • The version number (commit) is specified by the corresponding stdlib\*.version file
  • Download cache path: stdlib\srccache
  • Installation path: julia-wsl\stdlib

JLL binary dependencies to be downloaded

Print all JLL packages:
make -C julia-wsl/stdlib/ print-JLL_NAMES

MozillaCACerts_jll dSFMT_jll GMP_jll LibCURL_jll LibGit2_jll libLLVM_jll LibSSH2_jll LibUV_jll MbedTLS_jll MPFR_jll nghttp2_jll libblastrampoline_jll OpenBLAS_jll OpenLibm_jll p7zip_jll PCRE2_jll SuiteSparse_jll Zlib_jll LLVMLibUnwind_jll CompilerSupportLibraries_jll LibUnwind_jll
  • The corresponding thin wrapper is already located in the stdlib\ folder
  • The version number is specified by the corresponding stdlib\*_jll\Project.toml
  • Download and unpack and install it to julia-wsl\usr\

make -C julia-wsl/ julia-base

Equivalent to make -C julia-wsl/base/

A number of symbolic links are created, generating a number of .jl files containing constants.

make -C julia-wsl/ julia-cli-release

Equivalent to make -C julia-wsl/cli/ release

Build cli and loader.

make -C julia-wsl/ julia-src-release

Equivalent to make -C julia-wsl/src/ release

Build the core of julia.

Folder build order:

  • julia/src
  • julia/src/support
  • julia/src/flisp

make -C julia-wsl/ julia-sysimg-release

Warn: I don’t know much about this step, so I may not be describing it accurately.

Execute sysimage.mk to build the system image.
Compile the Base+stdlib .jl file into a binary.

This step also generates the REPL pre-compilation statements and executes them.

make -C julia-wsl/ julia-base-cache

Write the sys source cache in format readable by Base._read_dependency_src
—— etc\write_base_cache.jl

final

You can run make -C julia-wsl/ -j4. It will build two more test targets.
After that you can execute . /julia to open the julia REPL.

4 Likes
4

You can decide whether to use BB pre-built binary dependencies by specifying the following flag:

e.g.: Compile dsfmt from source code

echo 'USE_BINARYBUILDER_DSFMT := 0' > julia-wsl/Make.user

Some of julia’s dependencies have additional patches in deps\patches.
Pre-built dependencies in Yggdrasil will also have the same patches.

1 Like
5

Thank you @woclass , that looks amazing. I am going to study all this and might come with more questions. But that is more than anyone has done so far. Did you consider making a PR to document all this? I think that would be great for all the people trying to understand how Julia is built. I’ll mark your answer as solution for now even if I did not go through all the details you provided yet. THANK YOU again!

1 Like
6

@woclass , how were you able to figure out these steps? I can see that these are in the Makefile in the root directory but there are a lot of other steps as well:

  • julia-symlink
  • julia-libccalltest
  • etc…

Is there a main procedure which is called when one runs make?
I am asking because there is possibility in the future that these steps change and it would be nice to habe a way to find out what are the new steps if they change.
Thank you

7

make == make default
This may be a feature of make.

Dynamic debugging
I compile julia a lot and have contributed to the makefile in deps.
So, I had a rough idea that the build process could be divided into several stages.
I chose to add the -n flag and debug the make process directly, which prints out the current command to be executed but does nothing.
Then you can observe the behaviour of the makefile.

You could search for “Debugging Makefile” to find some flags and methods for debugging a makefile.

Reading Code
I mentioned above that make default is the first target, and then you can look at its dependencies, and the dependencies of the dependencies.
This will eventually form a tree. (I guess there are ready-made tools to visualize the tree)

Combining the two methods above, we can identify the relationship between the targets.
There are a lot of targets in the makefile, and you need to pick out some important points from them, so it’s up to you to make your own choices.
I chose the above targets to divide the compilation phase, and ignored the targets you listed because they are simple.

There are also some targets that will not be in the make default dependencies.

  • docs
  • check-whitespace
  • win-extras
  • …

I think this might be a good place to start.
We can filter out the targets not covered by make default and add comments to them one by one in place.

Perhaps we could also organize a small table and put it at the beginning of the makefile, or in a separate devdoc.

1 Like
8

Not before, because people usually don’t care about the compilation process, or just want to know how to compile.

Yes, these makefiles are not part of the API, and they change quickly, so it is difficult to maintain a development document over time, except by adding some short comments in place.

I’d be happy to answer those questions.
My main concern at the moment is the compilation and testing process of julia on windows, as well as the CI build.

If there are more in-depth questions that I may not be able to answer, you can go to slack’s #build-system, #ci-dev channel for help.

1 Like