Compile Julia from source with ease in VS Code (even on Windows!)

We just merged a devcontainer.json configuration into the Julia repo. This small change should make it much easier to compile and run Julia from source, in particular for folks that run on Windows or users that only occasionally want to compile Julia and don’t have their whole environment set up to work on Julia itself.

There are (minimally) three new scenarios enabled by this: 1) you can easily connect with a local VS Code installation to a remote development environment that runs on Azure in the cloud and has everything configured to compile Julia, 2) when you open the Julia repo in a local VS Code instance, it can auto-provision a local docker environment where everything is configured so that you can compile Julia easily, and 3) you can even work on and compile Julia entirely in your browser via VS Online. I’ll describe each of these in a bit more detail now.

Using a VS Online environment with a local VS Code installation to compile and run Julia

This is a good setup if you don’t want to install docker on your local machine, have an online connection but still want to use your local VS Code installation to work on Julia.

You start by signing up for Visual Studio Online. In terms of software on your local machine, you need to install VS Code and the Visual Studio Online extension for VS Code.

In VS Code you then select Create new environment...

image
If will first aks you what configuration you want to use. Just pick the default one. The next prompt will ask you to enter a repository you want to work on:

image
Just enter julialang/julia here for the Julia github repo. Finally it will ask you to enter a name for the new environment. This is the name inside Visual Studio Online for this new environment. You can just use the suggested default julia.

Now you need to wait a little bit until your new development environment in the cloud has been created. The good news is that the environment that is being created in the cloud will already be configured with all the software you need to compile and run Julia from source! No more need to read through build instructions on what prerequisites you need to install, it will all be pre-installed for you automatically.

Once the environment is created, VS Code will offer you an option to connect to this new environment:

image
Just click on Connect. At this point your local VS Code is connected to a remote development environment that runs in the cloud. Here is how it will look:


The key thing to note here is that none of these files are on your local machine, everything is happening in the cloud! You can tell that you are connected to a remote VS Online environment by looking at the green area in the lower left corner of the window: it says VS Online: julia, indicating that you are connected to that specific remote environment.

Next we will open up a terminal:

image
This opens a terminal in the lower half of VS Code:


This terminal is showing a shell that is running in the remote development environment, not your local system! In the screenshot above I already executed ls to show that I can just look at the directory content in my remote development environment.

Now we are ready to compile Julia from source. All we have to do is run make in the terminal. All necessary software to compile Julia is already around, so we don’t need to do anything else.

Once the build has finished, we can run this version of Julia in our development environment with ./julia in the terminal:


A couple of random observations:

  • The environment that is created will automatically have useful VS Code extensions installed, i.e. you will automatically get the Julia and C++ extension when you work in this environment.
  • You can at any point “suspend” your online environment, and then pick up work on it at some later point in time.
  • You can change the configuration of your environment even after you created it. Need more CPU or RAM? Just change the settings.

Using a local docker environment to build Julia

This is an excellent setup for those of us who already have docker installed on our systems and want to use the kind of isolation provided by docker to work on Julia. I think this is also an especially simple way for Windows users to compile Julia from source.

You need the following software installed locally on your system: VS Code, docker and the Remote - Containers VS Code extension.

Make sure to increase the default docker limits a bit, for example I had to increase the memory limit from 2GB to 4GB in the docker settings for the following to work.

The easiest option to get started is to select the New container command in VS Code:

In the following screen, select the option Open Repository in Container...:

In the next screen you should enter the repo name for the Julia repo:

You’ll then be asked about a volumn and name for the container, you can just select the defaults there.

At this point VS Code is creating a new docker environment for you that has everything that is needed to compile Julia in it. So, some waiting is in order :slight_smile:

Once that is finished, you are dropped into a VS Code window that is connected to a local docker instance:

You can tell that you are working in a docker environment now by looking at the green area in the lower left corner of the VS Code window.

From here on you would follow the same steps to compile and run Julia that you used in the previous section: you open a terminal (which will run inside the docker environment), you execute make, and once that is finished you can just start the version of Julia you just compiled with ./julia:

Here is a twist on how you can start the docker scenario: assume for a second that you cloned the Julia repo to a local folder on your machine, say ~\source\julia on a Windows system. Now you open that local folder in VS Code. VS Code will detect that the Julia repo now has configuration information for a docker image and will offer to open this folder in a container:

If you click Reopen in Container, VS Code will create a docker environment for you and automatically connect VS Code to that environment. There is one important difference to the previous docker scenario: VS Code will now mount your local folder into the docker environment, so when you make any edits to the Julia files in the docker environment, those changes are automatically reflect in the folder ~\source\julia on your local system.

Compiling and working on Julia from a web browser

There is now also an option to do everything in a web browser: editing Julia code, compiling it and running it! No need to install anything on your local machine.

To get started, surf in your favorite web browser to https://online.visualstudio.com/.

You should see something like this:

image
When you click on Create environmentyou will see a form where you can enter some details about the environment you want to create:


Make sure you fill in julialang/julia for the git repository value, and then click Create.

Now you need to wait a little while until your development environment is created. When your environment is read, you will see this screen:


Click on the Connect button, and you will be taken into a VS Code instance that is running entirely in the browser:


At this point you can essentially follow the instructions from the first section on how to compile Julia: open a terminal and run make inside that terminal in this VS Code instance. Once Julia has been compiled in the cloud, you can run it in the terminal:

Any environment you create in VS Online can be accessed either via VS Code (as described in the first section) or via the web browser (as described in this section), i.e. the only difference between the scenario described in the first and third section in this post is the choice of front-end UI (VS Code vs web browser), the underlying dev environment infrastructure is the same.

Summary

Please report back whether these options work for you or if you have any other tricks on how to make this kind of setup useful!

25 Likes

Awesome!

Now with the description of the docker workflow.

I didn’t see it explicitly stated but will this create a Linux or Windows build of Julia? Personally, when I work on Windows it is to test something that is Windows specific.

1 Like

It will create a Linux environment and then a Linux build.

My understanding is that we can’t cross-compile a Windows build from a Linux environment these days, right? Would certainly be neat if we could get that to work.

It is not clear to me whether one can create a Windows container with docker. If that is an option, one could presumably also create a docker image that has all the windows build tools that are needed to compile Julia.

1 Like

Yes my experience last month when I tried the mingw cross compilation from linux to windows (following the instructions included in doc/build/windows.md) was a lot of obscure error messages related to the build chain. I got further with XC_HOST = i686-w64-mingw32 than with XC_HOST = x86_64-w64-mingw32, but neither worked out.

I only recently read this post by @davidanthoff and I’m really enthusiastic about the possibility to work in the cloud via my local VSCode. Thanks @davidanthoff for your work on this!

There are two things that are still unclear to me. First, when I start Julia in the shell, I’m on version 1.6, but when I then execute code in a script from within VSCode, I get a prompt that, according to versioninfo(), runs version 1.4.2. Is there a way to make VSCode use version 1.6 as well? Should I, for instance, change the path in the extension settings? And then change it back when I want to use VSCode to work on my own machine?

Second, I have a paid Azure account, but I can get at most 8 cores in Codespaces. Is that a known limitation? Temporarily? Am I overlooking something?

Right now we don’t have an option to use a different Julia version for the LS and say the REPL. We have plans to statically compile the LS, at which this kind of scenario should become much easier. Right now you can just generally switch the Julia version you want to use, and that should also work on codespaces.

I think the 8 cores might be a limit that is there…