Enable multiple Cores for Jupyter Lab

I have found some resources on this online, but nothing which I actually is able to understand fully.

I want to enable multiple cores when I run JupyterLab (also notebooks).

This link (Setting JULIA_NUM_THREADS in an IJulia kernel) tells be that I am supposed to set
ENV["JULIA_NUM_THREADS"] = 4
However, where and how? This seems to be an environmental variable. I can read about these here Environment Variables · The Julia Language. However, either that page is not supposed to tell me what I need to know, or it fails to do so. Finally it seems like the juliarc.jl file might be interesting.

However, I am still not sure where to find it and how to change it (and a confirmation whenever this is the one I am looking for). This link seems to contain something (Update docs with new startup.jl file and location · Issue #26215 · JuliaLang/julia · GitHub), however, my .julia folder does not contain a config folder.
(I tried simply creating one, but a juliarc.jl file in it with the single line ENV["JULIA_NUM_THREADS"] = 4, did not work).

Finally, this link gives me more info (Where is juliarc (the Julia startup file)? - Stack Overflow). I can find the startup.jl file in the install folder (for julia-1.0.0, will this change when I update to e.g. julia-1.0.3. If so, will I have to redo stuff?). The startup.jl file looks like this:

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.
import Pkg

Is it this I should make changes to, or the non-existing “local” file in .julia/config?

Am I right that

  1. There is some file, somewhere, to which I must make a change to enable multiple cores in Juyter (preferable without specifying something every time I start Jupyter). If so, where is this file?
  2. In that file I should add a line ENV["JULIA_NUM_THREADS"] = 4 (possibly replacing 4 with another integer, depending on how many cores I want to use).

Thanks

Just open a REPL and type:

ENV["JULIA_NUM_THREADS"] = 4
using IJulia
notebook()

This will open a Jupyter session with 4 threads. Or if you want to open Jupyter with jupyter notebook in the terminal or Git bash, then use:

export JULIA_NUM_THREADS=4
jupyter notebook
3 Likes

Thanks.

Is there a way to “permanent-ise” this, so that I do not have to type that every time I open jupyter?

Set an environment variable in your OS.

2 Likes

There are a few ways:

(1) A simple way to do this is to put export JULIA_NUM_THREADS=4 in your ~/.bashrc or whatever the shell configuration you need to use.

(2) Another way to do this is to put ENV["JULIA_NUM_THREADS"] = 4 in ~/.julia/config/startup.jl. If ~/.julia/config does not exist, just try create one. However, note that:

Some variables, such as JULIA_NUM_THREADS and JULIA_PROJECT, need to be set before Julia starts, therefore adding these to ~/.julia/config/startup.jl is too late in the startup process.
Environment Variables · The Julia Language

For this reason, I don’t recommend this method. However, if you invoke IJulia.notebook() from standard REPL then ENV["JULIA_NUM_THREADS"] can still take an effect for the Julia processes inside Jupyter notebook, but not the Julia process invoking IJulia.notebook().

(3) If you want to set JULIA_NUM_THREADS only for Julia process inside Jupyter notebook, you can edit the kernel spec ~/.local/share/jupyter/kernels/julia-1.0/kernel.json to something like

{
  "display_name": "Julia 1.0.0",
  "argv": [
    "env",  # add this
    "JULIA_NUM_THREADS=4",  # add this
    "/PATH/TO/bin/julia",
    "-i",
    "--startup-file=yes",
    "--color=yes",
    "/home/USERNAME/.julia/packages/IJulia/0cLgR/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia"
}

It’s handy (say) if you want to add Julia kernel with different number of threads. For example, create ~/.local/share/jupyter/kernels/julia-1.0-one-thread/kernel.json with

{
  "display_name": "Julia 1.0.0 (one thread)",
  "argv": [
    "env",
    "JULIA_NUM_THREADS=1",
    "/PATH/TO/bin/julia",
    ...

Actual path to kernel.json depends on OS. See also Making kernels for Jupyter — jupyter_client 8.6.0 documentation

3 Likes

You can just create that variable on your system.
I did it on Windows 10 (environment variable) and it works.

Thanks for all the help everyone. This is useful.

For future reference, writing export JULIA_NUM_THREADS=4 into the terminal also seems to work.

I have used this one

using IJulia
installkernel("Julia (n threads)", env=Dict("JULIA_NUM_THREADS"=>"n"))

n is the number of threads you want to use, I m using 8 you can increase it to whatever number you want

10 Likes

correct, sorry for that, thank you, once again, pointing us to a superquick solution !
:hugs:

Not clear this is the right place for that setting.

Needs to be in the env dict within kernel.json:

"env": {
    "JULIA_NUM_THREADS": "4"
  },

Otherwise, Julia hangs when trying to open the kernel.