Enable multiple Cores for Jupyeter Lab

jupyter

#1

I have found some resources on this online, but nothing which I actually is able to understand fully.

I want to enable multiple cores when I run JupyterLab (also notebooks).

This link (Setting JULIA_NUM_THREADS in an IJulia kernel) tells be that I am supposed to set
ENV["JULIA_NUM_THREADS"] = 4
However, where and how? This seems to be an environmental variable. I can read about these here https://docs.julialang.org/en/v1/manual/environment-variables/index.html#Parallelization-1. However, either that page is not supposed to tell me what I need to know, or it fails to do so. Finally it seems like the juliarc.jl file might be interesting.

However, I am still not sure where to find it and how to change it (and a confirmation whenever this is the one I am looking for). This link seems to contain something (https://github.com/JuliaLang/julia/issues/26215), however, my .julia folder does not contain a config folder.
(I tried simply creating one, but a juliarc.jl file in it with the single line ENV["JULIA_NUM_THREADS"] = 4, did not work).

Finally, this link gives me more info (https://stackoverflow.com/questions/52175314/where-is-juliarc-the-julia-startup-file). I can find the startup.jl file in the install folder (for julia-1.0.0, will this change when I update to e.g. julia-1.0.3. If so, will I have to redo stuff?). The startup.jl file looks like this:

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.
import Pkg

Is it this I should make changes to, or the non-existing “local” file in .julia/config?

Am I right that

  1. There is some file, somewhere, to which I must make a change to enable multiple cores in Juyter (preferable without specifying something every time I start Jupyter). If so, where is this file?
  2. In that file I should add a line ENV["JULIA_NUM_THREADS"] = 4 (possibly replacing 4 with another integer, depending on how many cores I want to use).

Thanks


#2

Just open a REPL and type:

ENV["JULIA_NUM_THREADS"] = 4
using IJulia
notebook()

This will open a Jupyter session with 4 threads. Or if you want to open Jupyter with jupyter notebook in the terminal or Git bash, then use:

export JULIA_NUM_THREADS=4
jupyter notebook

#3

Thanks.

Is there a way to “permanent-ise” this, so that I do not have to type that every time I open jupyter?


#4

Set an environment variable in your OS.


#5

There are a few ways:

(1) A simple way to do this is to put export JULIA_NUM_THREADS=4 in your ~/.bashrc or whatever the shell configuration you need to use.

(2) Another way to do this is to put ENV["JULIA_NUM_THREADS"] = 4 in ~/.julia/config/startup.jl. If ~/.julia/config does not exist, just try create one. However, note that:

Some variables, such as JULIA_NUM_THREADS and JULIA_PROJECT, need to be set before Julia starts, therefore adding these to ~/.julia/config/startup.jl is too late in the startup process.
https://docs.julialang.org/en/v1.2-dev/manual/environment-variables/

For this reason, I don’t recommend this method. However, if you invoke IJulia.notebook() from standard REPL then ENV["JULIA_NUM_THREADS"] can still take an effect for the Julia processes inside Jupyter notebook, but not the Julia process invoking IJulia.notebook().

(3) If you want to set JULIA_NUM_THREADS only for Julia process inside Jupyter notebook, you can edit the kernel spec ~/.local/share/jupyter/kernels/julia-1.0/kernel.json to something like

{
  "display_name": "Julia 1.0.0",
  "argv": [
    "env",  # add this
    "JULIA_NUM_THREADS=4",  # add this
    "/PATH/TO/bin/julia",
    "-i",
    "--startup-file=yes",
    "--color=yes",
    "/home/USERNAME/.julia/packages/IJulia/0cLgR/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia"
}

It’s handy (say) if you want to add Julia kernel with different number of threads. For example, create ~/.local/share/jupyter/kernels/julia-1.0-one-thread/kernel.json with

{
  "display_name": "Julia 1.0.0 (one thread)",
  "argv": [
    "env",
    "JULIA_NUM_THREADS=1",
    "/PATH/TO/bin/julia",
    ...

Actual path to kernel.json depends on OS. See also https://jupyter-client.readthedocs.io/en/latest/kernels.html#kernelspecs


#6

You can just create that variable on your system.
I did it on Windows 10 (environment variable) and it works.


#7

Thanks for all the help everyone. This is useful.