Jupyter and Jupytext through IJulia.jl (Jupytext does not really work)

Tooling Jupyter-Notebook
, , , ,
1

Is anyone using IJulia to provide jupyter-lab and employing the jupytext extension? I was unable to configure my Arch Linux do to so. I did a fresh install of Julia 1.10.4, then installed the IJulia in an environment, and used the Conda dependency of IJulia (which it uses to install jupyter) to then install jupytext. It worked to a degree, because the context menu shows the jupytext options but then it errors if I try to use (right-click on jupytext notebook file) > Open With > Jupytext Notebook, the error message is:

UnreadableNotebook: .../file.jl
NotJSONError("Notebook does not appear to be JSON: '# ---\\n# jupyter:\\n# jupytext:\\n# ...")

I have tried debugging and the jupytext extension is not being really loaded:

[I 2024-06-27 22:02:03.170 LabApp] JupyterLab extension loaded from /home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyterlab
[I 2024-06-27 22:02:03.170 LabApp] JupyterLab application directory is /home/henrique/.julia/conda/3/x86_64/share/jupyter/lab
[I 2024-06-27 22:02:03.170 LabApp] Extension Manager is 'pypi'.
[I 2024-06-27 22:02:03.180 ServerApp] jupyterlab | extension was successfully loaded.
[W 2024-06-27 22:02:03.181 ServerApp] [Jupytext Server Extension] Async contents managers like AsyncLargeFileManager are not supported at the moment (https://github.com/mwouts/jupytext/issues/1020). We will derive a contents manager from LargeFileManager instead.
[I 2024-06-27 22:02:03.181 ServerApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager
[W 2024-06-27 22:02:03.181 ServerApp] jupyterlab_jupytext | extension failed loading with message: NameError("name 'build_jupytext_contents_manager_class' is not defined")
    Traceback (most recent call last):
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 360, in load_extension
        extension.load_all_points(self.serverapp)
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 232, in load_all_points
        return [self.load_point(point_name, serverapp) for point_name in self.extension_points]
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 232, in <listcomp>
        return [self.load_point(point_name, serverapp) for point_name in self.extension_points]
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 223, in load_point
        return point.load(serverapp)
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 151, in load
        return loader(serverapp)
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyterlab_jupytext/__init__.py", line 43, in load_jupyter_server_extension
        app.contents_manager_class = build_jupytext_contents_manager_class(base_class)
    NameError: name 'build_jupytext_contents_manager_class' is not defined. Did you mean: 'build_jupytext_contents_manager'?

If I go to that file I see at the start:

try:
    from jupytext.contentsmanager import build_jupytext_contents_manager_class
except ImportError as err:
    build_jupytext_contents_manager = reraise(err)

[...]

So the name should exist, unless it throws an exception. Which it does. If I remove the try-except, I get:

[W 2024-06-27 22:01:09.027 ServerApp] jupyterlab_jupytext | error adding extension (enabled: True): The module 'jupyterlab_jupytext' could not be found (No module named 'notebook'). Are you sure the extension is installed?
    Traceback (most recent call last):
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 323, in add_extension
        extpkg = ExtensionPackage(name=extension_name, enabled=enabled)
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 187, in __init__
        self._load_metadata()
      File "/home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyter_server/extension/manager.py", line 202, in _load_metadata
        raise ExtensionModuleNotFound(msg) from None
    jupyter_server.extension.utils.ExtensionModuleNotFound: The module 'jupyterlab_jupytext' could not be found (No module named 'notebook'). Are you sure the extension is installed?

However the files exists in /home/henrique/.julia/conda/3/x86_64/lib/python3.10/site-packages/jupyterlab_jupytext/ as well as a jupytext package and they define the build_jupytext_contents_manager_class it is just that it is not being found really.

So I think my question is, really, how I guarantee that this jupyter installation inside .julia/conda finds the correct Python packages.

2

(Note that you don’t need to use jupyter installed via Conda.jl. You can install jupyter via Arch Linux’s pacman, or in some other way. It will still find the IJulia kernel.)

3

The problem with this alternative is that jupyter has an official Arch package but jupytext only has one in the AUR (unofficial), and I believe it fails to build. This is the reason I decided to do the things this way.

4

jupyter / jupyter-lab / jupyter-lab (howto)

Installing jupyter by means of IJulia seems like a trap. If I do it, I am not able to make jupytext work with jupyter-notebook or jupyter-lab. So what worked for me follows.

  • In any folder you can run:
    • sudo aura -A miniconda3 # aura is my AUR package installer, use yours instead
    • sudo ln -s /opt/miniconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh
    • echo "[ -f /opt/miniconda3/etc/profile.d/conda.sh ] && source /opt/miniconda3/etc/profile.d/conda.sh" >> ~/.bashrc # I use bash, if you use zsh adapt
  • Reopen the terminal, so it updates the variables, and then run in any folder:
    • conda create -n myenv jupyter jupytext # you can give a better name than myenv
    • conda activate myenv # not sure if the next steps need conda activated, but we will keep it for the rest of the steps
  • The IJulia is still needed to create the Julia Kernel. If you install it locally to an environment (like I do, I never install packages to the global Julia environment), you will need to put the path to the environment in the kernel specification (and basically use one kernel per project).
  • So in the folder that you will use as environment for the Julia notebooks do:
    • julia --project=. -e 'using Pkg: Pkg; Pkg.add("IJulia");' # will not ask to install jupyter this way, if it asks then refuse
    • julia --project=. -e 'using IJulia: IJulia; IJulia.installkernel("Julia My Kernel Name", "-O2", "--project=$(pwd())", env=Dict("LD_LIBRARY_PATH" => ""))' # the LD_LIBRARY_PATH=“” comes from an annoying bug of Julia in Arch Linux
  • And then finally:
    • jupyter-lab
5

I wrote a book using jupytext a couple of years ago, and while it wasn’t all rainbows and kittens, it was pretty easy to use. But I installed and ran jupytext independently outside of Julia from the start. Sometimes I found it easier to copy and edit kernel definition files directly than to install kernels “officially.”