Quarto and Julia – how to set the python version to use?

New to Julia
1

I am not new to Julia, but I struggle every time I have to use Python, since I feel that I have so many versions of Python installed (Anaconda, Conda, Conda.jl), that I am not sure when which version is called.

If I just follow the Quarto installation instructions (Quarto installed with Homebrew)

and have a minimal file like

# Hi

```{julia}
println("Hi")
```

as test.qmd

calling (in the same folder) quarto render test.qmd yields

Starting julia-1.8 kernel...Traceback (most recent call last):
  File "/Applications/quarto/share/jupyter/jupyter.py", line 21, in <module>
    from notebook import notebook_execute, RestartKernel
  File "/Applications/quarto/share/jupyter/notebook.py", line 17, in <module>
    from nbclient import NotebookClient
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/nbclient/__init__.py", line 6, in <module>
    from .client import NotebookClient, execute  # noqa: F401
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/nbclient/client.py", line 14, in <module>
    from jupyter_client import KernelManager
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/jupyter_client/__init__.py", line 6, in <module>
    from .asynchronous import AsyncKernelClient  # noqa
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/jupyter_client/asynchronous/__init__.py", line 1, in <module>
    from .client import AsyncKernelClient  # noqa
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/jupyter_client/asynchronous/client.py", line 6, in <module>
    from jupyter_client.channels import HBChannel
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/jupyter_client/channels.py", line 12, in <module>
    import zmq.asyncio
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
    from zmq import backend
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/__init__.py", line 32, in <module>
    raise original_error from None
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/__init__.py", line 27, in <module>
    _ns = select_backend(first)
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/select.py", line 32, in select_backend
    mod = import_module(name)
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import (
ImportError: dlopen(/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/cython/constants.cpython-38-darwin.so, 0x0002): tried: '/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/cython/constants.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/cython/constants.cpython-38-darwin.so' (no such file), '/Users/ronnber/.julia/conda/3/lib/python3.8/site-packages/zmq/backend/cython/constants.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

sadly, I usually fight so much getting Python stuff to run at all, that I am a little lost where to start checking what is going wrong here. Any tipps would be helpful. It seems it’s an x86 Python and I need an M1-one on my Mac? How would I tell that to Conda.jl?

2 Likes
2

apparently there are quite some issues with quarto and julia > 1.8.3

I couldnt get quarto to work either, in the end what I ended up doing was removing IJulia, start a julia session with v1.8.3 and reinstall and rebuild IJulia again.

Your error looks different than what I got before though. I can quarto render your MWE with my hacked setup. What julia version do you use?

5 Likes
3

I am always on a very recent one, currently 1.8.4. – The problem you mention seems not to be the problem, I think, but I will check the next days with 1.7 maybe.

4

I would use the Conda package for Julia to check that you have the jupyter package installed.

import Conda
Conda.add("jupyter")
Conda.update()
Conda.clean(; sources=false);

I believe that quarto just needs the environment variable QUARTO_PYTHON set to, in your case, "Users/ronnber/.julia/conda/3/bin/python3" but, as you say, frequently there are so many different versions of Python and Conda and pip floating around that things get confused.

5 Likes
5

Cool, that did the trick! I would not have guessed that from the strange error message that mentioned more like things with different architectures.

6

I have a followup problem which might even bar the same. Me fighting Python versions.

Now quarto just always starts a python kernel though I specified Julia, I think it might be because it switched (but I have no clue how or why) to some other Python version (I feel I have about a dozen installed since every small software brings its own Python), but I can not figure out how to tell quarto to use the Conda.jl one?

edit: Deleting both packages and quarto – and reinstalling them again fixed the issue.

7

Is there a best order to re-install? Quarto first, then the packages, or the reverse?

8

I don’t know about best. Mine now was

  1. Quarto
  2. Conda
  3. IJulia
1 Like
9

Thanks!

10

In a discussion with the Quarto folks they suggested setting the environment variable QUARTO_PYTHON to point to the executable, such as ~/.julia/conda/3/bin/python3. The jupyterCapabilities function defined in bin/quarto.js from the Quarto 1.2.313 installation explicitly looks for the python in ~/.julia/conda/3/bin/ so I am not sure if this is needed now.

What version of Quarto do you have installed?

11

ah, that might also have been a reason, I do set the QUARTO_PYTHON as you suggested, but it still only worked after the new install.

I installed with brew ( quarto — Homebrew Formulae ) so it should be 1.2.313.

12

Hi,

I’ve alternatively been using JuliaCall R package. Here are some demos. The experience in Rstudio is nice, and in VSCode is good. What I’ve found is that is lighter and works directly without setting explicitly an engine in the YAML header. It works directly from the engine: knitr. Maybe there might be some improvements later.

1 Like
13

…but then I have to install R?

14

The problem for me with JuliaCall is that I am less confident in JuliaCall than in Jupyter/IJulia as an evaluation engine.

Because Julia is a very rich language and the ccall interface can deal with more-or-less any C API, it is possible to mimic the R data types and evaluation process in Julia, which is what the Julia package RCall does. That package is 100% Julia code and wraps the entire C API for R. All of the conversions to/from R from/to Julia are performed in Julia code.

JuliaCall, on the other hand, has to go through C++ code called from R to start and communicate with the Julia process and, in fact, relies on the RCall package for Julia to do much of the conversion of data types.

To me using RStudio and JuliaCall introduces a lot more moving parts in the whole assembly and increases the chance of things failing. I would expect evaluation through Jupyter/IJulia to be more robust.

2 Likes
15

Sure… But no need of conda, Conda.jl and IJulia.jl on the other hand.

16

Oh I see, I didn’t know all the internals

17

Although the original problem seems to be solved now, I am also experiencing an issue that is best described by the title of this topic. Therefore after some hesitation I decided not to open a new topic and instead describe my issue here. But it could possibly be moved to a separate topic.

In particular, I am hitting my head against the wall with how to tell Quarto within vscode to use the right version of Python (on macOS). As a new user of macOS (having switched from Linux), I am unable to get out from the mess of multiple installations of Python on my system. In particular, when clicking on the Render button in vscode for some demo Julia-based qmd file, this is what I get in the terminal within vscode:

quarto preview ~/tmp/demo.qmd --no-browser --no-watch-inputs
ERROR: Jupyter kernel 'julia-1.8' not found.

When typing quarto check in the (same) vscode terminal, I get

➜  tmp quarto check

[✓] Checking Quarto installation......OK
      Version: 1.1.251
      Path: /Applications/quarto/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.11.1
      Path: /opt/homebrew/opt/python@3.11/bin/python3.11
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

Now, checking version of Python installed on my computer through homebrew:

➜  ~ ls -al /opt/homebrew/opt | grep python
lrwxr-xr-x    1 hurak  admin    23  3 led 19:04 ipython -> ../Cellar/python/8.8.0
lrwxr-xr-x    1 hurak  admin    23  3 led 19:04 ipython@8 -> ../Cellar/ipython/8.8.0
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python3 -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python@3 -> ../Cellarpython@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python@3.10 -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28  4 led 17:08 python@3.11 -> ../Cellar/python@3.11/3.11.1

Obviously, python3 is symlinked to the version 3.10.9. But when calling Quarto from within vscode, a different version 3.11.1 is called instead.

If I now call quarto outside of vscode in a separate terminal, everything is fine:

Last login: Tue Jan 17 11:19:00 on ttys004
➜  ~ quarto check

[✓] Checking Quarto installation......OK
      Version: 1.1.251
      Path: /Applications/quarto/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.10.9
      Path: /opt/homebrew/opt/python@3.10/bin/python3.10
      Jupyter: 5.1.2
      Kernels: julia-1.8

      NOTE: No Jupyter kernel for Python found

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

Typing quarto preview demo.qmd finds and starts the julia-1.8 kernel without troubles.

I am unable to identify if this is an homebrew issue or vscode issue or Quarto issue.

18

Which Python version do you have in mind to use? The one from Conda.jl? Then see the QUARTO_PATH remark above :slight_smile: That helped me (though my quarto by default picked not the home-brew but my anaconda version).

19

I actually wanted to use what is already installed through homebrew (why duplicating Python installations though Conda?). And it appears that setting QUARTO_PYTHON was really the way to go.

Sort of. Now if I click Render button in vscode to produce a preview, it does the job.

But the strange thing is that in the vscode-tied Quarto terminal I get a warning that what I specified through QUARTO_PYTHON

export QUARTO_PYTHON=/opt/homebrew/opt/python3

does not exist:

(|) Checking Python 3 installation....WARNING: Specified QUARTO_PYTHON '/opt/homebrew/opt/python3' does not exist.
[✓] Checking Python 3 installation....OK
      Version: 3.10.9
      Path: /opt/homebrew/opt/python@3.10/bin/python3.10
      Jupyter: 5.1.2
      Kernels: julia-1.8

      NOTE: No Jupyter kernel for Python found

while it certainly does exist

➜  ~ ls -al /opt/homebrew/opt | grep python
lrwxr-xr-x    1 hurak  admin    23  3 led 19:04 ipython -> ../Cellar/ipython/8.8.0
lrwxr-xr-x    1 hurak  admin    23  3 led 19:04 ipython@8 -> ../Cellar/ipython/8.8.0
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python3 -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python@3 -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28 14 led 12:30 python@3.10 -> ../Cellar/python@3.10/3.10.9
lrwxr-xr-x    1 hurak  admin    28  4 led 17:08 python@3.11 -> ../Cellar/python@3.11/3.11.1

It appears that if QUARTO_PYTHON is now specified, then if for some reason the system cannot use it, it falls back to the system default

➜  ~ python3 --version
Python 3.10.9

which is however exactly what I attempted to specify through QUARTO_PATH

➜  ~ which python3
/opt/homebrew/bin/python3

To conclude, I can certainly pursue to the Conda path, I before resorting to installing yet another Python in this way, I just tried to direct Quarto to use what is already installed on the system through Homebrew. With several versions of Python installed through Homebrew, I struggled to tell Quarto to use the one that is symlinked to python3. While setting QUARTO_PATH the system complains that it cannot find the specified thing, still Quarto now picks the system symlinked version of Python. I can easily leave it at this because functionality is now achieved (even if understanding is still unachieved:-)

20

I think you have a typo in your

export QUARTO_PYTHON=/opt/homebrew/opt/python3

which should be

export QUARTO_PYTHON=/opt/homebrew/bin/python3

as your which indicates (note the change of the second opt to bin). And sure these tiny differences are sometimes hard to spot :wink:

1 Like