Julia, quarto and CSV fails precompile

General Usage
1

Hi All

I was experimenting with quarto and ran into a strange problem with CSV. I am on the latest Julia on a M1, with all latest packages. Running Julia by itself, no problems with using CSV but with quarto precompile fails, with errors below. All other packages I tried work.

any idea?

Thanks, Jack

code is qmd file (for quarto)

 ---

 ---

 ```{julia}
 using Statistics;  
 ```.

```{julia}
using CSV
```.

and what I get is

quarto render xx.qmd

Executing 'xx.ipynb'
  Cell 1/2...Done
  Cell 2/2...ERROR:

An error occurred while executing the following cell:
------------------
using CSV
------------------

Failed to precompile CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b] to /Users/jack/.julia/compiled/v1.8/CSV/jl_YGglru.

Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
LoadError: Failed to precompile CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b] to /Users/jack/.julia/compiled/v1.8/CSV/jl_YGglru.
2

When the compile cache errors I usually just remove either only the compilation cache for just the affected package or all packages, it’s in .julia/compiled or so. That seems to happen rarely when I exit out of precompilation early or Julia crashes during that phase.

1 Like
3

Thanks,

I did already try that and it still fails.

4

What version of quarto and what version of Julia? Run

quarto check

from the shell to get the version of quarto and

julia> versioninfo()

to get the version of Julia.

Do you have a local Project.toml file? Quarto uses conversion to a Jupyter notebook, as you can see in your code snippet, where it was running code chunks of xx.ipynb when the error was encountered. Is CSV available in the local project?

1 Like
5

Thanks

maybe the solution is to nuke .julia.

I have that below, but also made a new jupyter file and it gives even more errors on using CSV. For comparison using Statistics works. It seems like jupyter is using some old dependencies, and julia repl not,

I no local Project.toml file.

I add the Jupyter error dump below.

 quarto check

[✓] Checking Quarto installation......OK
      Version: 1.2.335
      Path: /Applications/quarto/bin

and in julia

using CSV
[ Info: Precompiling CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 8 × Apple M2
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 4 virtual cores

from Jupyter using CSV

[ Info: Precompiling CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
ERROR: LoadError: ArgumentError: Package Tables does not have OrderedCollections in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have Tables checked out for development and have
  added OrderedCollections as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with Tables
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1167 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] include
   @ ./Base.jl:419 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1554
 [6] top-level scope
   @ stdin:1
in expression starting at /Users/jack/.julia/packages/Tables/AcRIE/src/Tables.jl:1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Tables [bd369af6-aec1-5ad0-b16a-f7cc5008161c] to /Users/jack/.julia/compiled/v1.8/Tables/jl_IdeU7T.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1707
  [3] compilecache
    @ ./loading.jl:1651 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] include
    @ ./Base.jl:419 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1554
 [11] top-level scope
    @ stdin:1
in expression starting at /Users/jack/.julia/packages/CSV/b8ebJ/src/CSV.jl:1
in expression starting at stdin:1
Failed to precompile CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b] to /Users/jack/.julia/compiled/v1.8/CSV/jl_Kq1idG.

Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144

in juyter but not in jula repl

Pkg.instantiate()
 Warning: The active manifest file has dependencies that were resolved with a different julia version (1.7.0-beta3). Unexpected behavior may occur.
└ @ ~/Manifest.toml:0
6

I would probably create a local project and add CSV in the package prompt

(1.8) pkg> activate .
(localdir) pkg> add CSV

then try rendering the notebook. The Julia package system is designed to make it easy to create and preserve a local environment for projects and it seems worthwhile using it for that.

I have, in the past, nuked my ~/.julia directory to reset the whole system, and can understand the desire to do that. If you do choose to do that remember you will need to install IJulia to be able to use Julia code chunks in quarto.

2 Likes
7

Nuking .julia does not help. Some deeper problem with julia and jupyter that manifests itself with using CSV. Will try on a linux, hopefully only a bug on the M1 and not intel linux.

jupyter says

ERROR: LoadError: ArgumentError: Package Tables does not have OrderedCollections in its dependencies:
- You may have a partially installed environment.

REPL is fine.

8

I would suspect an error related to a Parsers version mismatch - I recall something like this used to happen with IJulia, maybe Quarto also has its own Parsers dependency?

1 Like
9

I took quarto out of the equation, now only trying jupyter. And nuked .julia, so would not expect a Parsers mismatch?

10

Ah sorry I didn’t properly pay attention then.

So what you are saying is you cannot do:

julia> using IJulia

julia> notebook()

## ... Jupyter browser tab opens, start a new notebook
# In notebook cells
] activate --temp
] add CSV
using CSV

?

1 Like
11

Not exactly, I did in the terminal

jupyter notebook

and there selected the Julia kernel and then did using CSV which is what fails.

When I followed your steps, I ran into another issue, in that is

julia> using IJulia
julia> notebook()
install Jupyter via Conda, y/n? [y]:

as jupyter is installed already (with pip and not conda) and i can call it in the terminal

12

Ah right so your issues seem to come from using your own python/anaconda version rather than using the miniconda which IJulia would install. That means it’ll probably be hard for others to repro given that the issue will be specific to your setup.

1 Like
13

Is that it? Julia needs to have its own python setup?

That creates its own problems. I am using quarto (which is based on jupyter), and what you say makes julia incompatible with quarto. You suggest I have to invoke jupyter inside julia, and cannot have jupyter invoke julia, which is what quarto (and jupyer books, and other jupyter utilities) need.

14

No, that’s not what I’m saying. Julia doesn’t require its own python installation, but by default it does use its own miniconda so most people will have that setup installed. IJulia (and I guess other tools using python) are capable of using a user supplied python installation.

My point was merely that if there are issues with Julia interacting with your local python installation it might be hard for others to repro as they might not be able to recreate your python setup easily.

1 Like
15

Thanks, I get it now.

I will try on linux on intel and see if that works. And if not, do down the rabbit hole of older versions until it works.

And on a tangent, python and its versions and breaking updates are a pain. If I get this working, I’ll put it inside a docker and use that for the foreseeable future to process our julia with quarto.

Thanks for all the help,

best jack

1 Like
16

You can select the version of python that is to be used by quarto by setting the environment variable QUARTO_PYTHON. For example, if you have configured the Julia package PyCall to install its own miniconda then

export QUARTO_PYTHON=~/.julia/conda/3/bin/python3

will get you that version.

I agree that having multiple conflicting installations of Python, Conda and/or Pip on a system is annoying but it seems almost inevitable.

2 Likes
17

Of course

makes perfect sense that that way,

Yes, Having to deal with virtual environments and multiple versions is a pain we escape in R and Julia.

Thanks all for the help

Jack