Testing Julia 1.10.5: Circular dependency detected warning

Just installed Julia 1.10.5, and I am getting new warnings:

┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│   CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
│   FilePathsBaseTestExt [36f6b4e4-024d-52bd-a01f-148eb20c09de]
│   FilePathsBaseMmapExt [968c5e7e-8fc3-5f0f-93d2-8a5e0e125e4f]
└ @ Pkg.API ~/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:1279

Steps to reproduce:

mkdir test
cd test
git clone https://github.com/ufechner7/KiteModels.jl.git
cd KiteModels.jl/bin
./create_sys_image --update

Any idea?

Possible related to Remove as much dependencies as possible by longemen3000 · Pull Request #172 · rofinn/FilePathsBase.jl · GitHub

From my experience creating package extensions on the standard library packages can easily create many strange issues when not done very carefully, at least in the current version of Julia. See also: Don't make Test a dependency by ChrisRackauckas · Pull Request #164 · JuliaObjects/Accessors.jl · GitHub

Related discussion:

Yes, it’s the same issue as Extensions: pre-compilation fails if more than one extension · Issue #52511 · JuliaLang/julia · GitHub reported last year – the warning message became less scary, but it’s still present.

And what is the way forward to fix this issue?

I think reverting Remove as much dependencies as possible by longemen3000 · Pull Request #172 · rofinn/FilePathsBase.jl · GitHub
and adding the changes back more carefully is the way forward.

There is some very fishy stuff going on in FilePathsBase.jl/ext/FilePathsBaseTestExt.jl at master · rofinn/FilePathsBase.jl · GitHub and FilePathsBase.jl/src/test_stub.jl at master · rofinn/FilePathsBase.jl · GitHub

Generally, any extension needing stubs like this indicates a big issue.

2 Likes

Do you think there is also a problem with the CSV package, or not so much?

CSV has implicit imports with a using FilePathsBase which should probably be made explicit using tools like GitHub - ericphanson/ExplicitImports.jl: Developer tooling for Julia namespace management: detecting implicit imports, improper explicit imports, and improper qualified accesses but I don’t think that is directly related to this issue.

Another issue is in KiteModels.jl/bin/create_sys_image at 48464497de7752b3e3dc55ffd3557019a27f6950 · ufechner7/KiteModels.jl · GitHub

This is adding a package to the global package environment, which you probably want to avoid doing in a build script.

You should try and run julia with:

JULIA_LOAD_PATH="@:@stdlib" julia --startup-file=no --project

In your script to avoid having compatibility issues with the global environment, or packages being used in the startup file.

No, this is what I want. I want that the users of my package to have TestEnv in the global environment because it is needed for the examples. If I have it in the project PackageEval will always fail because it is only available for released Julia versions.

I agree that this is just a workaround for the problem that Pkg has no command to activate a sub-environment, and the only sub-environment you can define in Project.toml is the test environment. I assume that when workspaces are supported (Julia 1.12+) this workaround will not be needed any longer.

Why is it needed for the examples? Why not have all the examples run in a self-contained environment with all the dependencies they need to work properly?

Also, with your current script I am getting a different error:

mkdir test
cd test
git clone https://github.com/ufechner7/KiteModels.jl.git
cd KiteModels.jl/bin
./create_sys_image --update
Precompiling project...
  23 dependencies successfully precompiled in 10 seconds. 324 already precompiled.
[ Info: Loading packages ...
[ Info: Creating sysimage ...
[ Info: PackageCompiler: Executing /home/nathan/Downloads/test/KiteModels.jl/test/test_for_precompile.jl => /tmp/jl_packagecompiler_phS3rT/jl_VauxTl
ERROR: LoadError: InitError: PyError (PyImport_ImportModule

The Python package matplotlib could not be imported by pyimport. Usually this means
that you did not install matplotlib in the Python version being used by PyCall.

PyCall is currently configured to use the Python version at:

/usr/bin/python3

and you should use whatever mechanism you usually use (apt-get, pip, conda,
etcetera) to install the Python package containing the matplotlib module.

One alternative is to re-configure PyCall to use a different Python
version on your system: set ENV["PYTHON"] to the path/name of the python
executable you want to use, run Pkg.build("PyCall"), and re-launch Julia.

Another alternative is to configure PyCall to use a Julia-specific Python
distribution via the Conda.jl package (which installs a private Anaconda
Python distribution), which has the advantage that packages can be installed
and kept up-to-date via Julia.  As explained in the PyCall documentation,
set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then,
To install the matplotlib module, you can use `pyimport_conda("matplotlib", PKG)`,
where PKG is the Anaconda package that contains the module matplotlib,
or alternatively you can use the Conda package directly (via
`using Conda` followed by `Conda.add` etcetera).

) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'matplotlib'")

Stacktrace:
  [1] pyimport(name::String)
    @ PyCall ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:558
  [2] pyimport_conda(modulename::String, condapkg::String, channel::String)
    @ PyCall ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:716
  [3] pyimport_conda
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:715 [inlined]
  [4] __init__()
    @ PyPlot ~/.julia/packages/PyPlot/rWSdf/src/init.jl:174
  [5] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1134
  [6] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1122
  [7] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::Nothing, depmods::Vector{Any})
    @ Base ./loading.jl:1067
  [8] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, ocachepath::Nothing, sourcepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1481
  [9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1574
 [10] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1938
 [11] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [12] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [13] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [14] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [15] macro expansion
    @ ./loading.jl:1790 [inlined]
 [16] macro expansion
    @ ./lock.jl:267 [inlined]
 [17] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [18] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [19] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [20] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [21] top-level scope
    @ ~/Downloads/test/KiteModels.jl/test/test_for_precompile.jl:12
during initialization of module PyPlot
in expression starting at /home/nathan/Downloads/test/KiteModels.jl/test/test_for_precompile.jl:1
ERROR: LoadError: failed process: Process(setenv(`/home/nathan/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/bin/julia --color=yes --startup-file=no --pkgimages=no --sysimage=/home/nathan/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/lib/julia/sys.so --compile=all --trace-compile=/tmp/jl_packagecompiler_phS3rT/jl_VauxTl test/test_for_precompile.jl`,[…]), ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run
    @ ./process.jl:477 [inlined]
  [4] run_precompilation_script(project::String, sysimg::String, precompile_file::String, precompile_dir::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:306
  [5] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, extra_precompiles::String, incremental::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:344
  [6] create_sysimg_object_file
    @ ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:311 [inlined]
  [7] create_sysimage(packages::Vector{Symbol}; sysimage_path::String, project::String, precompile_execution_file::String, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing, julia_init_h_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:628
  [8] top-level scope
    @ ~/Downloads/test/KiteModels.jl/test/create_sys_image.jl:14
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [10] top-level scope
    @ none:1
in expression starting at /home/nathan/Downloads/test/KiteModels.jl/test/create_sys_image.jl:14

I added the detection/ installation of matplotlib and added --startup-file=no

Does it work for you now?

1 Like

Yes, that works for me now, and I am getting the same Circular dependency warning you got.

1 Like

Can I somehow ask why a dependency is circular? Something like Pkg.why that explains how the circle appears. This would help on eliminating them.

Not for extensions you can use GitHub - KristofferC/EnvironmentGraph.jl

1 Like

Created a bug report: Circular dependency warning · Issue #179 · rofinn/FilePathsBase.jl · GitHub

This issue is really annoying because I get these warnings each time I run one of my examples.

This issue was connected to the fact that I precompiled the system image with the parameter “include_transitive_dependencies=false”. Changing that to “include_transitive_dependencies=true” made the warning go away.