I use two Linux servers. Although the two servers are not exactly identical, I can still compile Julia v1.6.x in one server and copy the usr/ to another server, and Julia works fine in both machines (I do it because Julia compiling is only succeeded on one server). However, when I compile Julia v1.7.2 in one machine and copied it to another, although I can still startup Julia without error, I cannot use or add any packages, even the internal packages:
julia> using Printf
ERROR: ArgumentError: Package Printf not found in current path:
- Run `import Pkg; Pkg.add("Printf")` to install the Printf package.
Stacktrace:
[1] require(into::Module, mod::Symbol)
@ Base ./loading.jl:967
julia> import Pkg
ERROR: ArgumentError: Package Pkg not found in current path:
- Run `import Pkg; Pkg.add("Pkg")` to install the Pkg package.
Stacktrace:
[1] require(into::Module, mod::Symbol)
@ Base ./loading.jl:967
(v1.7) pkg> add Pkg
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
ERROR: The following package names could not be resolved:
* Pkg (44cfe95a-1eb2-52ea-b672-e2afdf69b78f in manifest but not in project)
julia> LOAD_PATH
3-element Vector{String}:
"@"
"@v#.#"
"@stdlib"
julia> versioninfo()
Julia Version 1.7.2
Commit bf53498* (2022-02-06 15:21 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E7-4880 v2 @ 2.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-12.0.1 (ORCJIT, ivybridge)
Here is my Make.user:
CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0
MARCH=x86-64
JULIA_TARGET_CPU=x86-64
Are there any tricks I can do to make Julia v1.7 distributable as the previous v1.6 version?
I don’t think this is related to your issue, but why this? It’s a bit uncommon to use the C++03 string ABI nowadays.
Check the folder that Base.load_path_expand("@stdlib") shows. readdir on that path should show you something like:
julia> readdir(Base.load_path_expand("@stdlib"))
58-element Vector{String}:
"ArgTools"
"Artifacts"
"Base64"
⋮
"libblastrampoline_jll"
"nghttp2_jll"
"p7zip_jll"
If not, then probably something went wrong in your install and/or copying.
Indeed I’m not clear what’s its meaning but just copied the whole Make.user from somewhere when I compiled Julia 1.6, and it works. I tried removing this sentence and didn’t find anything changes for the compiling. Maybe it should be removed as you were concerned.
Thanks. I tried your suggestion. It seems rather normal, I think:
julia> readdir(Base.load_path_expand("@stdlib"))
58-element Vector{String}:
"ArgTools"
"Artifacts"
"Base64"
"CRC32c"
"CompilerSupportLibraries_jll"
"Dates"
"DelimitedFiles"
"Distributed"
"Downloads"
"FileWatching"
"Future"
"GMP_jll"
"InteractiveUtils"
"LLVMLibUnwind_jll"
"LazyArtifacts"
"LibCURL"
"LibCURL_jll"
"LibGit2"
"LibGit2_jll"
"LibSSH2_jll"
"LibUV_jll"
"LibUnwind_jll"
"Libdl"
"LinearAlgebra"
"Logging"
"MPFR_jll"
"Markdown"
"MbedTLS_jll"
"Mmap"
"MozillaCACerts_jll"
"NetworkOptions"
"OpenBLAS_jll"
"OpenLibm_jll"
"PCRE2_jll"
"Pkg"
"Printf"
"Profile"
"REPL"
"Random"
"SHA"
"Serialization"
"SharedArrays"
"Sockets"
"SparseArrays"
"Statistics"
"SuiteSparse"
"SuiteSparse_jll"
"TOML"
"Tar"
"Test"
"UUIDs"
"Unicode"
"Zlib_jll"
"dSFMT_jll"
"libLLVM_jll"
"libblastrampoline_jll"
"nghttp2_jll"
"p7zip_jll"
julia> using Pkg
ERROR: ArgumentError: Package Pkg not found in current path:
- Run `import Pkg; Pkg.add("Pkg")` to install the Pkg package.
Stacktrace:
[1] require(into::Module, mod::Symbol)
@ Base ./loading.jl:967
And you verified that that path is contained in the load path? I.e.
Base.load_path_expand("@stdlib") in Base.load_path()
returns true?
Indeed, the load path is contained:
julia> Base.load_path_expand("@stdlib") in Base.load_path()
true
Does e.g. the Pkg entrypoints exist?
julia> isfile(joinpath(Base.load_path_expand("@stdlib"), "Pkg", "Project.toml"))
true
julia> isfile(joinpath(Base.load_path_expand("@stdlib"), "Pkg", "src", "Pkg.jl"))
true
1 Like
Thanks to your suggestion, I found the problem. The folders under usr/share/julia/stdlib/v1.7 are all softlinks. They became invalid after being moved to another server. After I replace them with the real folders from stdlib/, the problem is solved. Now the Julia packages work fine on both servers. Many thanks for your help!
It is because the folders under usr/share/julia/stdlib/v1.7 are all symbolic links. They became invalid after being moved to another server. (Thanks @fredrikekre 's suggestion, checking with isfile(joinpath(Base.load_path_expand("@stdlib"), "Pkg", "Project.toml")) return false.) After I replaced them with the real folders from stdlib/, the problem is solved. Maybe I should distribute self-compiled julia using rsync -L to avoid invalid symbolic links.
2 Likes
If you run make binary-dist you will get a tarball that contains everything and you can copy and unpack that one instead.
2 Likes
Indeed binary-dist does not work for me. It ended with the below error. That’s why I directly moved the usr/ folder for distribution. FYI:
# Copy in all .jl sources as well
mkdir -p /data/projects/programs/julia1.0/julia/julia-5efa17c3db/share/julia/base /data/projects/programs/julia1.0/julia/julia-5efa17c3db/share/julia/test
cp -R -L /data/projects/programs/julia1.0/julia/base/* /data/projects/programs/julia1.0/julia/julia-5efa17c3db/share/julia/base
cp -R -L /data/projects/programs/julia1.0/julia/test/* /data/projects/programs/julia1.0/julia/julia-5efa17c3db/share/julia/test
cp -R -L /data/projects/programs/julia1.0/julia/usr/share/julia/* /data/projects/programs/julia1.0/julia/julia-5efa17c3db/share/julia
cp: cannot copy cyclic symbolic link '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.0/Pkg-aaf4e6e8ab784f7435c1bdc56832bc03e014dedb/Pkg-aaf4e6e8ab784f7435c1bdc56832bc03e014dedb'
cp: cannot copy cyclic symbolic link '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.0/Statistics-cde87c8062032883165cd242f4a5c6b7943cb0b1/Statistics-cde87c8062032883165cd242f4a5c6b7943cb0b1'
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.3/Statistics': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.3/SuiteSparse': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.1/Statistics': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.1/SuiteSparse': No such file or directory
cp: cannot copy cyclic symbolic link '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.5/Pkg/Pkg-aaf4e6e8ab784f7435c1bdc56832bc03e014dedb'
cp: cannot copy cyclic symbolic link '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.5/Statistics/Statistics-cde87c8062032883165cd242f4a5c6b7943cb0b1'
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.5/SuiteSparse': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.4/SuiteSparse': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.2/Statistics': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.2/SuiteSparse': No such file or directory
cp: cannot stat '/data/projects/programs/julia1.0/julia/usr/share/julia/stdlib/v1.6/SuiteSparse': No such file or directory
/data/projects/programs/julia1.0/julia/Makefile:233: recipe for target 'install' failed
make[1]: *** [install] Error 1
make[1]: Leaving directory '/data/projects/programs/julia1.0/julia'
Makefile:426: recipe for target 'binary-dist' failed
make: *** [binary-dist] Error 2
Can you try again on a fresh clone, or after running make cleanall? I suspect there are lots of files in there from previous builds (perhaps from older Julia versions, since there are many different version numbers in /stdlib).
make cleanall doesn’t solve the problem as I tried before, but compiling on a fresh julia clone is worth trying. I indeed git pulled and makeed on the same julia clone in several years for each version update. That might be the problem. Thanks for your suggestion!