PackageCompiler.jl Portability Issue

I’m having an issue with PackageCompiler.jl, the error I’m getting on the sister PC is:

julia -JCrashSimPackages.dll
fatal: error thrown and no exception handler available.
InitError(mod=:Libiconv_jll, error=ErrorException("Artifact "Libiconv" was not installed correctly. Try `using Pkg; Pkg.instantiate()` to re-install all missing resources."))
error at .\error.jl:33
_artifact_str at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Artifacts\src\Artifacts.jl:551
jfptr__artifact_str_73397 at K:\DevOps\_git\EmbeddedJuliaREPL\bin\x64\Debug\Julia\bin\CrashSimPackages.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1703 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:672
jl_f__apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:686 [inlined]
jl_f__apply_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:722
#invokelatest#2 at .\essentials.jl:707
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1703 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:672
invokelatest at .\essentials.jl:706
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1703 [inlined]
do_call at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:115
eval_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:204
eval_stmt_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:155 [inlined]
eval_body at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:575
jl_fptr_interpret_call at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:649
macro expansion at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Artifacts\src\Artifacts.jl:680 [inlined]
find_artifact_dir at C:\Users\CarmanBr\.julia\packages\JLLWrappers\bkwIo\src\wrapper_generators.jl:15
__init__ at C:\Users\CarmanBr\.julia\packages\Libiconv_jll\tGWLj\src\wrappers\x86_64-w64-mingw32.jl:8
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1703 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:74
_julia_init at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:797
repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:693
mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:51
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)

As can be seen, the complaint is about Libiconv_jll. The issue is this package is not in my manifest! Here is the status from my project:

julia> Pkg.status(mode=PKGMODE_MANIFEST)
      Status `D:\Work\IST\Projects\Packages\Manifest.toml`
  [f03e0fa6] AM2App v0.3.0
  [621f4979] AbstractFFTs v1.0.1
  [79e6a3ab] Adapt v3.3.1
  [13072b0f] AxisAlgorithms v1.0.0
  [a74b3585] Blosc v0.7.0
  [336ed68f] CSV v0.8.5
  [43e84be3] CatView v0.3.3
  [86aefda3] CatapultFunctions v0.3.7
  [324d7699] CategoricalArrays v0.10.0
  [d360d2e6] ChainRulesCore v0.10.11
  [35d6a980] ColorSchemes v3.13.0
  [3da002f7] ColorTypes v0.11.0
  [5ae59095] Colors v0.12.8
  [34da2185] Compat v3.31.0
  [a81c6b42] Compose v0.9.2
  [d38c429a] Contour v0.5.7
  [7ad07ef1] CoupledFields v0.2.0
  [a8cc5b0e] Crayons v4.0.4
  [717857b8] DSP v0.7.0
  [9a962f9c] DataAPI v1.7.0
  [a93c6f00] DataFrames v1.2.0
  [864edb3b] DataStructures v0.18.9
  [e2d170a0] DataValueInterfaces v1.0.0
  [b4f34e82] Distances v0.10.3
  [31c24e10] Distributions v0.24.18
  [ffbed154] DocStringExtensions v0.8.5
  [e2ba6199] ExprTools v0.1.6
  [7a1cc6ca] FFTW v1.4.1
  [5789e2e9] FileIO v1.10.1
  [1a297f60] FillArrays v0.11.9
  [53c48c17] FixedPointNumbers v0.8.4
  [59287772] Formatting v0.4.2
  [c91e804a] Gadfly v1.3.3
  [42e2da0e] Grisu v1.0.2
  [f67ccb44] HDF5 v0.15.5
  [a1b4810d] Hexagons v0.2.0
  [9b13fd28] IndirectArrays v0.5.1
  [a98d9a8b] Interpolations v0.13.3
  [d8418881] Intervals v1.5.0
  [41ab1584] InvertedIndices v1.0.0
  [c8e1da08] IterTools v1.3.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [4138dd39] JLD v0.12.3
  [692b3bcd] JLLWrappers v1.3.0
  [682c06a0] JSON v0.21.1
  [e5e0dc1b] Juno v0.8.4
  [5ab0869b] KernelDensity v0.6.3
  [4345ca2d] Loess v0.5.3
  [2ab3a3ac] LogExpFunctions v0.2.5
  [1914dd2f] MacroTools v0.5.6
  [442fdcdd] Measures v0.3.1
  [e89f7d12] Media v0.5.0
  [e1d29d7a] Missings v1.0.0
  [78c3b35d] Mocking v0.7.1
  [d8a4904e] MutableArithmetics v0.2.20
  [6fe1bfb0] OffsetArrays v1.10.2
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.1
  [9b87118b] PackageCompiler v1.2.6
  [69de0a69] Parsers v1.1.0
  [f27b6e38] Polynomials v2.0.14
  [2dfb63ee] PooledArrays v1.2.1
  [21216c6a] Preferences v1.2.2
  [08abe8d2] PrettyTables v1.1.0
  [1fd47b50] QuadGK v2.4.1
  [c84ed2f1] Ratios v0.4.0
  [3cdcf5f2] RecipesBase v1.1.1
  [189a3867] Reexport v1.1.0
  [ae029012] Requires v1.1.3
  [79098fc4] Rmath v0.7.0
  [91c51154] SentinelArrays v1.3.3
  [992d4aef] Showoff v1.0.3
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v1.5.1
  [90137ffa] StaticArrays v1.2.6
  [82ae8749] StatsAPI v1.0.0
  [2913bbd2] StatsBase v0.33.8
  [4c63d2b9] StatsFuns v0.9.8
  [856f2bd8] StructTypes v1.7.2
  [46d4f523] Structures v0.1.3
  [f2d4222d] SystemData v0.1.26
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.4.4
  [f269a46b] TimeZones v1.5.6
  [b8865327] UnicodePlots v1.3.0
  [efce3f68] WoodburyMatrices v0.5.3
  [0b7ba130] Blosc_jll v1.21.0+0
  [f5851436] FFTW_jll v3.3.9+7
  [0234f1f7] HDF5_jll v1.12.0+1
  [1d5cc7b8] IntelOpenMP_jll v2018.0.3+2
  [5ced341a] Lz4_jll v1.9.3+0
  [856f044c] MKL_jll v2021.1.1+1
  [458c3c95] OpenSSL_jll v1.1.10+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [3161d3a3] Zstd_jll v1.5.0+0
  [0dad84c5] ArgTools
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions
  [44cfe95a] Pkg
  [de0858da] Printf
  [9abbd945] Profile
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML
  [a4e569a6] Tar
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll
  [deac9b47] LibCURL_jll
  [29816b5a] LibSSH2_jll
  [c8ffd9c3] MbedTLS_jll
  [14a3606d] MozillaCACerts_jll
  [83775a58] Zlib_jll
  [8e850ede] nghttp2_jll

So where does Libiconv_jll come from?

Additionally, and possibly the real issue, when I update my package using Pkg.update(“AM2App”), this makes a lot of changes to the Manifest.toml file. But, there is a known issue with FFTW, which must be constrained to v1.4.1 (the current version 1.4.3 has a portability issue) and FFTW_jll which must be constrained to v3.3.9+7. I have added a constraint to FFTW so that it remains at v1.4.1, but the FFTW_jll will still update to v3.3.9+8. The format of the version number doesn’t seem to work with the Project.toml [compat] section, so I can’t constrain the FFTW_jll package. Therefore, after the Manifest.toml file updates to v3.3.9+8, I manually copy back the FFTW_jll section to Manifest.toml

[[FFTW_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "5a0d4b6a22a34d17d53543bd124f4b08ed78e8b0"
uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a"
version = "3.3.9+7"

After saving Manifest.toml, I run Pkg.instantiate() and then run PackageCompiler.create_sysimage()

Am I possibly breaking something by doing this? The compiled sys image from PackageCompiler.jl does work fine on my PC, but when I try to use on a sister PC it fails. Note, it did work before I made any changes to the Manifest file, in which case the EzXML → XML2_jll → Libiconv_jll chain was removed from my Manifest because of an update to TimeZones.

Finally, is there anyway to only update the specified package and leave the rest of the Manifest alone? TimeZones is a dependency of a dependency which I certainly did not need to update. In my case I’d like to make the smallest changes to Manifest possible when I update a specific package (i.e. the one I need for my sys image generation).

1 Like

I have somewhat the same issue with HDF5… I thought that packages using the Artifact system should be portable as long as include_transitive_dependencies = true, but it does not seem to be the case.

I thought that this option was not working recursively at the beginning but it’s adverstised as such so …

@Brad_Carman I am also having the same issue, but for me it happens while deploying to Google Cloud Build. Did you manage to fix this issue ? Would appreciate any suggestions, thanks!!

For portability I suggest using the create_app() function. You can read this thread which discusses how I sorted thru the issue with building and testing a portable system image: InitError and artifact folder · Issue #544 · JuliaLang/PackageCompiler.jl (github.com)