Distributions does not play nicely with PackageCompilerX

General Usage
1

Using PackageCompilerX, I wish to create an app from a package that uses functions in Distributions. Unfortunately, create_app fails as shown below.

I’m not using much functionality from Distributions so I should be able to extract out what I need, and just looking at the code of Distributions has been a helpful exercise.

Is it worth reporting the fact that Distributions can’t be compiled? Perhaps that fact would be obvious to the package’s authors, given that I think it depends on Rmath, coded in C++ rather than Julia. I wonder if making packages compatible with PackageCompilerX is a priority for package authors, or even on their radar? I’d love for that to be so…

By the way, other packages I’m using (Dates, Dierckx, FileIO, Interpolations, JSON, LinearAlgebra, NamedArrays, NLsolve, Random, Serialization, Sobol, CSV, DataFrames, DataStructures) do seem to work with PackageCompilerX, though I haven’t yet tested for relocatability.

MWE

file MyExample/src/MyExample.jl

module MyExample
using Distributions

function julia_main()::Cint
    try
        test_if_quantile_works()
        return 0
    catch
        return 1
    end
end

function test_if_quantile_works()
    show(quantile.(Normal(0, 1), 0:0.1:1))
end

end # module

Using create_app from the REPL:

julia> MyExample.julia_main()
[-Inf, -1.2815515655446004, -0.8416212335729143, -0.5244005127080409, -0.2533471031357997, 0.0, 0.2533471031357997, 0.5244005127080407, 0.8416212335729143, 1.2815515655446004, Inf]0

julia> using PackageCompilerX
[ Info: Precompiling PackageCompilerX [dffaa6cc-da53-48e5-b007-4292dfcc27f1]

julia> create_app("MyExample","CompiledMyExample",force=true)
┌ Warning: Package Rmath has a build script, this might indicate that it is not relocatable
└ @ PackageCompilerX C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:449
[ Info: PackageCompilerX: creating base system image (incremental=false)...
[ Info: PackageCompilerX: creating system image object file, this might take a while...
[ Info: PackageCompilerX: creating system image object file, this might take a while...
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export BoBag: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export Rmath: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export Users: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export bin: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export ccalllib_C:: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export deps: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export packages: symbol not found
C:/Users/Philip/.julia/artifacts/572b61b5075459e3ed62317e674398166ca98dd4/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: Cannot export usr: symbol not found
collect2.exe: error: ld returned 1 exit status
ERROR: failed process: Process(`'C:\Users\Philip\.julia\artifacts\572b61b5075459e3ed62317e674398166ca98dd4\mingw64\bin\gcc.exe' -m64 -shared '-LC:\Users\philip\AppData\Local\Julia-1.3.1\bin' -o MyExample.dll -Wl,--whole-archive 'C:\Users\Philip\AppData\Local\Temp\jl_EDE5.tmp.o' -Wl,--no-whole-archive -ljulia -Wl,--export-all-symbols`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at .\process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] #8 at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:386 [inlined]
 [5] #1 at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:34 [inlined]
 [6] withenv(::PackageCompilerX.var"#1#2"{PackageCompilerX.var"#8#9"{Cmd}}, ::Pair{String,String}) at .\env.jl:161
 [7] windows_compiler_artifact_path(::PackageCompilerX.var"#8#9"{Cmd}, ::String) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:33
 [8] create_sysimg_from_object_file(::String, ::String) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:385
 [9] #create_sysimage#7(::String, ::String, ::Array{String,1}, ::Array{String,1}, ::Bool, ::Bool, ::Bool, ::String, ::String, ::Bool, ::typeof(create_sysimage), ::Symbol) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:355
 [10] (::PackageCompilerX.var"#kw##create_sysimage")(::NamedTuple{(:sysimage_path, :project, :incremental, :precompile_execution_file, :precompile_statements_file, :cpu_target, :base_sysimage, :isapp),Tuple{String,String,Bool,Array{String,1},Array{String,1},String,String,Bool}}, ::typeof(create_sysimage), ::Symbol) at .\none:0
 [11] (::PackageCompilerX.var"#11#12"{Array{String,1},Bool,Bool,String,String})() at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:546
 [12] cd(::PackageCompilerX.var"#11#12"{Array{String,1},Bool,Bool,String,String}, ::String) at .\file.jl:93
 [13] #create_app#10(::Array{String,1}, ::Array{String,1}, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(create_app), ::String, ::String) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:535
 [14] (::PackageCompilerX.var"#kw##create_app")(::NamedTuple{(:force,),Tuple{Bool}}, ::typeof(create_app), ::String, ::String) at .\none:0
 [15] top-level scope at REPL[15]:1
caused by [exception 1]
IOError: symlink: operation not permitted (EPERM)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] symlink(::String, ::String) at .\file.jl:851
 [3] probe_symlink_creation(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\PlatformEngines.jl:131
 [4] #unpack#95(::Bool, ::typeof(Pkg.PlatformEngines.unpack), ::String, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\PlatformEngines.jl:727
 [5] #unpack at .\logging.jl:0 [inlined]
 [6] #download_verify_unpack#100(::Nothing, ::Bool, ::Bool, ::Bool, ::typeof(Pkg.PlatformEngines.download_verify_unpack), ::String, ::String, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\PlatformEngines.jl:894
 [7] #download_verify_unpack at .\none:0 [inlined]
 [8] #download_artifact#38(::Bool, ::typeof(Pkg.Artifacts.download_artifact), ::Base.SHA1, ::String, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\Artifacts.jl:744
 [9] #download_artifact at .\none:0 [inlined]
 [10] #ensure_artifact_installed#42(::Pkg.BinaryPlatforms.Platform, ::Bool, ::typeof(Pkg.Artifacts.ensure_artifact_installed), ::String, ::Dict{String,Any}, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\Artifacts.jl:869
 [11] (::Pkg.Artifacts.var"#kw##ensure_artifact_installed")(::NamedTuple{(:platform,),Tuple{Pkg.BinaryPlatforms.Windows}}, ::typeof(Pkg.Artifacts.ensure_artifact_installed), ::String, ::Dict{String,Any}, ::String) at .\none:0
 [12] do_artifact_str(::String, ::Dict{String,Any}, ::String, ::Module) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Pkg\src\Artifacts.jl:983
 [13] #invokelatest#1 at .\essentials.jl:709 [inlined]
 [14] invokelatest at .\essentials.jl:708 [inlined]
 [15] get_compiler() at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:47
 [16] create_sysimg_from_object_file(::String, ::String) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:382
 [17] #create_sysimage#7(::String, ::String, ::Array{String,1}, ::Array{String,1}, ::Bool, ::Bool, ::Bool, ::String, ::Nothing, ::Bool, ::typeof(create_sysimage), ::Array{Symbol,1}) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:355
 [18] (::PackageCompilerX.var"#kw##create_sysimage")(::NamedTuple{(:sysimage_path, :project, :incremental, :filter_stdlibs, :cpu_target),Tuple{String,String,Bool,Bool,String}}, ::typeof(create_sysimage), ::Array{Symbol,1}) at .\none:0
 [19] (::PackageCompilerX.var"#11#12"{Array{String,1},Bool,Bool,String,String})() at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:542
 [20] cd(::PackageCompilerX.var"#11#12"{Array{String,1},Bool,Bool,String,String}, ::String) at .\file.jl:93
 [21] #create_app#10(::Array{String,1}, ::Array{String,1}, ::Bool, ::Bool, ::Bool, ::Bool, ::typeof(create_app), ::String, ::String) at C:\Users\Philip\.julia\packages\PackageCompilerX\IaYi5\src\PackageCompilerX.jl:535
 [22] (::PackageCompilerX.var"#kw##create_app")(::NamedTuple{(:force,),Tuple{Bool}}, ::typeof(create_app), ::String, ::String) at .\none:0
 [23] top-level scope at REPL[15]:1
2

Yes, we just identified this bug yesterday (in Use artifacts on windows to get the compiler by KristofferC · Pull Request #91 · JuliaLang/PackageCompilerX.jl · GitHub). This is actually a (likely) Julia bug `ccall()` with an absolute library path while outputting an `.o` file causes linker issues on Windows · Issue #34680 · JuliaLang/julia · GitHub.

However, this problem should be fixed (worked around) for Rmath via Use Rmath_jll in place of BinaryProvider by ararslan · Pull Request #68 · JuliaStats/Rmath.jl · GitHub. So you can try add Rmath#master in the Pkg REPL the get the newest Rmath and try again. Also, when Rmath releases a new version, it should just work automatically.

1 Like
3

Thanks Kristoffer, add Rmath#master did the trick.

1 Like