Weird PackageCompiler problem with parallel builds

General Usage
1

Hello,

we recently tried to update from PackageCompiler v0.6.0 to v0.6.4 and we’re hitting a weird problem. Because it takes 3-5 minutes to build a binary and we have half a dozen binaries to build, we fire off a bunch of calls to PackageCompiler in parallel (each with its own build directory). With v0.6.0, this worked fine. With v0.6.4, all the compilations succeed, but the resulting binaries seem to all be for only one of the original source files.

The issue seems to be related to firing off all the builds at the same time. If I run them in sequence, everything works as expected. For example consider the test programs below:

$ cat /tmp/other.jl 
Base.@ccallable function julia_main(ARGS::Vector{String})::Cint
    println("DARA other")
    return 0
end
$ cat /tmp/test.jl 
Base.@ccallable function julia_main(ARGS::Vector{String})::Cint
    println("DARA test")
    return 0
end

If I build as follows, they both produce the same output. Does anybody have an idea what might have changed? I’ve been going through the PackageCompiler changes, but nothing pops out. Note that this issue happens independent of Julia version (also see it on 1.1.0).

julia /test/julia-1.3.0-rc3/pkg/dev/PackageCompiler/juliac.jl -vae -c -t -d /tmp/build_other /tmp/other.jl &
julia /test/julia-1.3.0-rc3/pkg/dev/PackageCompiler/juliac.jl -vae -c -t -d /tmp/build_test /tmp/test.jl &
wait

Then run the binaries

$ /tmp/build_other/other
DARA other
$ /tmp/build_test/test
DARA other
$ strings /tmp/build_test/test.so | grep 'DARA test'
$
2

Found the problem. PackageCompiler is saving the source wrapper file to one global location. As a result the later compile command clobbers that file and all executables are compiled from the same source. Opened a bug to track this (https://github.com/JuliaLang/PackageCompiler.jl/issues/282) - workaround is to use a local temp directory for each build, copy the file to the global location afterwards.

1 Like