Relocatable app workflow with PackageCompiler

General Usage
, , ,
1

Thanks to the help of @oheil, @mkitti, and @kristoffer.carlsson, I have compiled a julia app using PackageCompiler and PkgTemplate that accepts a file path as a command line argument and prints it out as a dataframe.

Here is my workflow:

$ julia
$ using PkgTemplates
$ t = Template()
$ t(“demoApp”)
$ ] activate <path listed in the previous command >
$ ] add DataFrames, CSV, 
$ cd(“<path listed above for package>”)
$ Using demoApp

- Add Artifacts.toml file to project directory. PkgTemplates does not include this, but it is necessary.

- Add source code to the directory at ~/.julia/dev/demoApp/src/demoApp.jl. 

$ create_app("../demoApp","demoAppCompiled";force=true)

TO RUN:

$ ~/.julia/dev/demoApp/demoAppCompiled/bin/demoApp test.txt

2×3 DataFrame
 Row │ a      b      c
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      2      3
   2 │     4      5      6

I grabbed the Artifacts.toml file from here:
https://github.com/JuliaLang/PackageCompiler.jl/blob/master/examples/MyApp/Artifacts.toml

My source code for the app looks like this:

module demoApp

using DataFrames, CSV

using Pkg.Artifacts

fooifier_path() = joinpath(artifact"fooifier", "bin", "fooifier" * (Sys.iswindows() ? ".exe" : ""))

function julia_main()
    try
        real_main()
    catch
        Base.invokelatest(Base.display_error, Base.catch_stack())
        return 1
    end
    return 0
end

function real_main()
    df = CSV.read(string(ARGS[1]), DataFrame);
    println(df)
    return
end

if abspath(PROGRAM_FILE) == @__FILE__
    real_main()
end

end

It compiles properly and I get a directory demoAppCompiled/ with the appropriate contents. I then run demoAppCompiled/bin/demoApp test.txt on my machine and it works perfectly.

Now, I’ve zipped demoAppCompiled/, moved the zipped object to a new location on my machine, unzipped, and tried to run the same command. It breaks with this error:

(base) connor@connors-MacBook-Pro:~/Downloads/rppApp$ rppgAppCompiled/bin/rppgApp workingTest.csv 
fatal: error thrown and no exception handler available.
InitError(mod=:OpenSpecFun_jll, error=ErrorException("could not load library "/Users/connor/Downloads/rppApp/rppgAppCompiled/artifacts/083301655942999198605bee77f89f1d41bf820e/lib/libopenspecfun.1.4.dylib"
dlopen(/Users/connor/Downloads/rppApp/rppgAppCompiled/artifacts/083301655942999198605bee77f89f1d41bf820e/lib/libopenspecfun.1.4.dylib, 1): no suitable image found.  Did find:
	/Users/connor/Downloads/rppApp/rppgAppCompiled/artifacts/083301655942999198605bee77f89f1d41bf820e/lib/libopenspecfun.1.4.dylib: code signature in (/Users/connor/Downloads/rppApp/rppgAppCompiled/artifacts/083301655942999198605bee77f89f1d41bf820e/lib/libopenspecfun.1.4.dylib) not valid for use in process using Library Validation: library load disallowed by system policy"))
jl_errorf at /usr/local/opt/julia/lib/julia/libjulia-internal.dylib (unknown line)
jl_load_dynamic_library at /usr/local/opt/julia/lib/julia/libjulia-internal.dylib (unknown line)
YY.dlopenYY.3 at /Users/connor/Downloads/rppApp/rppgAppCompiled/bin/rppgApp.dylib (unknown line)
dlopen at ./libdl.jl:114
unknown function (ip: 0x109301971)
jl_apply_generic at /usr/local/opt/julia/lib/julia/libjulia-internal.dylib (unknown line)
__init__ at /Users/connor/Downloads/rppApp/rppgAppCompiled/bin/rppgApp.dylib (unknown line)
jl_apply_generic at /usr/local/opt/julia/lib/julia/libjulia-internal.dylib (unknown line)
jl_module_run_initializer at /usr/local/opt/julia/lib/julia/libjulia-internal.dylib (unknown line)
_julia_init at /usr/local/opt/julia/lib/julia/libjulia-internal.dylib (unknown line)
main at /Users/connor/Downloads/rppApp/rppgAppCompiled/bin/rppgApp (unknown line)

I see there are some nice docs describing that my problem is that my app is not relocatable. The docs are here:
https://julialang.github.io/PackageCompiler.jl/dev/devdocs/relocatable_part_3/

However, it is unclear to me how to incorporate the suggestions in these into my workflow. They recommend using gcc and what not, but is there anyway to make my app work at any location and ultimately, on any machine, without leaving PackageCompiler?

Thank you.

2

I have never done this, but my guess is, that it’s not about PackageCompiler but about Artifacts location:
http://pkgdocs.julialang.org/v1/artifacts/#Overriding-artifact-locations

There are some examples with absolute paths, perhaps one can specify a relative path here, too. A relative path to the library would be relocatable.

But really, I am only guessing here, perhaps the others have experience with it and know how to do it.

3

The error is about code signing and is specific to macOS.

If you google “not valid for use in process using Library Validation: library load disallowed by system policy”, you will find several work arounds ranging from disabling library validation to code signing.

Here are some links:

The Hardened Runtime enables library validation by default. This security-hardening feature prevents a program from loading frameworks, plug-ins, or libraries unless they’re either signed by Apple or signed with the same Team ID as the main executable. The macOS dynamic linker ( dyld ) provides a detailed error message when the system prevents code from loading due to library validation. Use the Disable Library Validation Entitlement if your program loads plug-ins that are signed by other third-party developers.

Perhaps @giordano and @staticfloat know something more about how the artifacts are signed and how you might able to create a compatible signature.

1 Like
4

Thanks. So you’re thinking this is a specific Mac issue and not necessarily related to the relocatable app documentation on PackageCompiler?

Theoretically, if I compile on a different OS with the same process, then it should work across platforms then?

5

As far as I know, the system image produced by PackageCompiler.jl is not cross-platform. The system image consists of code native to each platform. You cannot compile the system image on Linux or Windows and have it work on macOS.

1 Like
6

Gotcha, I guess I didn’t have a proper understanding of PackageCompiler. So if I want it to run on multiple platforms, I should compile separately on multiple platforms?

Regarding the original problem, it sounds like its just a Mac issue. So if I repeat the compilation steps on a linux machine, there in theory should be any problems?

1 Like