# Zygote.jl and PackageCompiler.jl

**URL:** <https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839>\
**Category:** General Usage\
**Tags:** package-compiler\
**Created:** [March 11, 2020, 12:41pm UTC](https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839 "2020-03-11T12:41:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Peter\_Adelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peter_adelman/32/11907_2.png) [@Peter\_Adelman](https://discourse.julialang.org/u/Peter_Adelman)\
**Post date:** [March 11, 2020, 12:41pm UTC](https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839/1 "2020-03-11T12:41:23Z")

</div>

I have been compiling my package for deployment using PackageCompilerX and now PackageCompiler, but recently I added some dependencies (including Zygote) and now it seems to dislike execution after compilation.

- Ubuntu 18.04
- gcc 7.5.0
- Julia 1.3.1
- PackageCompiler 1.0.2 and 1.1.0 tested
- Zygote 0.4.9

A small example that demonstrates this problem is:

1. `]generate test`
2. `]activate test`
3. `]add Zygote`
4. `]activate`
5. edit test.jl to be:

```julia
module test

using Zygote

Base.@ccallable function julia_main()::Cint
    try
        real_main()
    catch
        Base.invokelatest(Base.display_error, Base.catch_stack())
        return 1
    end
    return 0
end

function real_main()
    if ARGS[1] == "test"
	f(x) = 5*x + 3
	println(f(10))
	println(f'(10))
        println("TEST")
    end
end

end # module

```

1. `using PackageCompiler` (v1.02 or v1.1 tested)
2. `create_app("test","test_app")`
3. Get logs like this:

```julia
┌ Warning: NNlib has a dependency on Requires.jl, code in `@require` will not be run
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/D8oaG/src/PackageCompiler.jl:510
┌ Warning: Zygote has a dependency on Requires.jl, code in `@require` will not be run
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/D8oaG/src/PackageCompiler.jl:510
┌ Warning: Package NNlib has a build script, this might indicate that it is not relocatable
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/D8oaG/src/PackageCompiler.jl:523
┌ Warning: Package FFTW has a build script, this might indicate that it is not relocatable
└ @ PackageCompiler ~/.julia/packages/PackageCompiler/D8oaG/src/PackageCompiler.jl:523
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
[ Info: PackageCompiler: creating system image object file, this might take a while...

```

1. `exit()`
2. `test_app/bin/test test`
3. get stack trace like this:

```julia
fatal: error thrown and no exception handler available.
#<null>
rec_backtrace at /buildworker/worker/package_linux64/build/src/stackwalk.c:94
record_backtrace at /buildworker/worker/package_linux64/build/src/task.c:224 [inlined]
jl_throw at /buildworker/worker/package_linux64/build/src/task.c:461
require at ./loading.jl:900
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:399 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:436
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:656
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:873
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:878
include at ./boot.jl:328 [inlined]
include_relative at ./loading.jl:1105
include at ./Base.jl:31 [inlined]
include at /home/dev/.julia/packages/Zygote/ApBXe/src/Zygote.jl:1 [inlined]
precompile at /home/dev/.julia/packages/Zygote/ApBXe/src/Zygote.jl:46 [inlined]
#1827 at /home/dev/.julia/packages/Requires/qy6zC/src/init.jl:11
unknown function (ip: 0x7f9d6007f68c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2141 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
__init__ at /home/dev/.julia/packages/Requires/qy6zC/src/init.jl:18
jfptr ___init___ 15723 at /home/dev/projects/testcompiled/bin/test.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2135 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2305
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1631 [inlined]
jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:74
_julia_init at /buildworker/worker/package_linux64/build/src/init.c:821
main at ./test (unknown line)
__libc_start_main at /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
_start at ./test (unknown line)

```

Line 46 in [Zygote.jl](https://github.com/FluxML/Zygote.jl/blob/master/src/Zygote.jl) is:  
`precompile() = include(joinpath(@ __DIR__ , "precompile.jl"))`

This is kind of a generic “where is the error / how do I fix it?” question.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [March 11, 2020, 12:57pm UTC](https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839/2 "2020-03-11T12:57:54Z")

</div>

Packages that use certain dynamic features like including source files at runtime are pretty much impossible to compile into apps.

---

<div class="post-metadata">

**Author:** ![Peter\_Adelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peter_adelman/32/11907_2.png) [@Peter\_Adelman](https://discourse.julialang.org/u/Peter_Adelman)\
**Post date:** [March 11, 2020, 1:11pm UTC](https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839/3 "2020-03-11T13:11:51Z")

</div>

I guess my solution in this case is to branch Zygote.jl and set it to whichever version of its conditional load settings that I want?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [March 11, 2020, 1:18pm UTC](https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839/4 "2020-03-11T13:18:06Z")

</div>

That could maybe work, yeah.

For conditional dependencies to work well with PackageCompiler we need a conditional dependencies system that works together with the package precompilation system ([https://github.com/JuliaLang/Pkg.jl/issues/1285](https://github.com/JuliaLang/Pkg.jl/issues/1285)). It is not super easy to come up with a good design though.

---

<div class="post-metadata">

**Author:** ![Peter\_Adelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/peter_adelman/32/11907_2.png) [@Peter\_Adelman](https://discourse.julialang.org/u/Peter_Adelman)\
**Post date:** [March 11, 2020, 2:17pm UTC](https://discourse.julialang.org/t/zygote-jl-and-packagecompiler-jl/35839/5 "2020-03-11T14:17:24Z")

</div>

Yeah. I did it and it worked. Thanks

I see the complexity there. One potential solution would be a riff on Stefan’s proposal. We could just have `MyPkg/glue/A/` be a directory with its own environment definition (Project.toml and perhaps Manifest.toml). A could depend on MyPkg and whatever other packages it wants. When the compat/dependency requirements of the glue conditions were met, the package `glue/A` would be loaded. This would mean that:

1. glue/A could be precompiled independently, no?
2. every new package added to an environment would require a scan of existing, un-loaded glue packages in its Manifest.toml
3. we get to leverage exactly the existing dependency system
4. The definition of a “Project” would have to be changed a little, because `MyPkg/glue/A/` would not itself be a git repo