Precompile weirdness with my first package

General Usage
,
1

I’ve built an energy system model using JuMP and am preparing to open-source it, so as a first step I converted it into a package (named “Supergrid” for now). This is my first package so I may have messed something up, but here’s what’s happening.

Everything works perfectly if I manually precompile my package before importing it, like this:

(v1.1) pkg> activate Supergrid

(Supergrid) pkg> precompile
Precompiling project...

julia> using Supergrid, AxisArrays

julia> runmodel(carboncap=0.0);

Reading input data...
Building model...
Solving model...
[snip - everything is fine]

However, if I immediately try to import and run my stuff in a new Julia session without recompiling, this happens:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.1.0 (2019-01-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Supergrid, AxisArrays
[ Info: Recompiling stale cache file C:\Users\niclas\.julia\compiled\v1.1\Supergrid\v9ZK0.ji for Supergrid [32628ed0-2df8-11e9-34a4-b5e7f6273a90]
ERROR: LoadError: LoadError: syntax: extra token "MatlabHDF5File" after end of expression
Stacktrace:
 [1] include at .\boot.jl:326 [inlined]
 [2] include_relative(::Module, ::String) at .\loading.jl:1038
 [3] include at .\sysimg.jl:29 [inlined]
 [4] include(::String) at C:\Users\niclas\.julia\packages\MAT\Pn0pR\src\MAT.jl:27
 [5] top-level scope at none:0
 [6] include at .\boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at .\loading.jl:1038
 [8] include(::Module, ::String) at .\sysimg.jl:29
 [9] top-level scope at none:2
 [10] eval at .\boot.jl:328 [inlined]
 [11] eval(::Expr) at .\client.jl:404
 [12] top-level scope at .\none:3
in expression starting at C:\Users\niclas\.julia\packages\MAT\Pn0pR\src\MAT_HDF5.jl:39
in expression starting at C:\Users\niclas\.julia\packages\MAT\Pn0pR\src\MAT.jl:31
ERROR: LoadError: LoadError: Failed to precompile MAT [23992714-dd62-5051-b70f-ba57cb901cac] to C:\Users\niclas\.julia\compiled\v1.1\MAT\3FHIv.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1197
 [3] _require(::Base.PkgId) at .\loading.jl:960
 [4] require(::Base.PkgId) at .\loading.jl:858
 [5] require(::Module, ::Symbol) at .\loading.jl:853
 [6] include at .\boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at .\loading.jl:1038
 [8] include at .\sysimg.jl:29 [inlined]
 [9] include(::String) at C:\Stuff\Julia\Supergrid\src\Supergrid.jl:1
 [10] top-level scope at none:0
 [11] include at .\boot.jl:326 [inlined]
 [12] include_relative(::Module, ::String) at .\loading.jl:1038
 [13] include(::Module, ::String) at .\sysimg.jl:29
 [14] top-level scope at none:2
 [15] eval at .\boot.jl:328 [inlined]
 [16] eval(::Expr) at .\client.jl:404
 [17] top-level scope at .\none:3
in expression starting at C:\Stuff\Julia\Supergrid\src\inputdataEurope8.jl:1
in expression starting at C:\Stuff\Julia\Supergrid\src\Supergrid.jl:11
ERROR: Failed to precompile Supergrid [32628ed0-2df8-11e9-34a4-b5e7f6273a90] to C:\Users\niclas\.julia\compiled\v1.1\Supergrid\v9ZK0.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1197
 [3] _require(::Base.PkgId) at .\loading.jl:960
 [4] require(::Base.PkgId) at .\loading.jl:858
 [5] require(::Module, ::Symbol) at .\loading.jl:853

Even the fact that it insists on “recompiling stale cache file” is a bit strange since I didn’t make any changes after the successful run above. The error I get is probably because the MAT package I use to read input data from Matlab still doesn’t support Julia 1.0. But I thought I had fixed that by listing an unofficial fork of MAT in my Project.toml file following this suggestion. This worked fine before I converted my code into a package, and it works fine if I manually precompile as above. Another clue: when I call precompile myself immediately after the failed run, it needs to recompile MAT and Supergrid (and does so successfully).

(Supergrid) pkg> precompile
Precompiling project...
Precompiling MAT
[ Info: Precompiling MAT [23992714-dd62-5051-b70f-ba57cb901cac]
Precompiling Supergrid
[ Info: Precompiling Supergrid [32628ed0-2df8-11e9-34a4-b5e7f6273a90]

To aid any diagnostics, here is the results of calling status on my package:

(v1.1) pkg> activate Supergrid

(Supergrid) pkg> status
Project Supergrid v0.1.0
    Status `C:\Stuff\Julia\Supergrid\Project.toml`
  [39de3d68] + AxisArrays v0.3.0
  [a076750e] + CPLEX v0.4.3
  [5789e2e9] + FileIO v1.0.5
  [2e9cd046] + Gurobi v0.5.9
  [f67ccb44] + HDF5 v0.11.0
  [033835bb] + JLD2 v0.1.2
  [4076af6c] + JuMP v0.18.5
  [23992714] + MAT v0.4.0+ #v0.7-update (https://github.com/halleysfifthinc/MAT.jl)
  [86f7a689] + NamedArrays v0.9.2
  [d96e819e] + Parameters v0.10.3
  [91a5bcdd] + Plots v0.23.0
  [f3b207a7] + StatsPlots v0.10.1
  [f269a46b] + TimeZones v0.8.5
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [10745b16] + Statistics
    Status `C:\Stuff\Julia\Supergrid\Manifest.toml`
  [39de3d68] + AxisArrays v0.3.0
  [a076750e] + CPLEX v0.4.3
  [5789e2e9] + FileIO v1.0.5
  [2e9cd046] + Gurobi v0.5.9
  [f67ccb44] + HDF5 v0.11.0
  [033835bb] + JLD2 v0.1.2
  [4076af6c] + JuMP v0.18.5
  [23992714] + MAT v0.4.0+ #v0.7-update (https://github.com/halleysfifthinc/MAT.jl)
  [86f7a689] + NamedArrays v0.9.2
  [d96e819e] + Parameters v0.10.3
  [91a5bcdd] + Plots v0.23.0
  [f3b207a7] + StatsPlots v0.10.1
  [f269a46b] + TimeZones v0.8.5
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [10745b16] + Statistics

Any ideas? Did I screw up somehow or is something else going on?

2

I’m guessing you have the 1.0 compatible PR of MAT.jl loaded in your Supergrid environment, but not in your default environment. You’ll need to re-do the same process of checking out the 1.0 compatible PR in your default environment before your package will work.

3

You’re absolutely right, that fixed the problem. Thanks for the help! But I can’t understand why this worked. I only use MAT inside my package, never in my default environment. And apparently I don’t need to add any other dependencies from my package into the default environment, just this unregistered one. Why is this needed, and is this behavior documented somewhere?

4

This is because you are using a particular version of a package. This choice is reflected in the Manifest.toml of each environment which details the particulars (version, path, etc) of each package. When you start Julia in the default environment, it ignores any Manifest files which may be present in packages (eg your Supergrid package), and since you had omitted to tell Pkg that you need a specific branch of MAT.jl in your default environment, it failed to precompile.

1 Like
5

I see, thanks for explaining!