Why using ``] test package`` recompiles the package again?

General Usage
, ,
1

I’ve had this experience many times but now just installed a package from scratch. All worked as expected (maybe except the very long time), packaged installed and and compiled. Next I run the tests and … recompiled the package again. Why is that? And I further noticed that there are now two sets of files in compiled/v1.9/package with different sizes

2 Likes
2

Are you using additional test dependencies? This might introduce recompilation of the package

3

Pkg.test uses --check-bounds=yes, which is incompatible with the precompilation image that was generated during package installation.

3 Likes
4

Thanks, but would you mind elaborating a bit more?
When developing the package I often restart Julia to force recompilation. If I run the tests immediately after using GMT (where a recompilation takes place), it always recompiles GMT again. This is no package installation. Sorry, but I don’t find this normal.

5

For me it worked fine, but maybe I did not follow the same workflow you have?

Installing, then precompiling, then precompiling test environment, then testing

stefan@cube ~> julia --project=(mktemp -d) -tauto
(tmp.byjxPCUpkb) pkg> dev --local GMT
     Cloning git-repo `https://github.com/GenericMappingTools/GMT.jl.git`
   Resolving package versions...
    Updating `/tmp/tmp.byjxPCUpkb/Project.toml`
  [5752ebe1] + GMT v0.44.6 `dev/GMT`
    Updating `/tmp/tmp.byjxPCUpkb/Manifest.toml`
    ...
    Building GMT → `/tmp/tmp.byjxPCUpkb/dev/GMT/deps/build.log`

(tmp.byjxPCUpkb) pkg> test GMT
     Testing GMT
      Status `/tmp/jl_PnSTfU/Project.toml`
      ...
      Status `/tmp/jl_PnSTfU/Manifest.toml`
      ...
Precompiling environment...
  6 dependencies successfully precompiled in 112 seconds. 18 already precompiled.

     Testing Running tests...
Test Summary: | Pass  Total  Time
GMT           |    1      1  0.1s
     Testing GMT tests passed

and starting it again in a new julia process (it did not trigger precompilation)

stefan@cube ~> julia --project=/tmp/tmp.byjxPCUpkb -tauto
(tmp.byjxPCUpkb) pkg> test GMT
     Testing GMT
      Status `/tmp/jl_aLRnm5/Project.toml`
      ...
      Status `/tmp/jl_aLRnm5/Manifest.toml`
      ...
     Testing Running tests...
Test Summary: | Pass  Total  Time
GMT           |    1      1  0.1s
     Testing GMT tests passed

on

julia> versioninfo()
Julia Version 1.10.0-DEV.834
Commit 66c58505e90 (2023-03-19 15:18 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 1700 Eight-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver1)
  Threads: 16 on 16 virtual cores
6

I think the difference is that your attempt didn’t really installed the GMT binaries, that’s why you got the (nothing actually tested)

Test Summary: | Pass  Total  Time
GMT           |    1      1  0.1s

I am testing a new JLL solution but there are mysterious errors between OSs
For example this is what I get in a MacOS Big Sur (but would be the same in Linux)

(@v1.9) pkg> up
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Updating git-repo `https://github.com/GenericMappingTools/GMT.jl.git`
    Updating `~/.julia/environments/v1.9/Project.toml`
  [5752ebe1] ~ GMT v0.44.6 `https://github.com/GenericMappingTools/GMT.jl.git#GMT_jll` ? v0.44.6 `https://github.com/GenericMappingTools/GMT.jl.git#GMT_jll`
    Updating `~/.julia/environments/v1.9/Manifest.toml`
  [5752ebe1] ~ GMT v0.44.6 `https://github.com/GenericMappingTools/GMT.jl.git#GMT_jll` ? v0.44.6 `https://github.com/GenericMappingTools/GMT.jl.git#GMT_jll`
    Building GMT ? `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/795e50b2f3064632d01f54d700187e5982c2a266/build.log`
Precompiling environment...
  1 dependency successfully precompiled in 119 seconds. 87 already precompiled.

(@v1.9) pkg> 

julia> 
Joaquims-MacBook-Pro:mironeGIT j$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-rc1 (2023-03-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using GMT

(@v1.9) pkg> test GMT
     Testing GMT
      Status `/private/var/folders/yz/zrrm34bx14b0myqpl1848qdr0000gn/T/jl_ch8DX8/Project.toml`
  [5752ebe1] GMT v0.44.6 `https://github.com/GenericMappingTools/GMT.jl.git#GMT_jll`
  [08abe8d2] PrettyTables v2.2.3
  [bd369af6] Tables v1.10.1
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ? have new versions available but compatibility constraints restrict them from upgrading.
Precompiling environment...
  Progress [>                                        ]  0/1
  ? GMT
...
...
7

Start Julia with --check-bounds=yes.
Precompilation images in Julia 1.9 with --check-bounds=yes and --check-bounds=no are incompatible, so you need separate precompilations for both.

This was not the case in Julia 1.8. @Krastanov, which version of Julia are you using?

1 Like
8

nightly; added to the post above

9

Does not sound good to me. We’ll all end up with dozens of repeated precompiled files … and for some packages they are not exactly small in >=1.9

10

There’s no good way around it. Bounds-checking is implemented in the compiler: when it’s on, it generates different code than when it’s off. Since Julia 1.9 saves the compiled code, the mode you compile in matters: you can’t just change the mode after compilation is complete.

But that means you need to compile twice: once with bounds-checking on, and once with bounds-checking off. Now, we could do that up-front, but that would ~double the compilation time in many cases where the user never has any intention of running the tests. So better is to precompile once with default settings, and then when running tests we have to recompile (if the up-to-date source code has not already been compiled with --check-bounds=yes).

Compiling C is little different: want both -O0 and -O3? You have to compile twice.

If the amount of compilation time is annoying, and if you’re using SnoopPrecompile, consider turning it off in your local package dev environment: PSA for SnoopPrecompile: turning off extra workload for specific packages - #5 by tim.holy

1 Like
11

Perhaps a difference is that when developing C++, I’m basically only compiling with -O0, sanitizers, debug information, and assertions.
Perhaps we should also have the habit of generally developing Julia under --check-bounds=yes, etc.

1 Like
12

Also, as inference often takes the bulk of the compile time, I’d also like more options to weaken inference results.
Something like this: Comparing JuliaLang:master...YingboMa:myb/concrete_only · JuliaLang/julia · GitHub
It has two benefits:

  1. Improving compile times.
  2. It doesn’t decrease the performance ceiling of optimal code, unlike options like -O1. Instead, it makes suboptimal and bad code worse and more obvious. Thus, it’s actually easier to optimize code, which isn’t worth doing with -O1 or lower levels.
2 Likes
13

2 posts were split to a new topic: How to speed up re-running tests?