Precompiling module each time without any change

General Usage
1

Since I migrated to Julia 1.9 I get a pre-compilation of the module I am developing each time I start a test script. This was not the case with 1.8, for which I had sometimes to force a recompilation by modifying some of the files. Does anybody know how I can debug this to know what triggers the recompilation.

The module uses CxxWrap to interface to a C++ package. It is defined like this:

module Geant4

using CxxWrap
@wrapmodule("jlGeant4")

function __init__()
    @initcxx
end

end #module
1 Like
2

You do have to compile once for normal usage and once when running Pkg.test because it sets coverage=true, and we compile different code for those two cases. (Turning on coverage slows down runtime performance.) But if you’ve just run Pkg.test, and you re-run it without changing your source code, it shouldn’t need to Precompiling... any packages.

The only things I can thing of are

  • are you sure it’s precompiling? Pkg will “reinstall” the test dependencies of your project, but if they’re already installed and precompiled from the previous Pkg.test run, that will basically be a no-op (lots of printing, but very little time, and specifically no Precompiling...).
  • might there be anything wrong with the clock on your system?
  • are you changing the DEPOT each time? That will result in fresh compilation

for which I had sometimes to force a recompilation by modifying some of the files

Sounds like you might benefit from include_dependency, if some files are required for your project but don’t fit within the definition of ordinary Julia code.

1 Like
3

Yes, it is precompiling, at least is what is informing me.

Geant4.jl mato$ julia --project=. -i examples/basic/B2/B2a.jl 
[ Info: Precompiling Geant4 [559df036-b7a0-42fd-85df-7d5dd9d70f44]

If I execute again immediately after without any change, I get the same INFO message.

Thanks for the hint, which is now not needed until I resolve the current problem.

4

BTW, this Info message only appears with the interactive option -i, but I’m sure it does the precompilation judging from the time it takes to start the script.

5

In non-interactive mode it’s a @debug so setting the env var JULIA_DEBUG=Geant4 will make it show up

Edit: As @mkitti pointed out below this should be JULIA_DEBUG=loading

6

How is Geant4 loaded? Have you modified the LOAD_PATH or set any environment variables? The results of julia -e "using InteractiveUtils; versioninfo()" would be informative here.

What is the output of julia -E "DEPOT_PATH[1]" ?

You might want to check ~/.julia/compiled/v1.9/Geant4 to see if there anything strange with the modified times of the files there.

7

No, it does not show up.

Geant4.jl mato$ JULIA_DEBUG=Geant4 julia --project=. examples/basic/B2/B2a.jl 

**************************************************************
 Geant4 version Name: geant4-11-01-patch-01 [MT]   (10-February-2023)
  << in Multi-threaded mode >> 
                       Copyright : Geant4 Collaboration
                      References : NIM A 506 (2003), 250-303
                                 : IEEE-TNS 53 (2006), 270-278
                                 : NIM A 835 (2016), 186-225
                             WWW : http://geant4.org/
**************************************************************
...
8

Try JULIA_DEBUG=loading

1 Like
9

Indeed, I got it wrong above. It will also tell you why any existing caches aren’t being used.

10

I am developing Geant4.jl, so my current directory is the checkout area. This is why I set the project --project=..

No

Geant4.jl mato$ julia -e "using InteractiveUtils; versioninfo()"
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin22.4.0)
  CPU: 8 Ă— Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores

Geant4.jl mato$ julia -E "DEPOT_PATH[1]"
"/Users/mato/.julia"

What I see strange is the date for the .dSYM files.

Geant4.jl mato$ ls -ls ~/.julia/compiled/v1.9/Geant4
total 152960
37624 -rwxrwxrwx  1 mato  staff  19263064 May 31 18:55 GogkO_DkPlq.dylib
    0 drwxr-xr-x  3 mato  staff        96 May 10 18:13 GogkO_DkPlq.dylib.dSYM
  136 -rw-r--r--  1 mato  staff     68134 May 31 18:55 GogkO_DkPlq.ji
37624 -rwxrwxrwx  1 mato  staff  19263064 Jun  1 09:48 GogkO_d0CIK.dylib
    0 drwxr-xr-x  3 mato  staff        96 May 10 21:50 GogkO_d0CIK.dylib.dSYM
  136 -rw-r--r--  1 mato  staff     68134 Jun  1 09:48 GogkO_d0CIK.ji
37624 -rwxrwxrwx  1 mato  staff  19263064 Jun  1 16:57 GogkO_hxYin.dylib
    0 drwxr-xr-x  3 mato  staff        96 May 10 18:04 GogkO_hxYin.dylib.dSYM
  136 -rw-r--r--  1 mato  staff     67987 Jun  1 16:57 GogkO_hxYin.ji
    0 -rw-------  1 mato  staff         0 May 19 08:53 jl_HPGSxk
  136 -rw-r--r--  1 mato  staff     65953 May 16 10:52 jl_fTnfXe
    0 -rw-------  1 mato  staff         0 May 16 10:52 jl_vW7QTQ
39544 -rw-r--r--  1 mato  staff  20244600 May 16 10:52 jl_wJrpie
11

Great this helps!!

Geant4.jl mato$ JULIA_DEBUG=loading julia --project=. examples/basic/B2/B2a.jl 
┌ Debug: Rejecting stale cache file /Users/mato/.julia/compiled/v1.9/Geant4/GogkO_hxYin.ji because file /Users/mato/Development/Julia/Geant4.jl/gen/build/lib/libGeant4Wrap does not exist
â”” @ Base loading.jl:2798
┌ Debug: Rejecting stale cache file /Users/mato/.julia/compiled/v1.9/Geant4/GogkO_d0CIK.ji (mtime 1.684766294152994e9) because file /Users/mato/Development/Julia/Geant4.jl/src/Geant4.jl (mtime 1.6856198847355766e9) has changed
....

The file that exists is /Users/mato/Development/Julia/Geant4.jl/gen/build/lib/libGeant4Wrap.dylib with the suffix.

What I am doing in the module is:

module Geant4
    using CxxWrap
    using Geant4_jll

    # Check whether the wrappers have been build locally otherwise use the binary package Geant4_julia_jll
    gendir = normpath(joinpath(@__DIR__, "../gen"))
    if isdir(joinpath(gendir, "build/lib"))
        include(joinpath(gendir, "jl/Geant4-export.jl"))
        @wrapmodule(joinpath(gendir, "build/lib", "libGeant4Wrap"))
    else
        using Geant4_julia_jll
        include(Geant4_julia_jll.Geant4_exports)
        @wrapmodule(Geant4_julia_jll.libGeant4Wrap)
    end
    ....

I do to fully specify the suffix for portability. Probably I should specify it depending on the platform I am.

12

Problem solved. If I specify the full library name the precompilation is done only once on a change.

-        @wrapmodule(joinpath(gendir, "build/lib", "libGeant4Wrap"))
+        @wrapmodule(joinpath(gendir, "build/lib", "libGeant4Wrap.$(Libdl.dlext)"))

However, I think this is a regression between 1.8 and 1.9. In dlopen says that can be omitted.

13

If you could come up with a MWE that would be fantastic for fixing the regression. (We need a test to see if the problem has been fixed and stays that way.) Either way, it should be reported.

1 Like