How to delete precompile cache?

Hello,

I played around with precompilation of my own modules some time ago, but do not use it any more for my own modules and have removed all _precompile_ whatever from the sources. Now, I suddenly get this error message every time I want to run a file in julia

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module ode_integratoren with build ID 2113025778823145 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

Remark: the actual code runs fine, but it takes about 10 seconds to load now.

How do I get rid of this ? I am fine with nuking the whole cache or whatever, but do not want to go ballistic and rm -rf ~/.julia yet. The documentation is unhelpful, i.e. does not mention anything about this at all as far as I can see, and How do you clear cache from a __precompile__()? does not work any more.

Best Regards

1 Like

rm -rf ~/.julia/compiled

6 Likes

Thank you, but it does not work. I in fact did only a

cd ~/.julia/compiled
rm -rf v0.7

because

% ls ~/.julia/compiled 
v0.7

After that I still get

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module ode_integratoren with build ID 2114178747401639 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

To be sure, a find . -name \*jl|xargs grep -i precompile in the directory tree which contains all files comes up empty.

1 Like

It may be that julia only checks for the existence of ~/.julia/compiled before attempting to load anything, regardless of whether the files are there or not. In that case removing ~/.julia/compiled should really do the trick.

1 Like

No, sorry. Did a

% rm -rf ~/.julia/compiled
% ls  ~/.julia/
clones  environments  logs  packages  registries

Still

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module ode_integratoren with build ID 2115199876604024 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

Well, guess I will do a rm -rf .julia now to be sure.

1 Like

__precompile__(true) is now the default so perhaps you need a __precompile__(false).

2 Likes

Well, unfortunately

% find . -name \*jl|xargs grep -i precompile
%

So, if there are no __precompile__(true) any more, where and why should I suddenly put that __precompile__(false) ? And there have been none for at least a week now. I just got that suddenly now, on top. I am also a bit surprised that there is no built in function in pkg3 to clear that cache if it is the default, on windows I would now have to start hunting around for the directory I guess.

Where: Above the module declaration, same as where you put __precompile__(true).
Why: Because the default changed.

No, this cannot be it … Somehow I am completely on the wrong track. I did now actually do a

% rm -rf ~/.julia
% ls  ~/.julia/  
ls: cannot access '/home/ckoe/.julia/': No such file or directory

Then re-added the few packages I was using. After that I still get the

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
...

Any other ideas ?

Edit: julia --compiled-modules=no ... of course silences this.

Further trying to get to the bottom of that rabbit hole. After a fresh rm -rf ~/.julia

julia> push!(LOAD_PATH, "../Hilfsfunktionen/")
julia> using hilfsfunktionen
[ Info: Precompiling hilfsfunktionen [top-level]

OK, that works (not sure how to stop this markup from switching to red).
Next module it was complaining about.

julia> push!(LOAD_PATH, "../ODE_Integratoren/")
julia> using ode_integratoren
[ Info: Precompiling ode_integratoren [top-level]
ERROR: LoadError: ArgumentError: Package StaticArrays not found in current path:
- Run `Pkg.add("StaticArrays")` to install the StaticArrays package.

Fair.

(v0.7) pkg> add StaticArrays
...
julia> using ode_integratoren
[ Info: Precompiling ode_integratoren [top-level]

Quit julia. Run the problematic code and get

% julia  crtbp_phasenraum.jl 
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module ode_integratoren with build ID 2120499899079677 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

Any help appreciated. This obviously has nothing to do with my old precompilation attempt, so

I am not going to wade through a week of git logs to match the state of the files.

So, deeper. The crtbp_phasenraum.jl starts with

push!(LOAD_PATH, "../ODE_Integratoren/")
push!(LOAD_PATH, "./")
using ode_integratoren,ode_symplektisch,crtbp_modul,Printf

Copying and pasting those lines into julia after removing .julia and re-installing StaticArrays gives the error

julia> using ode_integratoren,ode_symplektisch,crtbp_modul,Printf
[ Info: Precompiling ode_integratoren [top-level]
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
[ Info: Precompiling ode_symplektisch [top-level]
[ Info: Precompiling crtbp_modul [top-level]
┌ Warning: Module ode_integratoren with build ID 2121584251246112 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

Now, the modules in crtbp_modul

push!(LOAD_PATH, "../Hilfsfunktionen/")
push!(LOAD_PATH, "../ODE_Integratoren/")
using ode_integratoren,ode_symplektisch,hilfsfunktionen,Printf,DelimitedFiles

and in ode_integratoren.jl

push!(LOAD_PATH, "../Hilfsfunktionen/")
using StaticArrays,LinearAlgebra,hilfsfunktionen

The ode_symplektisch.jl does not use anything.

This stinks of some kind of race condition.

Edit: Yes, might well be https://github.com/JuliaLang/julia/issues/17320 !

Well, setting this on top of the two modules it complains about, i.e. hilfsfunktionen and ode_integratoren, before the module keyword, I hope that is the right location, does not completely solve this anyway (of course I deleted ~/.julia/compiled beforehand)

┌ Warning: Module ode_integratoren with build ID 2126327656566032 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

So, this new precompilation default is simply broken for me at the moment. I will try to make a MWE and then open a new issue (or do so anyway if I cannot distill a MWE).

Are you importing that package from another where precompilation is enabled?

Not that I am aware of. As shown above a find . -name \*jl|xargs grep -i precompile in the directory which contains all of my julia files comes up empty.

I am on NFS though and I read in the issues linked by #17320 that locking may not be implemented completely. I tried to test that by moving the whole tree on a local ext4, then symlinkig ~/.julia (export JULIA_PKGDIR=/tmp/… did not work ?)
to the same local ext4. Then I installed StaticArrays and verified that the local filesystem was used for that, i.e. (after the fact)

% ll /tmp/jpackages/compiled/v0.7/
 20 -rw------- 1 ckoe bccms  18737 Aug 17 20:25 hilfsfunktionen.ji
152 -rw------- 1 ckoe bccms 152251 Aug 17 20:25 ode_integratoren.ji
100 -rw------- 1 ckoe bccms  99795 Aug 17 20:25 ode_symplektisch.ji
  4 drwxr-xr-x 2 ckoe bccms   4096 Aug 17 20:25 StaticArrays/

Unfortunately, there is no effect on the error I am observing. Would have been nice to blame it on NFS locking issues :slight_smile:

I tried it with 1.0.0, too, but no change.

I will try to distill a MWE tomorrow.

Thank you !

As I have said, precompilation is the default now (on 0.7+).

OK, I misunderstood the question probably !

Of course if I set __precompile__(false) in all three module files, i.e. crtbp_modul.jl, …/ODE_Integratoren/ode_integratoren.jl and …/Hilfsfunktionen/hilfsfunktionen.jl there is no error.

If crtbp_modul.jl does not contain __precompile__(false) I get

┌ Warning: Module ode_integratoren with build ID 2134155791241394 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

Thank you for your help with that !

EDIT: If __precompile__(false) is set in crtbp_modul.jl one can remove it from ode_integratoren.jl without consquences. Removing it from hilfsfunktionen.jl in addition to that triggers

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963

Additional data point: There are no problems at all with

Version 0.7.0-beta2.0 (2018-07-13 19:54 UTC)

I am unsure when exactly it became the default, but I was using this beta 2 till about a week or so ago, so some time after 0.7.0 release. Probably that is why I did not see this before.

Edit: Yes, beta 2 does not precompile apparently

% ll /tmp/jpackages/compiled     
ls: cannot access '/tmp/jpackages/compiled': No such file or directory

@kristoffer.carlsson : Thank you for your patience with me and your help. Here is the MWE as promised.

Please note, that to trigger the problem it is apparently necessary to have the main program and the modules in different directories layed out like this:

% ls *
MAIN:
crtbp_modul.jl  main.jl

MODUL_1:
ode_integratoren.jl

MODUL_2:
hilfsfunktionen.jl

The contents of the files are then

% cat MAIN/main.jl 
push!(LOAD_PATH, "../MODUL_1/")
push!(LOAD_PATH, "./")
using ode_integratoren,crtbp_modul,Printf
jacobic=readinput()
% cat MAIN/crtbp_modul.jl 
module crtbp_modul
export readinput
push!(LOAD_PATH, "../MODUL_2/")
push!(LOAD_PATH, "../MODUL_1/")
using ode_integratoren,hilfsfunktionen,Printf,
  DelimitedFiles
  function readinput()
    print("Reading Input : ")
    jacobic=parse(Float64,readline())
    return jacobic
  end
end
% cat MODUL_1/ode_integratoren.jl 
module ode_integratoren
push!(LOAD_PATH, "../MODUL_2/")
using StaticArrays,LinearAlgebra,hilfsfunktionen
end
% cat MODUL_2/hilfsfunktionen.jl 
module hilfsfunktionen
using LinearAlgebra
export interpolation
function interpolation(xdata,ydata,x)
  return xdata
end
end

Then the precompilation fails after rm -rf ~/.julia/compiled/v0.7 like this for me:

MAIN % ~/bin/julia-0.7.0/bin/julia main.jl
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module ode_integratoren with build ID 2140014571188290 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941
Reading Input : 

Please note that with the old beta 2 (so without precompilation default) after rm -rf ~/.julia/compiled/v0.7:

MAIN % ~/bin/julia-0.7.0-beta2/bin/julia main.jl
Reading Input :

Now, the surprise ! If you put all files in one single directory and modify them accordingly like this

ALL_IN_ONE % cat main.jl 
push!(LOAD_PATH, "./")
using ode_integratoren,crtbp_modul,Printf
jacobic=readinput()
ALL_IN_ONE % cat crtbp_modul.jl 
module crtbp_modul
export readinput
push!(LOAD_PATH, "./")
using ode_integratoren,hilfsfunktionen,Printf,
  DelimitedFiles
  function readinput()
    print("Reading Input : ")
    jacobic=parse(Float64,readline())
    return jacobic
  end
end
ALL_IN_ONE % cat ode_integratoren.jl 
module ode_integratoren
push!(LOAD_PATH, "./")
using StaticArrays,LinearAlgebra,hilfsfunktionen
end
ALL_IN_ONE % cat hilfsfunktionen.jl 
module hilfsfunktionen
using LinearAlgebra
export interpolation
function interpolation(xdata,ydata,x)
  return xdata
end
end

the precompilation starts working:

ALL_IN_ONE % ~/bin/julia-0.7.0/bin/julia main.jl
Reading Input : 

I still believe this is a bug of some kind, although my use of julias facilities might be a bit non-standard !

Not editing the previous post but adding this one because I actually like seeing the readinput() so that I know it actually started instead of just dying silently. But it also breaks down without the functions:

% cat MAIN/main.jl 
push!(LOAD_PATH, "../MODUL_1/")
push!(LOAD_PATH, "./")
using ode_integratoren,crtbp_modul,Printf
% cat MAIN/crtbp_modul.jl 
module crtbp_modul
push!(LOAD_PATH, "../MODUL_2/")
push!(LOAD_PATH, "../MODUL_1/")
using ode_integratoren,hilfsfunktionen,Printf,
  DelimitedFiles
end
% cat MODUL_1/ode_integratoren.jl 
module ode_integratoren
push!(LOAD_PATH, "../MODUL_2/")
using StaticArrays,LinearAlgebra,hilfsfunktionen
end
% cat MODUL_2/hilfsfunktionen.jl 
module hilfsfunktionen
using LinearAlgebra
end

gives

MAIN % ~/bin/julia-0.7.0/bin/julia main.jl
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for ode_integratoren [top-level]
│   exception = Required dependency hilfsfunktionen [top-level] failed to load from a cache file.
└ @ Base loading.jl:963
┌ Warning: Module ode_integratoren with build ID 2141753802256008 is missing from the cache.
│ This may mean ode_integratoren [top-level] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:941

% ~/bin/julia-0.7.0-beta2/bin/julia main.jl
%

Decided now to submit a new issue, see https://github.com/JuliaLang/julia/issues/28756