Alright, thank you so much for the thoughtful discussion.
To me this means that making the effort to precompile packages isn’t worth it, as likely I won’t get the benefits. Unless there exists a clear list of “if you do XYZ then you’re guaranteed that your pre-compilation won’t get invalidated”, then from my POV whether or not invalidations will happen depend on Julia internals, and I don’t want to have to learn this specific aspect, which might change with Julia version, or with PrecompileTools version etc. Unless you tell me I’m missing something here…?
Yep, precompilation should occur whether it’s add or dev, add packages just needs extra work after changes so it’s better to dev when you’re making many changes quickly.
This is definitely type piracy. B did not define any of the input types: typeof(A.printstats), Val{:H5}, Any.
To elaborate on danielwe’s explanation of invalidation, you wouldn’t even need separate package caches for a callee to invalidate a compiled method:
a(b::Bool) = b ? t() : f() # (caller) a(b) calls (the callees) t() or f()
t() = 1
a(true) # compiles here with a UndefVarError for f()
f() = 0 # invalidates dependent a(b::Bool)
a(true) # recompiles here with the defined f()
Not sure, but you seemed to imply that the @precompile_workload printstats(path) was done in package B that imports A, so B’s cache should have printstats(path), despite not owning the method. If you import B or A then B without nothing in between, I wouldn’t expect B’s cache to be invalidated there.
No, it’s pretty straightforward that if you change a function’s methods, other methods that call that function may have to be recompiled. The hard part is finding where, though type piracy and inherently poor type inference is usually a red flag. I’m also not entirely convinced this is just invalidation.
This seems important. What other imports are there and in what order, are they custom packages that do anything to A.printstats?
Any environment with B should at least have those 6 dependencies, in the manifest if not the project. Version compatibility with other packages in the environment can result in those dependencies (or B itself) downgrading to previous versions, check ] st with the environment activated. Environments with changes should automatically precompile their packages (you’ll see a printout) at some point, though you need to restart the session to use it if the packages were already loaded.
Not necessarily, inlined methods (automatic or @inline) only need to be compiled for their caller’s instance. I think whether the method is also separately compiled depends on the circumstance and Julia version; I tried to remake an example showing that an inlined method isn’t separately compiled, but it doesn’t work anymore (v1.11.5).
Ok I see your point. At the same time the type piracy in itself is not an issue: I’ll never publish any of these packages, so I have control over how this function and these types are used. The only scenario where something could do wrong is if I tried to a use a public dependency that exports Val{:H5}. I’ll take the chance.
I also feel like we’re over-focusing on this specific aspect because that’s what little code I gave you. In reality I just made a few experiments in terms of adding and removing other imports, all of which made different degrees of impact in re-compile time, and none of them have anything to do with my type piracy, as far as I can tell. So I’m confident that Daniel’s guess is likely correct when he said:
problem may be some of your other dependencies invalidating stuff that’s deeper down in your implementation
The rest of your post reinforces my view that trying to learn and keep up with what Julia can and can’t currently do in terms of avoiding invalidations, and trying to adjust my code to benefit, is not a good investment of time to me.
I don’t have time for a more elaborate reply, but I think the right tool for what you’re trying to do is a dedicated startup package for the notebook environment where you’re using A and B. The purpose of the startup package is just precompilation and nothing else. It should import all the packages you’ll use in the environment, perhaps with @recompile_invalidations wrapped around the imports, and then define a precompile workload that includes things like printstats(filename). See the documentation for PrecompileTools.jl for details on how to implement this.
Not involving other people does help, but personally practicing type piracy still has all the same risks as multiple people doing it. For example, one of the worst things is two packages accidentally defining the same method signature, causing the surviving method to depend on a load order that can depend on your dependencies’ imports as much as the imports you type e.g. using B, A would actually load A first because B depended on A.
Types aren’t exported, their names are; export Val{:H5} won’t even parse, though it will work for a type alias. The types associated with the names Val, Symbol, or any alias are defined thus owned by Base/Core, which you are automatically using.
Given your stopwatch results, probably. Invalidations and recompilation could be caused anywhere else, and the timing discrepancy may not even be caused by invalidation. Can’t rule anything out with so little known.
The compile-time benefits (like preventing invalidations) and run-time benefits are worth avoiding type piracy and making the types of variables and expressions in your methods inferrable from the input types, though it’s worth mentioning that limited, controlled poor inference is a feature of a dynamically typed language. That’s just in general, not necessarily related to your issue.