RFC: a "workaround" for the multi-project precompilation cache problem without long-term code debt

This is pretty much what I was hoping for. :smile: I think you’ve got a pretty solid plan here, which is only slightly foiled by activate.

A solution to this problem could be to first make precompilation dependent on the static dependency tree defined in the project, and not worrying about further activates until they happen. I see this as being acceptable because activate somewhat breaks the concept of a self-contained project as being atomic, and even allows packages which are already loaded to change versions from the perspective of code loading (even if you can’t actually load them).

Therefore, I think initially we should focus on just precompiling each project which is loaded in isolation, before any further activates occur. That way, the common case of loading and using only the first activated project would work well and be fast (common cases in my mind being running package tests, and starting a REPL with the --project flag). Note that we’ll also want separate precompilation directories for any targets like testing, since tests can contain additional dependencies.

We’d need to potentially get more fancy for multiple activates, which could be handled by creating “merged projects” which are pseudo-projects that have been precompiled with respect to the dependency trees of multiple projects (often a package’s project + one of the v1.x projects when doing development). This might be harder to do, but is also in my mind an edge case in the entire code loading scheme, so I believe it deserves special treatment anyway.

2 Likes