Hey folks, I am surprised with the whirlwind of events that happened at the Discourse yesterday and I am honestly shocked to even ask stuff now, but I need help with something and I still believe there is hope and goodness in the community because I believe in it. On a side note, do y’all keep you packages global or for every project you invoke an environment for it? for me I like keeping stuff global, believe it not or I took this habit of mine from my python coding processes. And also for Julia is keeping stuff global recommended or keeping toml files with individual package names recommended per environment/project?
No I keep my packages local to my project. I keep my “global” v1.12 environment to a few utilities. I keep most of my packages in project directories and increasingly use the workspaces feature. The reason for this is reproducibility. If I keep my Project.toml and Manifest.toml local, I have some chance of reproducing the environment in the future.
In Python, pixi and to a lesser degree, uv, encourage similar workflows.
I don’t think there is a blanket recommendation, but I concur with mkitti. The caveat of a massive environment running into needless incompatibilities is just as true for Julia as it is for Python. Another caveat is that a bigger environment has more packages to potentially re-precompile when it changes. That’s a much smaller problem in Python with far less optimizing compilation and far more isolated AOT-compiled dependencies that could thus be downloaded from a cache. The default versioned environment could work well enough for some people; some may even only need one small environment for everyday work.
Of course you can, --compiled-modules=no ignores precompile caches entirely. The caveat is you’d have to evaluate and JIT-compile everything per Julia process, the very motivation for precompiling some things per environment plus Julia installation.
Precompilation isn’t something wholly new in Julia; other languages specifying things to compile in advance is very familiar as AOT compilation. The user friction in Julia is that we currently have to precompile packages ourselves. It’s like building libraries from source, but the difference in some other languages is those libraries could be compiled in isolated “environments” yet interact with each other via a limited C interface. If I distribute the compiled library to people with the same operating system and architecture, it can immediately work in their projects too.
On the other hand, Julia packages are compiled more deeply together within each environment, so compiled (Julia) code across environments with small version differences are not compatible. That increases the possible states of our precompile caches, to the point it’s not worth distributing (this could change but I won’t talk about super hypothetical/experimental stuff here). Another way to think about it is the environment loosely puts us in an active compilation unit (if we switch among many environments in a process, conflicts can force restarts), and we don’t have a solid way to mix environments yet (base Julia developers are working on JuliaC which uses the C interface and reduces Julia’s large footprint, but again I won’t talk about experimental stuff).
100% local for each project is what I recommend in my Good Scientific Code Workshop. (This is shameless plug on the one hand, but on the other hand I strongly believe this practices is necessary for fostering good reproducibility practices in research; the block 6 of the workshop provides several arguments why you want to stick with local environments)
