Hey folks, I am surprised with the whirlwind of events that happened at the Discourse yesterday and I am honestly shocked to even ask stuff now, but I need help with something and I still believe there is hope and goodness in the community because I believe in it. On a side note, do y’all keep you packages global or for every project you invoke an environment for it? for me I like keeping stuff global, believe it not or I took this habit of mine from my python coding processes. And also for Julia is keeping stuff global recommended or keeping toml files with individual package names recommended per environment/project?
No I keep my packages local to my project. I keep my “global” v1.12 environment to a few utilities. I keep most of my packages in project directories and increasingly use the workspaces feature. The reason for this is reproducibility. If I keep my Project.toml and Manifest.toml local, I have some chance of reproducing the environment in the future.
In Python, pixi and to a lesser degree, uv, encourage similar workflows.
I don’t think there is a blanket recommendation, but I concur with mkitti. The caveat of a massive environment running into needless incompatibilities is just as true for Julia as it is for Python. Another caveat is that a bigger environment has more packages to potentially re-precompile when it changes. That’s a much smaller problem in Python with far less optimizing compilation and far more isolated AOT-compiled dependencies that could thus be downloaded from a cache. The default versioned environment could work well enough for some people; some may even only need one small environment for everyday work.
Of course you can, --compiled-modules=no ignores precompile caches entirely. The caveat is you’d have to evaluate and JIT-compile everything per Julia process, the very motivation for precompiling some things per environment plus Julia installation.
Precompilation isn’t something wholly new in Julia; other languages specifying things to compile in advance is very familiar as AOT compilation. The user friction in Julia is that we currently have to precompile packages ourselves. It’s like building libraries from source, but the difference in some other languages is those libraries could be compiled in isolated “environments” yet interact with each other via a limited C interface. If I distribute the compiled library to people with the same operating system and architecture, it can immediately work in their projects too.
On the other hand, Julia packages are compiled more deeply together within each environment, so compiled (Julia) code across environments with small version differences are not compatible. That increases the possible states of our precompile caches, to the point it’s not worth distributing (this could change but I won’t talk about super hypothetical/experimental stuff here). Another way to think about it is the environment loosely puts us in an active compilation unit (if we switch among many environments in a process, conflicts can force restarts), and we don’t have a solid way to mix environments yet (base Julia developers are working on JuliaC which uses the C interface and reduces Julia’s large footprint, but again I won’t talk about experimental stuff).
100% local for each project is what I recommend in my Good Scientific Code Workshop. (This is shameless plug on the one hand, but on the other hand I strongly believe this practices is necessary for fostering good reproducibility practices in research; the block 6 of the workshop provides several arguments why you want to stick with local environments)
Hey folks one more question if you see, Workspaces are quite a modern feature(Julia 1.12+) should I adopt it because I am a bit scared of the backward compatibility aspect of it
At this level of generality this can be viewed as a (in fact, even Julia-unrelated) question if one should use backward incompatible new features of the language. Equally general answer could perhaps be that it depends on you goals, context, circumstances, … Are you developing a software package that is planned to be part of an already existing ecosystem, perhaps maintained by your institution to which you have some obligations? Or are you just about to start some individual project of your own, no obligations to anyone?
Anything larger than a throwaway script, I make into a project.
Every project is made into a package right from the beginning. There is simply no reason not to.
PackageMaker.jl (a GUI frontend for PkgTemplates.jl) makes package creation a breeze.
My “global” v1.12 environment contains the bare minimum of utilities — for example, it doesn’t include PackageMaker.jl. Everything needed in a project/package goes directly into that project’s environment. Anything that may be of occasional use, like benchmarking or plotting, but does not necessarily belong to the project under development, is put into a shared environment and managed by ShareAdd.jl. Thus, e.g. PackageMaker.jl lives in the @PackageMaker shared environment on my computer and is imported when necessary via the @usingany macro from ShareAdd.jl.
