Reflections on Developer Experience After Building a Large Julia Codebase

[cuihantao]

Thank you! I need to try your JetBrains IDE plugin! Does it depend on LanguageServer.jl as well?

[Benny]

@cuihantao

For the months on Python project X, a single mamba activate xx in .bashrc will work for all the terminals opened. For Julia, I need to remember to activate the proper environment a lot.

I personally prefer specifying environments or making custom shell scripts that do so, but if you really want a global setting like that, you could make an alias with a default flag or alter the environment variable JULIA_PROJECT. I really wouldn’t mess with startup.jl for this.

@cuihantao

If I try to do activate SpareArrays in TestDrive, it will error because SpareArrays has yet to be add to TestDrive. I will then have to duplicate all dependencies from my package being developed.

I can’t tell what you’re trying to do here because there’s a lot of unexplained package names involved, but it sounds really off to duplicate dependencies just to switch environments. Pkg mode’s activate should be able to switch environments freely in practice. Not sure what you mean by for-debug packages, but shared environments and stacked environments sound relevant.

@cuihantao

venv decouples pyproject.toml from what’s actually being installed in the virtual env…But in Julia, when developing my package (in VS Code, the environment is on by default), adding any package will keep modifying Project.toml, either desirable or not.

That’s only because venv predates pyproject.toml and never incorporated it. That’s generally considered a weakness, that’s why tools built on pyproject.toml like Poetry base environments on it like Julia does with Project.toml.

@cuihantao

here’s the repo for reproducing the error:

First off, very nice MWE. Unfortunately, I either make entirely separate packages or incorporate non-package submodules simply because my submodules and packages never got big or important enough to warrant a monorepo package (and not all monorepos are themselves packages). Someone else who does maintain one should chime in here. In fact, this might be worth its own topic rather than one point in this one, it’ll help get more eyes on the specific issue without having to wade through the other points.

I’ll mention a few things in case they help, anyway. It looks like syncing the Manifest.toml is the problem, and very recently, the [workspace] feature was developed to merge monorepo manifests into one for the base project. Again, I never had to use it, so I don’t know if it’s useful here. Second, a small oddity I’ve noticed is that the base Project.toml has Statistics in it despite the source code never importing it. Just in case this is a pattern of duplication, packages don’t need to add the dependencies of their direct dependencies, in fact it risks bloat as the direct dependencies can drop their dependencies. The Manifest.toml will contain indirect dependencies on its own.

[cuihantao]

Very nice. Thanks again!! I will find ways to incorporate your ways of using the environment into my workflow!

That Statistics at the top level.. is because in my monorepo setup, the activated environment is the top monorepo. This mimics my use case of running test scripts that use Statistics when the top-level environment is active. I will check [workspace] as well.

[Tamas_Papp]

@Benny

shared environments and stacked environments sound relevant

In theory, yes. But in practice they are a nightmare because the actual versions may resolve differently, and a lot of debugging / optimization packages do rely on Julia’s internals and/or get a lot of bugfixes after each release.

The least painful solution I found for 1.12 is

[workspace]
projects = ["test", "debug"]

with the debug sub-environment containing the debug tools I use (eg Cthulhu.jl, ProfileCanvas.jl, etc).

[madppiper]

Thank you! I need to try your JetBrains IDE plugin! Does it depend on LanguageServer.jl as well?

Runs with (LanguageServer or JETLS) or without. You can activate them in the settings.

We have our inbuilt indexing system, which we are slowly upgrading on, so you can actually use it without as well (I do). It covers most of the LS features you are looking for (autocomplete, documentation, navigation etc) and because it is inbuilt. There are also smart templates now that you can use to complete certain aspects more easily.

Give it a go and please give me feedback, so I know what you are looking for in this, too. I already wrote down dot function annotations for the next release

[ChrisRackauckas]

@cuihantao

Performance: The new library is very promising. With a proper variable step integrator for stiff ODE, the simulation performance is probably 100x faster than my Python version, both in small test cases (e.g., for one with ~100 variables, which the Python version takes 1 sec to run and the Julia variant takes 10 ms) and large ones (e.g., another one with ~300k variables, Python’s is 1 hour + and Julia’s is 60 sec.). This is very impressive.

Can I get an MWE there? I think we can maybe just precompile that better.

[cuihantao]

Chris: the performance data you quoted here are run time after warming up, meaning that no precompilation time was included. The run time is very good.

The precompilation time related to DiffEq is not bad; it’s somewhere close to 5 seconds, including the precompilation of sparse AD and solver.

I don’t have an MWE ready to share because it’s hierarchical, and the codebase is not ready yet. I will let you know when ready to optimize!

BTW, how close am I to the largest ODE problem ever simulated with DiffEq?

[ChrisRackauckas]

@cuihantao

Chris: the performance data you quoted here are run time after warming up, meaning that no precompilation time was included. The run time is very good.

I mean the first run time, ttfx. That’s fixable.

@cuihantao

largest

@cuihantao

BTW, how close am I to the largest ODE problem ever simulated with DiffEq?

I think the largest I saw was a few hundred million?

[mkitti]

@cuihantao

The first point should be clear. For the months on Python project X, a single mamba activate xx in .bashrc will work for all the terminals opened. For Julia, I need to remember to activate the proper environment a lot. To get this working, I can think of using startup.jl and activate my project being developed, then it gets to my second point.

I typically invoke julia as julia --project which activates the Julia project in the current directory or parent directories. You may even want to create a shell alias for this.

$ touch Project.toml

$ alias julia="~/julia-1.12.1/bin/julia"
$ julia -E "Base.active_project()"
"/home/mkitti/.julia/environments/v1.12/Project.toml"      

$ alias jlproj="~/julia-1.12.1/bin/julia --project"        
$ jlproj -E "Base.active_project()"                        
"/home/mkitti/foo/Project.toml"

[liuyxpp]

It is a bit annoying that the version is hard coded in the alias. If you use juliaup, you can simply do (I believe, I haven’t test it though)

$ alias jlproj="julia --project"