[ANN] Numerics.jl – The Extended Standard Library

What’s the effective difference between putting using StaticArrays into atreplinit and making it a top-level statement in startup.jl?

atreplinit is not executed when running Julia non-interactively

2 Likes

Here are the first few lines of my startup.jl file. This has been working fine for me. Have I missed something?

using LinearAlgebra
using SuiteSparse
using SparseArrays
using AbstractFFTs
using FFTW
using IJulia
using JLD2

I thought it was there: https://juliahub.com/ui/Documentation , is this something you want?

Then your numerics must be a lot different from my numerics. Except for LinearAlgebra I do not require any of the other packages. What about FFTs, optimization, machine learning, PDE solvers,… I mean half of Julia’s ecosystem is built for doing numerics and there are probably a gazillion packages which could be useful.
Though I respect the motivation, I think it’s a very bad idea of having a meta package with such a generic and for many people misleading name.

4 Likes

I’m also not sure this is a good idea overall unless the individual libraries are very closely related and are truly evolving together. One reason is that the library version of Numerics.jl will need to have its major version bumped any time that any of the sub-packages have their major version changed. This means the rate of breaking changes in Numerics will be rather high. Maybe that’s not a problem for “convenience use” from the REPL. But I worry it will introduce a lot of unnecessary constraints into the graph of compatible package versions if people start to depend on Numerics.

We had a somewhat similar problem recently where CoordinateTransformations.jl was updated to no longer re-export symbols from Rotations.jl. The new setup is better with less version constraints, but transitioning led to some compatibility oddities which one normally doesn’t need to deal with.

6 Likes

Well, this is “circumvented” by not having any compat info at all in Numerics.jl.

1 Like

Might be neat to have a way for a package to be “terminal” - that is, something that no registered package should depend on. Something like this seems fine for interactive use (I agree that using up the name isn’t great), but if you’re writing a package, you should explicitly depend on the underlying packages.

I don’t know how that would be feasible, since you’d need to allow it to be included in a Project.toml, but maybe this could be at the level of the registry?

2 Likes

Speaking of naming, I have no idea how packages are registered, but is there no moderation of that process?? What keeps a malicious actor from submitting 120,000 packages each with a different english dictionary word as their name and no useful content? Just out of curiosity.

Looking it up: https://github.com/JuliaRegistries/RegistryCI.jl#automatic-merging-guidelines

It looks like it is theoretically possible for a malicious actor to register a lot of packages with a lot of dictionary words through the automated system. (note, I’m not saying Numerics was at ALL a malicious case, it just made me think about naming, and the internet these days is a place where you always need to assume there exists a malicious actor who might try to exploit every automated system. I spend a bunch of time helping people with their OpenWrt routers for example and am struck by how many portscans hit routers every hour).

There is a three day waiting period. If someone tried, I’m sure there would be manual intervention during that time frame.

4 Likes

I think even ad-hoc analysis scripts should come with a Manifest.toml, otherwise they are guarenteed to get out of date with the installed packages and become non-reproducible.

That’s not to say end users should have to think about this on average, rather the tooling should make it so easy that it happens by default. I feel part of doing that might be to embed the environment in the script somehow to make it a single file, maybe something like https://github.com/c42f/CodeEnvironments.jl.

2 Likes

I agree completely. When I say ‘interactive use’, I also mean projects where you’re going to save the Project.toml and Manifest.toml and your analysis code. When I say a package shouldn’t depend on this, I mean “package” in the strict sense, something that’s intended for using by sometime else. My point was addressed at your idea that you’d need to constantly bump the version of the meta package. I was thinking that, if the meta package is “terminal,” its bounds for its own dependences could be much more loose, perhaps not even having version bounds.

In other words ]add Numerics should just be like an alias for ]add All The Packages (and the same for using). It doesn’t add any functionality, so it’s version has no meaning beyond the version of it’s underlying packages, and the version bounds of the packages it reports should be sufficient to reason about compatibility. BUT, it wouldn’t do for this kind of pass through to propagate further.

3 Likes

Oh that’s interesting. It’s more like a way to seed a new environment with a bunch of packages? So it sounds to me that this might actually be quite useful, but it’s not exactly a package? More like having a reference environment which you can copy. Something vaguely like

pkg> add PkgA
pkg> add PkgB
pkg> ...
pkg> env save my_standard_env

later…

pkg> add my_standard_env
pkg> using my_standard_env   # ? maybe a bad idea
9 Likes

Oh, yeah - this seems like a more sensible way to think about it.

I think this is the primary use of such a thing :man_shrugging:

2 Likes

Environment setup is one thing. The other is which of the packages from the environment one actually draws in. So Project.toml and Manifest.toml for the definition of the env, and then include("mystuff.jl"), using mystuff for convenient access to multiple packages with one using.

pkg> resolve will make a Manifest.toml, and so will other package operations on the active project, but I am not sure it should be done by default upon activation and other operations. Pkg can’t really tell if the project needs a manifest or not if there isn’t one.

I think that end users should be aware of the concept. Most scientists embrace the idea of a reproducible environment with enthusiasm.

I really like this idea as well. For precedence, it reminds me of Debian’s concept of “package groups”, although this sounds more flexible.

That might be achieved with something like this… (using this TOML parser for simplicity, but it could be easily written without any external dependency).

using TOML
deps = TOML.parsefile(Base.active_project())["deps"]
for package in keys(deps)
    @eval Main using $(Symbol(package))
end
3 Likes

That’s interesting. I had in mind that it’s something very lightweight users could create for themselves. But curated environments specialized to particular domains would be handy, and perhaps more similar to package groups.

I made a Pkg feature request out of this thread https://github.com/JuliaLang/Pkg.jl/issues/1854

3 Likes