Fixing Package Fragmentation

Community
13

This kind of feels like two separate issues:

  1. Big packages that are broken up into several small ones with perhaps a top level package re-exporting those methods. Most of the time a single or small team of contributors are maintaining all of these packages.

  2. Many packages that are trying to achieve similar functionality (plotting, data formats, interfaces, AD) that are mainly developed by different people with only somewhat overlapping user bases.

I must admit I particularly don’t like the first approach where I have to look through many sub-packages to see where methods are called. In that sense I prefer the monolithic approach given by SciPy. It sometimes feels like a larger issue I have with Julia packages is that it can be much easier for me to read libraries like BLIS and SciPy quickly and understand what is going on and where things are coming from than some Julia packages. In some sense, this (for me) creates a lot of friction contributing to some of these packages. This could perhaps be part of the reason for the second issue is it can be easier and more rewarding to work on your own packages. I don’t think that is a bad thing!

Now, the ecosystem has done this mainly as a means to reduce latency (pre v1.9) which has worked very well. I have considered breaking up Bessels.jl into many small sub-packages for different functions (Airy, Bessel, Gamma) but I have been resistant as it personally is much easier for me to develop in a big mono-repo. I also find that it is easier to maintain a single documentation, a single place to discuss issues, maintain CI, attract users as well as pool contributors. I don’t think that would be similar if this was scattered amongst many different packages. It also provides some stamp of implementation quality. Like if an implementation is in SciPy I am pretty sure that it has some level of quality and testing.

I think the release of v1.9 has opened many different possibilities. I of course realize Bessels.jl is pretty much the perfect example of the advantages of the new caching code. I have shown this before but I’ll give a new example with the recent stable release.

# Version 1.8.2 (2022-09-29)
julia> @time @eval using Bessels
  0.030072 seconds (45.18 k allocations: 4.904 MiB)

julia> @time @eval airyai(1.2 + 1.1im)
  0.528501 seconds (962.46 k allocations: 48.968 MiB, 2.77% gc time, 99.91% compilation time)

# Version 1.9.0 (2023-05-07)
julia> @time @eval using Bessels
  0.030401 seconds (34.75 k allocations: 3.267 MiB)

julia> @time @eval airyai(1.2 + 1.1im)
  0.000290 seconds (62 allocations: 3.125 KiB)

Pre v1.9 I was seriously considering the cost of adding new functions to each version to the user and how it affected package load times and time to first function evaluation. Now, I am really just considering how many functions I should explicitly precompile and their affect on cache file size. I believe a lot of effort on v1.10 and beyond might be on reducing these file sizes.

Now, I am also aware that users might only use one function out of hundreds and so they are also paying for all of the other functions and features we are slowly adding. We have moved to a module base approach (that I guess in the future someone could split into subpackages) that isolates dependencies and functions more similar to SciPy. I guess we kind of ended up at the first approach with many submodules that are re-exported in the end :man_shrugging: