Package Structure in Julia Ecosystem

Internals & Design
,
1

Hi,

I have noticed that in the Julia Ecosystem, the trend is of maintaining a large number of small packages, quite unlike the Python ecosystem where we have do-it-all libraries like Pandas or sklearn or numpy.
Is there any advantage to such an approach with regards to Julia?

Thanks!

1 Like
2

Personally, I think there are a couple factors contributing to the difference:

  1. Package ecosystem youth: Julia just hasn’t been around as long and had packages stabilize/mature to a point where a larger “meta” package makes sense. Pre-1.0 I think this is totally to be expected, and yet even now we’re starting to see things starting to go that direction: JuliaIO, DiffEq, and I’d argue even JuliaPro are all examples where foundational packages are stabilizing on APIs and higher-level meta packages can be built efficiently to provide these sorts of “batteries included” kind of experiences for certain domains.

  2. Fairly well-coordinated package developer community: this is partly due to structure (having a centrally managed official package repository), but I also think it’s due to the unique culture Julia has built up around coordinating efforts in packages. At least more so than other package ecosystems I’m aware of. Between various Julia domain organizations, meetups, and JuliaCon, I’ve seen a lot of interest and effort go into trying to create strong, independent packages that play nice with others. I think this contributes to the package ecosystem in having smaller, stronger functionally packages and avoids the need for one person or small group to have to build everything from scratch for their own higher-level application.

8 Likes
3

Python also somewhat forces you to build monolithic libraries, because they are often written in C under the hood. If you combine two separate C libraries with Python in the middle there’s a large overhead, so instead you build all your C code together.

Julia has no such problem; you can combine two libraries and get specialised code for the combination. This enables a nice “lego blocks” approach wherein you can build a larger library from parts, replace parts at will and so on.

11 Likes
4

Also, loading a big package takes time in Julia, so there’s pressure to avoid them. Hence the creation of interface packages like MathProgBase, StatsBase, ScikitLearnBase, etc.

1 Like