# Fixing Package Fragmentation

**URL:** <https://discourse.julialang.org/t/fixing-package-fragmentation/98712>\
**Category:** Community\
**Created:** [May 12, 2023, 2:13am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712 "2023-05-12T02:13:26Z")\
**Posts on this page:** 12\
**Page:** 4

<div class="post-metadata">

**Author:** ![xiaoxi](https://avatars.discourse-cdn.com/v4/letter/x/a9adbd/32.png) [@xiaoxi](https://discourse.julialang.org/u/xiaoxi)\
**Post date:** [May 22, 2023, 11:12am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/81 "2023-05-22T11:12:05Z")

</div>

How could the Julia community help developers avoid duplication of efforts in future packages?

In case some developers want to write a package similar to a current one, how could the Julia community help them to make this package as compatible as possible with current packages?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 22, 2023, 3:01pm UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/82 "2023-05-22T15:01:05Z")

</div>

That does sometimes happen upon registering new packages. Some folks do keep tabs on the new packages being registered and ask questions like these. Check out the new packages label on the General registry — and see, for example, ones where someone has talked about something being [“different”](https://github.com/search?q=repo%3AJuliaRegistries%2FGeneral+label%3A%22new+package%22++different&type=pullrequests) (often appearing as “how is this different from X”?).

---

<div class="post-metadata">

**Author:** ![xiaoxi](https://avatars.discourse-cdn.com/v4/letter/x/a9adbd/32.png) [@xiaoxi](https://discourse.julialang.org/u/xiaoxi)\
**Post date:** [May 22, 2023, 6:52pm UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/83 "2023-05-22T18:52:41Z")

</div>

Could the Julia community foster a culture of explaining the usefulness of packages when someone plans to do it rather than when someone plans to register it?

For example, the forum could have a section or thread to discuss ideas for new packages so that developers can get guidance from more knowledgeable developers before starting the package.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [May 22, 2023, 7:14pm UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/84 "2023-05-22T19:14:15Z")

</div>

Users already ask what is available to solve their particular problem here in this forum.  
That also goes for package developers, but in that case they have often done a thorough search  
for existing functionality at that point.

I personally think that there _is no_ problem with package fragmentation: If you’d like to write your own, why don’t you? Just make sure that potential users understand the tradeoffs between your package and its competition. (I.e., write a README…)

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 23, 2023, 10:59pm UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/92 "2023-05-23T22:59:13Z")

</div>

In the middle of this discussion I also think that we are forgetting the personal factor of open source development. Two people who don’t get along may have different opinions and may choose to spend their time implementing opinionated versions of the same package. What’s wrong with that? In open source, you can’t tell people what to spend time on unless you pay them. So we have to keep this in mind. Also a GitHub org is not the answer to everything. Some orgs have a BDFL. Not everyone is comfortable making someone else a BDFL of their work. Moving a package to an org is a kind of irreversible decision to make someone else have more say over your code than you even if they choose not to exercise this power out of courtesy. So any “policy” we come up with should respect this personal side of open source development.

---

<div class="post-metadata">

**Author:** ![xiaoxi](https://avatars.discourse-cdn.com/v4/letter/x/a9adbd/32.png) [@xiaoxi](https://discourse.julialang.org/u/xiaoxi)\
**Post date:** [May 24, 2023, 6:41am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/101 "2023-05-24T06:41:19Z")

</div>

Wouldn’t it be better to split this thread into several: one for how to avoid further fragmentation with future packages and another for each possible merge of similar packages?

It is a complex subject with many aspects, and each should deserve its own thread.

---

<div class="post-metadata">

**Author:** ![gustaphe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustaphe/32/18174_2.png) [@gustaphe](https://discourse.julialang.org/u/gustaphe)\
**Post date:** [May 24, 2023, 6:47am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/102 "2023-05-24T06:47:39Z")

</div>

We don’t need a thread collecting packages that can be merged. If you identify two packages you think can be merged, raise that discussion.

---

<div class="post-metadata">

**Author:** ![Kirby\_Zhang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kirby_zhang/32/36204_2.png) [@Kirby\_Zhang](https://discourse.julialang.org/u/Kirby_Zhang)\
**Post date:** [May 25, 2023, 7:04pm UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/109 "2023-05-25T19:04:03Z")

</div>

there should be super packages which promise the best ways to do a certain set of related things. A good super package can be almost like a Linux distribution, curated and providing support for interworking of its components.

This discourse site is the most active place of Julia community, but it lacks a dashboarding feature that shows the state of packages (such as those collected on [Home · JuliaHub](https://juliahub.com/ui/Home)). If visitors of the most active Julia site can tell at a glance, at what rate packages are being updated, how many dependents they have, which packages are related or similar, where in the classification hierarchy (or hierarchies) the package falls under, i.e. things Juliahub might have tried to do but has not, then I believe there will be a natural consolidation over time, instead of attention being spent on unused, un-updated packages.

---

<div class="post-metadata">

**Author:** ![Eric\_Sadit\_Tellez](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eric_sadit_tellez/32/33615_2.png) [@Eric\_Sadit\_Tellez](https://discourse.julialang.org/u/Eric_Sadit_Tellez)\
**Post date:** [May 25, 2023, 8:01pm UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/110 "2023-05-25T20:01:49Z")

</div>

Giving users tools to find and choose packages, use better the package’s API, organize them, etc., is fundamental. This can help avoid someone starting a new package and choosing another working package that requires a minor intervention to do whatever is needed, and in the same way, enable more effective use in general through documentation.

On this matter, I was on the idea of creating full-text search indexes for packages descriptions and docs,

> [@Searching and clustering julia packages by name, description, topics, and readme](https://discourse.julialang.org/t/searching-and-clustering-julia-packages-by-name-description-topics-and-readme/98813):
>
> Hi, This post is an early proposal to tackle some discoverability and fragmentation issues on the Julia package ecosystem with information tools. The idea is to complement the call of gathering efforts and standardization with discoverability strategies and tools that take advantage of this and specific text information in packages. The intuition is that providing these tools allows people to find the required package and avoid unnecessary package multiplication. I know that Juliahub has some …

For instance, several package managers support package searching and some categorization, but most will not support full-text search.

I kept working on the proof of concept. The new version uses web scrapping instead of GitHub API and includes support for packages hosted in Gitlab and possibly others.

[https://github.com/sadit/Search-Julia-Packages/blob/main/search-pkgs.ipynb](https://github.com/sadit/Search-Julia-Packages/blob/main/search-pkgs.ipynb)

It can search on package names (with errors) and by readme content (also topics and descriptions if they are in Github). It supports changing name and content weighting scores.

It also adds support for FT search on module documentation based on `Base.Docs` and uses the `Lunr` search index that `Documenter` produces (searching without installing).

[https://github.com/sadit/Search-Julia-Packages/blob/main/search-docs.ipynb](https://github.com/sadit/Search-Julia-Packages/blob/main/search-docs.ipynb)

It is pretty similar to packages but for documentation. It is in an early stage and requires a lot of work, but also just works.

In addition, visualization and a cluster are generated  
[https://github.com/sadit/Search-Julia-Packages/blob/main/clustering.ipynb](https://github.com/sadit/Search-Julia-Packages/blob/main/clustering.ipynb)  
These are based on package README files. This can help get a view of similar packages as a whole.

A small web API using the `Oxygen` package was created to see who to use the API. Additionally, the package database is slow to load per query, so if it is used someday, it should be as a web server or something like that.

There are some sites to search for packages (juliahub included), but most of them need a browser or install other kinds of tools. The proposal is to create tools that work in the REPL or any Julia process.

So, the idea is to gather people interested in going in this direction and produce some tools that can help the community using the information already on the packages. I am open to discussing and collaborating on these ideas (negative ideas that make me stop this effort are also welcome).

---

<div class="post-metadata">

**Author:** ![Kirby\_Zhang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kirby_zhang/32/36204_2.png) [@Kirby\_Zhang](https://discourse.julialang.org/u/Kirby_Zhang)\
**Post date:** [May 26, 2023, 7:44am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/111 "2023-05-26T07:44:19Z")

</div>

I tried to use ChatGPT for a simple dataframe groupby. It generated what looked like complete code, so I was pretty excited. But it turns out to have totally misunderstood the API. The helpful part is it got me started looking at what functions to use, starting with a prompt was fun and mysterious, compared to starting by dredging through documentation.

---

<div class="post-metadata">

**Author:** ![pukpr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pukpr/32/49114_2.png) [@pukpr](https://discourse.julialang.org/u/pukpr)\
**Post date:** [May 28, 2023, 1:20am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/112 "2023-05-28T01:20:22Z")

</div>

Some of it is mysterious. I thought the code was incomplete when I saw some parameters followed by “…”. Oh yeah, the var arg list constructor, used to to convert a vector to a list of args.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 28, 2023, 9:40am UTC](https://discourse.julialang.org/t/fixing-package-fragmentation/98712/113 "2023-05-28T09:40:13Z")

</div>

> [@mohamed82008](#):
>
> Moving a package to an org is a kind of irreversible decision to make someone else have more say over your code

I am don’t see why. If the license is FOSS, you can always fork out the project. Worst case scenario is having to pick a different name.

Sure, forks are uncommon in the Julia community, but I would love to see more of them. A lot of useful packages just sit there semi-abandoned (or “community maintained”) when an interested developer could just fork them and see what happens.

[Previous page](https://discourse.julialang.org/t/fixing-package-fragmentation/98712.md?page=3)
