# RFC: ClusteringAPI.jl

**URL:** <https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258>\
**Category:** Statistics\
**Tags:** statistics, data, cluster, clustering\
**Created:** [March 28, 2024, 10:51pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258 "2024-03-28T22:51:34Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [April 2, 2024, 9:41pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/21 "2024-04-02T21:41:01Z")

</div>

> [@juliohm](#):
>
> I believe that this API is not beginner-friendly, nor it helps new users to quickly get their jobs done

it may be important context that afaik `LearnAPI.jl` is not intended for direct use by beginners, but rather as a `Tables.jl` -like abstract API for package developers

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 2, 2024, 9:42pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/22 "2024-04-02T21:42:38Z")

</div>

From what I have seen in the past, these attempts to create large complex APIs for ML tend to freeze development of basic features for most users.

I’d focus on getting a simple `cluster/clusterprob` pair working nicely with multiple models, and only then try to generalize things for more advanced use cases.

---

<div class="post-metadata">

**Author:** ![00krishna](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/00krishna/32/8843_2.png) [@00krishna](https://discourse.julialang.org/u/00krishna)\
**Post date:** [April 6, 2024, 1:50am UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/23 "2024-04-06T01:50:48Z")

</div>

The other thing missing in the existing clustering package, is an optimal clustering assignment. As we all know, different clustering algorithms can give different numbers of clusters, and can also assign a single datapoint to different clusters.

Here is the doco for NbClust,

> **[NbClust function - RDocumentation](https://www.rdocumentation.org/packages/NbClust/versions/3.0.1/topics/NbClust)**
>
> \<p\>\<code\>NbClust\</code\> package provides 30 indices for determining the number of clusters and proposes to user the best clustering scheme from the different results obtained by varying all combinations of number of clusters, distance measures, and...

The R package `NbClust` has a really nice feature where is runs a bunch of clustering algorithms over the data. I think there are like 20 different methods. And then the `NbClust` package spits out the optimal number of clusters and assignments, based upon different voting methods for each clustering method. This is one of the functions that I use a lot in R, and that I keep having to use RCall to pull into Julia.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 18, 2024, 1:21pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/24 "2024-04-18T13:21:12Z")

</div>

I plan to go ahead and finish this soon. It is only a matter of deciding how to do it.

> [@ablaom](#):
>
> While it might be okay for a low-level API, it may be preferable to avoid raw integers for label encoding in clustering algorithms. Sometimes you need to know the total number of labels but when subsampling _not all labels may be manifest_. This can lead to crashes, or silently wrong behavior. Better would be `CategoricalArray`s, which are well optimised, and which store the complete class pool in every view. While this can be mitigated by passing the total number of labels as metadata, this is difficult to accommodate in complicated pipelines.

Let’s focus on the low level api for now. At the moment nobody stops you from wrapping the raw output in a cetegorical array. I would also second what @juliohm said. The focus here is just the pure clustering API itself. It is better to put priority into having this API down, than to rather try to future-design and cater to ML applications. ML applications can always use a good API no matter how it is designed, provided it is good. But going the other way, using a ML-catered API for non-ML things, is certainly not as simple.

* * *

`cluster_labels` I think is better for `cluster_indices`. Furthermore, as was pointed out already, there is no reason to limit the `labels` into the integers. They could be anything. Although, since it could be anything, it might as well be the integers. Specific algorithms can provide specific vectors of anything to match the integers should they wish so. Personally I don’t see the point yet.

The question is what to use for representing “unclustered” data points that the algorithm failed to assign? If using the positive integers for the labels, the integer `-1` or `0` could be used for them. So this is a simple argument in favor of integers as labels.

* * *

At the moment the community seems split between `cluster(algorithm, data)` and `cluster(data, algorithm)` but the evidence so far shows that `cluster(algorithm, data)` is used in a larger part of the overall Julia ecosystem, so I’ll go with that unless substantial data are provided to favor the alternative.

* * *

> [@00krishna](#):
>
> The other thing missing in the existing clustering package, is an optimal clustering assignment. As we all know, different clustering algorithms can give different numbers of clusters, and can also assign a single datapoint to different clusters.

That is great suggestion, but also 100% orthogonal to the API discussion. Once the API is in place you should consider doing a PR that adds this functionality. Since you used it often enough you are likely the most qualified!

* * *

@juliohm @sylvaticus @RoyiAvital @aplavin @alyst @ablaom thank you for participating in the disucssion. Yould you also mind voting on the 3 proposals for where and how to put this API into place (poll at the top post)? THe cummunity is 50-50 now.

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [April 18, 2024, 1:36pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/25 "2024-04-18T13:36:48Z")

</div>

The BetaML API has the advantage that the user can choose, when met with new data, to (a) continue the training using the new data making another `fit!` (with the effect to adjust the medoids/mixtures/representatives…) or (b) limiting to “predict” the class assignments of the new data using the saved medoids/mixtures/representatives and without change them.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 18, 2024, 1:39pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/26 "2024-04-18T13:39:59Z")

</div>

This assumes that the clustering method used _can_ utilize “previous training”. From my perspective, all methods I used so far cannot do this. DBSCAN, to take an example: when faced with “new data” you will have to recluster everything by merging in new and old data. Hence, this “retraining” option cannot be used as a basis for the low level API that should be usable with **every single** clustering algorithm.

---

<div class="post-metadata">

**Author:** ![Datseris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datseris/32/13406_2.png) [@Datseris](https://discourse.julialang.org/u/Datseris)\
**Post date:** [April 18, 2024, 1:42pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/27 "2024-04-18T13:42:18Z")

</div>

It should be simple to provide a `recluster` function that takes in the output format and new data.

Remember with my proposed design `cluster(algorithm, data)` does not return a vector of labels. It returns a return type with arbitarry many fields. The function `cluster_labels()` takes the return type and returns the labels. Hence, it should be rather trivial to define `retype = recluster(out_type, new_data)` for algorithms that have this possibility. We can make `recluster` part of the API if ML people think it is useful!

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [April 18, 2024, 9:11pm UTC](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258/28 "2024-04-18T21:11:54Z")

</div>

@Datseris I don’t have a strong opinion about where to house the repo but can arrange an invitation to either JuliaML or JuliaAI if needed.

[Previous page](https://discourse.julialang.org/t/rfc-clusteringapi-jl/112258.md?page=1)