# FlexiJoins vs SortMerge (particularly in astronomy workflows)

**URL:** <https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632>\
**Category:** Astro/Space\
**Created:** [April 28, 2025, 11:18pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632 "2025-04-28T23:18:21Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 28, 2025, 11:18pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/1 "2025-04-28T23:18:21Z")

</div>

> [@Mapping Julia’s Astronomy Package Ecosystem: Bridging Gaps and Highlighting Strengths](https://discourse.julialang.org/t/mapping-julia-s-astronomy-package-ecosystem-bridging-gaps-and-highlighting-strengths/128249/42):
>
> Should [SortMerge.jl](https://github.com/gcalderone/SortMerge.jl/) be in the list?  
> It is not as polished as [FlexiJoins.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/FlexiJoins) but its performance are very good (e.g. it can cross-match 2 catalogs each with 10^6 entries in ~2 sec, [example here](https://github.com/gcalderone/SortMerge.jl/?tab=readme-ov-file#match-arrays-of-geographical-coordinates)).

I wouldn’t call FlexiJoins.jl “polished” tbh 🙂 It does support lots of different join conditions with generally efficient implementations, but stuff like error messages and internal implementation design can definitely be improved.  
A major feature of FlexiJoins.jl is uniform convenient interface for a wide range of join conditions and types. In particular, it supports SkyCoords.jl directly – joining two catalogs is just

```julia
innerjoin((A, B), by_distance(x -> x.coords, separation, ≤(1u"arcsecond")))

```

assuming elements of both datasets contain the `coords` property with SkyCoords objects.

I haven’t checked, but if SortMerge.jl is more performant, it could be added as an alternative backend to FlexiJoins.jl. Currently, it uses NearestNeighbors.jl trees for by-distance joins.

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [April 30, 2025, 11:01pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/2 "2025-04-30T23:01:54Z")

</div>

I made some test and the outcome is that FlexiJoins typically performs better than SortMerge, but the latter is faster when the catalogs are pre-sorted.

MWE (see documentation [here](https://github.com/gcalderone/SortMerge.jl/?tab=readme-ov-file#match-arrays-of-astronomical-coordinates)):

```julia
using SkyCoords, AstroLib, SortMerge
using FlexiJoins, Unitful, UnitfulAstro

# Generate some random coordinates
nn = 10_000_000
c1 = ICRSCoords.(rand(nn) .* 2pi, rand(nn) .* pi .- pi/2);
c2 = ICRSCoords.(rand(nn) .* 2pi, rand(nn) .* pi .- pi/2);

# Cross match using SortMerge:
lt(v, i, j) = (v[i].dec < v[j].dec)
function sd(v1, v2, i1, i2, threshold_arcsec)
    threshold_rad = threshold_arcsec / 3600. * pi / 180.

    d = (v1[i1].dec - v2[i2].dec) / threshold_rad
    (abs(d) >= 1) && (return sign(d))

    maxd = max(abs(v1[i1].dec), abs(v2[i2].dec))
    if pi/2. - maxd > pi / 180. # avoid this optimization in regions below 1 deg from the poles
        d = abs(v1[i1].ra - v2[i2].ra)
        (d > pi) && (d = 2pi - d)
        d *= cos(maxd) / threshold_rad
        (d >= 1) && (return 999)
    end
    
    dd = gcirc(0, v1[i1].ra, v1[i1].dec, v2[i2].ra, v2[i2].dec)    
    (dd < threshold_rad) && (return 0)
    return 999
end
@time jj = sortmerge(c1, c2, lt1=lt, lt2=lt, sd=sd, 1.);

# Cross match using FlexiJoins:
@time result = innerjoin((c1, c2), by_distance(identity, separation, <(1u"arcsecond")));

```

To cross match two catalog, each with 10 million entries, SortMerge takes ~33 sec on average, while FlexiJoin provides some ~30% better performance with an average of ~24 s.

On the other hand, when the input catalogs are sorted by declination:

```julia
cs1 = c1[sortperm([r.dec for r in c1])];
cs2 = c2[sortperm([r.dec for r in c2])];

# Cross match using SortMerge:
@time jj = sortmerge(cs1, cs2, sd=sd, 1., sorted=true);

# Cross match using FlexiJoins:
@time result = innerjoin((cs1, cs2), by_distance(identity, separation, <(1u"arcsecond")));

```

SortMerge takes ~9.6 sec, while FlexiJoins is requires ~16 sec.

As discussed during the meeting, I will submit a PR to list SortMerge in the JuliaAstro ecosystem.

Thanks (and sorry for the slightly off-topic post).

---

<div class="post-metadata">

**Author:** ![Matt\_jl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matt_jl/32/52364_2.png) [@Matt\_jl](https://discourse.julialang.org/u/Matt_jl)\
**Post date:** [May 1, 2025, 12:40pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/3 "2025-05-01T12:40:02Z")

</div>

That’s awesome, but the documentation from FlexiJoin is not exactly clear. Would it be possible to return the indices of the matches rather than the matches themselves?

---

<div class="post-metadata">

**Author:** ![The\_Mastermage](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/the_mastermage/32/49444_2.png) [@The\_Mastermage](https://discourse.julialang.org/u/The_Mastermage)\
**Post date:** [May 1, 2025, 2:59pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/4 "2025-05-01T14:59:49Z")

</div>

Oh that’s super cool to know

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 1, 2025, 9:34pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/5 "2025-05-01T21:34:13Z")

</div>

> [@Matt\_jl](#):
>
> That’s awesome, but the documentation from FlexiJoin is not exactly clear. Would it be possible to return the indices of the matches rather than the matches themselves?

First of all, I cannot help but wonder why you need this 🙂 Would be curious to hear about the usecase, why indices and not elements themselves are needed.

Nevertheless indices are easy to retrieve – FlexiJoins.jl returns views of the original datasets:

```julia
julia> A = rand(100);
julia> B = rand(10);

julia> j = innerjoin((;A, B), by_distance(identity, Euclidean(), ≤(0.01)))
15-element StructArray(view(::Vector{Float64}, [21, 24, 24, 31, 47, 48, 50, 51, 58, 72, 75, 77, 90, 91, 91]), view(::Vector{Float64}, [6, 2, 4, 10, 7, 5, 9, 5, 6, 8, 6, 7, 7, 2, 4])) with eltype @NamedTuple{A::Float64, B::Float64}:
 (A = 0.2736830841831832, B = 0.2703319077489147)
 (A = 0.6382293710508645, B = 0.6413080028298531)
<...>

# individual element of the join result:
julia> j[1]
(A = 0.2736830841831832, B = 0.2703319077489147)

# indices of A in the join result:
julia> parentindices(j.A)
([21, 24, 24, 31, 47, 48, 50, 51, 58, 72, 75, 77, 90, 91, 91],)

```

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [May 1, 2025, 10:10pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/6 "2025-05-01T22:10:19Z")

</div>

`parentindices`, like `parent`, goes all the way up the ancestry tree, so the `parentindices` of a `view` of a `view` doesn’t necessarily give the indices into the joined arrays.

```julia
julia> parentindices(view(view([10, 20, 30, 40], 2:4), 1:2))
(2:3,)

```

So it sometimes will do what OP is asking for, but sometimes not quite.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 2, 2025, 6:00am UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/7 "2025-05-02T06:00:26Z")

</div>

Good point regarding nested views. There’s `joinindices(...)` with the same arguments as `innerjoin(...)` that returns indices instead of elements – in fact, all actual calculations happen within `joinindices`.  
If “getting indices of matches” is something generally useful, it can be streamlined and documented. I don’t think I ever needed/used it myself, so curious about the usecases @Matt_jl.

---

<div class="post-metadata">

**Author:** ![gcalderone](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gcalderone/32/1539_2.png) [@gcalderone](https://discourse.julialang.org/u/gcalderone)\
**Post date:** [May 5, 2025, 8:42am UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/9 "2025-05-05T08:42:13Z")

</div>

A use case is as follows: assume your tables are so large they don’t fit in RAM, you’ll need to

- load just the coordinates column;
- cross-match tables;
- load remaining columns only for matching entries.

---

<div class="post-metadata">

**Author:** ![cgarling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cgarling/32/29278_2.png) [@cgarling](https://discourse.julialang.org/u/cgarling)\
**Post date:** [August 28, 2025, 7:08pm UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/10 "2025-08-28T19:08:57Z")

</div>

To cover this use case I have put together [#68](https://github.com/JuliaAstro/SkyCoords.jl/pull/68) which adds a `NearestNeighbors.jl` extension to SkyCoords to provide basic catalog matching functionality. The main user-facing function is

```julia-auto
match(refcoords::AbstractArray{<:AbstractSkyCoords}, 
          matchcoords::AbstractArray{<:AbstractSkyCoords};
          nthneighbor::Int = 1)

```

which basically does the same thing as Astropy’s `match_coordinates_sky` if you are familiar with that API. It returns vectors of indices and separations for the `nthneighbor` to each coordinate in `matchcoords` found in `refcoords`.

Underlying it is a constructor for `NearestNeighbors.KDTree(data::AbstractArray{<:AbstractSkyCoords})` and I have extended the `nn`, `knn`, and `inrange` methods of that package to take `AbstractSkyCoords` inputs, so general nearest neighbor queries are supported. The `inrange` function is very similar to Astropy’s `search_around_sky` function, finding all neighbors within a given angular separation.

@gcalderone 's benchmark takes ~16 seconds with this implementation on my system:

```julia
julia> @time match(c1, c2)
 15.753582 seconds (40.00 M allocations: 2.388 GiB, 4.04% gc time)

```

Edit: This has been merged and is available in SkyCoords v1.6.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 2, 2025, 3:17am UTC](https://discourse.julialang.org/t/flexijoins-vs-sortmerge-particularly-in-astronomy-workflows/128632/11 "2025-09-02T03:17:20Z")

</div>

> [@cgarling](#):
>
> To cover this use case

Just to clarify: FlexiJoins.jl definitely supports this usecase, ie it has (and always has had 🙂 ) the ability to return indices. This is rarely needed, but the functionality is there as `joinindices(...)` – with the exact same interface as `*join(...)`.

* * *

Following the questions and discussion here and around, I added a separate page to FlexiJoins.jl docs demonstrating astronomy catalogs matching: [here](https://aplavin.github.io/FlexiJoins.jl/notebooks/skycoords.html). I work with astronomy catalogs a lot myself, and naturally they are well-supported in FlexiJoins.

[Main FlexiJoins docs](https://aplavin.github.io/FlexiJoins.jl/notebooks/examples.html) were also streamlined recently.  
I hope it helps 🙂