# Need memory and/or parallel optimization for pairwise distance metrics

**URL:** <https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943>\
**Category:** Performance\
**Created:** [November 17, 2025, 5:18pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943 "2025-11-17T17:18:04Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [November 18, 2025, 7:48pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943/21 "2025-11-18T19:48:34Z")

</div>

> [@kevbonham](#):
>
> Yeah, essentially. In my actual use case, it’s a sparse boolean matrix (presence/absence of microbial genes in stool samples)

In this case I’d recommend using a vector of BitSet as input, where the BitSet contain the indices that are set. These use only 1 bit per entry and are very fast to compare for dissimilarity (just bitwise `xor` followed by a `count_ones` which is a single assembly instruction).

The dissimilarity would be just:

```julia-auto
dissimilarity(set1, set2) = length(union(set1, set2)) - length(intersection(set1, set2))

```

Which probably doesn’t quite compile down to the aforementioned xor but should be rather fast nonetheless. Total input size would be 1kx800k bits = 100MB

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [November 18, 2025, 11:09pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943/22 "2025-11-18T23:09:27Z")

</div>

> [@kevbonham](#):
>
> I keep hitting OOM on my laptop

Try counting the nonzero entries first.

That is, show us the output of

```julia-auto
@views function sparse_distance_count(mat, df=jaccard)
    lk = ReentrantLock()
    counter_all = Ref(0)
    @sync for ic in chunks(1:size(mat,2); n=Threads.nthreads()) 
        @spawn begin
            counter = 0
            for i in ic
                for j in axes(mat, 2)
                    i >= j && continue
                    j % 100_000 == 0 && @info i, j
                    d = df(mat[:, i], mat[:, j])
                    d == 1 && continue
                    counter += 1
                end
            end
            lock(lk) do 
                counter_all[] += counter
            end
        end
    end
    return nnz(mat), counter_all[]
end

```

together with info whether the timing is acceptable to you, and how much RAM you have on your laptop.

You can definitely stream the output to disk. To do that, you simply need to write a file once you have enough data for it. For example, you could do

```julia-auto
@views function sparse_distance(mat, dist_func=jaccard)
    @sync for ic in chunks(1:size(mat,2); n=round(Int, size(mat,2)/1000)) 
        suffix = 1
        filename = "dmats/chunk_$ic_$suffix.csv"

        isfile(filename) && continue
        @spawn begin
            is = Int[]; js = Int[]; vs = Float64[]
            for i in ic
                for j in axes(mat, 2)
                    i >= j && continue
                    j % 100_000 == 0 && @info i,j
                    d = dist_func(mat[:, i], mat[:, j])
                    d == 1. && continue
                    push!(is, i)
                    push!(js, j)
                    push!(vs, 1. - d)
                    if length(is) > 10_000_000
                        CSV.write(filename, (; i=is, j=js, v=vs))
                        empty!(is); empty!(js); empty!(vs);
                        suffix += 1
                        filename = "dmats/chunk_$ic_$suffix.csv"
                    end
                end
            end
            isempty(is) || CSV.write(filename, (; i=is, j=js, v=vs))
        end
    end
    return nothing
end

```

Even better would be to use a streaming interface of CSV.jl, but I don’t think that exists.

> [@abraemer](#):
>
> I just googled this and it seems that Jaccard distances boils down to counting the coordinates that are identical between 2 vectors

Not quite; per Distances.jl Readme.md, it’s supposed to compute `1 - sum(min(x, y)) / sum(max(x, y))`; and it actually does compute that if all entries are nonnegative (they are nonnegative, right?).

Jaccard should be exactly one if and only if the two vectors have disjoint support. And by my superficial reading of Distances.jl that should also hold for floating point, and the compiler should not have leeway to break that property by re-associating; so exact comparison to 1 should be correct, unless somebody sneaked in a dirty `@simd` or `@fastmath` or something like that. (floating point does guarantee that `x-x==0.0` and `x+0 == x` exactly, and that’s all we should need)

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [November 19, 2025, 5:15pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943/23 "2025-11-19T17:15:55Z")

</div>

Alright, thanks for the help everyone! Here’s the final version - this worked to keep the memory footprint from exploding. I did a little file optimization too, since the outputs were pretty large - I removed the `x` vector (also saves some allocations, I guess), since each file is now a single index (and can be recovered from the filename), and I rounded the floats to 4 digits to save some bytes. This is 242Gb of files 😬

```julia-auto
@views function sparse_distance(mat, dist_func=jaccard)
    lk = ReentrantLock()
    for i in axes(mat, 2)
        filename = "dmats/chunk_$i.csv"
        isfile(filename) && continue
        i % 100 == 0 && @info "Column $i"
        # create file for this i
        yi,zi = Int32[], Float32[]
        @sync for jc in chunks(axes(mat, 2); n=Threads.nthreads())
            @spawn begin
                y, z = Int32[], Float32[]
                for j in jc
                    i >= j && continue
                    d = dist_func(mat[:, i], mat[:, j])
                    push!(y, j)
                    push!(z, round(1 - d; digits=4))
                end
                lock(lk) do
                    append!(yi, y)
                    append!(zi, z)
                end
            end
        end
        CSV.write(filename, (; j=yi, v=zi))
    end
    return nothing
end

```

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [November 19, 2025, 7:53pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943/24 "2025-11-19T19:53:12Z")

</div>

> [@kevbonham](#):
>
> I rounded the floats to 4 digits to save some bytes.

If you don’t need to have them ‘human-readable’, it is probably easier to use some binary storage format. JLD2.jl makes that straight-forward. Just replace

```julia-auto
CSV.write(filename, (; j=yi, v=zi))

```

with

```julia-auto
JLD2.save(filename; j=yi, v=zi)

```

Then you don’t need to round 🙂

To load the data back, you can do

```julia-auto
j, v = JLD2.load(filename, "j", "v")

```

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [November 19, 2025, 8:14pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943/25 "2025-11-19T20:14:31Z")

</div>

Heh, good thinking! I have a student working on the outputs they’re using python to do the downstream stuff, so this will have to do for now. We’re just working on a prototype model, but this is worth considering for when/if it works and we need to do more “production” models.

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [November 19, 2025, 8:17pm UTC](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943/26 "2025-11-19T20:17:51Z")

</div>

Actually JLD2 is HDF5 compatible, so you could use the files from Python as well 🙂 should take just a few mins to figure out how exactly the info needs to be extracted using `h5py`.

[Previous page](https://discourse.julialang.org/t/need-memory-and-or-parallel-optimization-for-pairwise-distance-metrics/133943.md?page=1)