# How to efficiently find columns of the matrix which are the same?

**URL:** <https://discourse.julialang.org/t/how-to-efficiently-find-columns-of-the-matrix-which-are-the-same/105873>\
**Category:** New to Julia\
**Tags:** question, optimization\
**Created:** [November 6, 2023, 6:56pm UTC](https://discourse.julialang.org/t/how-to-efficiently-find-columns-of-the-matrix-which-are-the-same/105873 "2023-11-06T18:56:47Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 6, 2023, 7:27pm UTC](https://discourse.julialang.org/t/how-to-efficiently-find-columns-of-the-matrix-which-are-the-same/105873/3 "2023-11-06T19:27:49Z")

</div>

> [@dgsob](#):
>
> Is there a more efficient way to search for something like this in Julia than these nasty nested loops?

How big are your matrices? `length(unique(eachcol(matrix)))` is almost 500x faster than your version on my machine for a random 1000x1000 matrix (i.e. all unequal columns). (And that’s if I put your code [in a function](https://docs.julialang.org/en/v1/manual/performance-tips/#Performance-critical-code-should-be-inside-a-function); in global scope your code is even worse.) This is the number of unique columns — I’m not entirely clear on what you are trying to compute, but the number of columns which are duplicates is `size(matrix, 2) - number_unique_cols`.

There’s nothing wrong with loops in performance-critical Julia code, it’s just that you are using a suboptimal algorithm. The basic reason is that `unique` constructs a `Set` of columns, which uses a hash table so it avoids comparing whole columns in the common case of unequal columns with distinct hashes. So, the time is roughly linear in the number of columns rather than quadratic as in your algorithm. (Also, it uses column [views rather than copied slices](https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-views), and it avoids allocating unnecessary intermediate arrays like the result of your `.==` operation.)

You can also use `size(unique(matrix, dims=2), 2)`, but that is slower because it allocates a whole new matrix to store the result of `unique`, rather than an array of views as in `unique(eachcol(matrix))`.

> [@jstrube](#):
>
> Depending on the data, it may be faster to compute a hash value for each column, which then only needs a lookup of duplicate hashes.

`unique` does this for you. (If you implement this strategy yourself, you have to be wary of accidental hash collisions.)

See also [Finding unique columns of a matrix](https://discourse.julialang.org/t/finding-unique-coluns-of-a-matrix/57421)

PS. Note that you can compare two columns of a matrix for equality with `A[:,i] == A[:,j]` rather than using `all(A[:,i] .== A[:,j])` which allocates an array of booleans first (from `.==`) and then checks that they are all true. And put `@views` in front (or better yet, on a whole block of code like a whole function) to make slices return views rather than copies.

---

_[View the full topic](https://discourse.julialang.org/t/how-to-efficiently-find-columns-of-the-matrix-which-are-the-same/105873)._