# Is this a bug of the Julia function "unique"?

**URL:** <https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141>\
**Category:** General Usage\
**Tags:** question\
**Created:** [October 21, 2021, 10:41am UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141 "2021-10-21T10:41:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [October 21, 2021, 10:41am UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/1 "2021-10-21T10:41:39Z")

</div>

I notice that any NaN values inside a data will mess up Julia’s unique function.

For example I have a matrix A with 3 duplicate rows:

```julia
A = [1 NaN 3; 1 NaN 3; 1 NaN 3];
3×3 Matrix{Float64}:
 1.0 NaN 3.0
 1.0 NaN 3.0
 1.0 NaN 3.0

```

If we apply `unique` to it as below:

```julia
B = unique(A, dims=1)

```

We would expect the B to be like this:

```julia
1×3 Matrix{Float64}:
 1.0 NaN 3.0

```

But in reality, the B generated by Julia is as below:

```julia
4×3 Matrix{Float64}:
 1.0 NaN 3.0
 1.0 NaN 3.0
 1.0 NaN 3.0
 1.0 NaN 3.0

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [October 21, 2021, 11:13am UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/2 "2021-10-21T11:13:19Z")

</div>

I suspect this is because

```julia
julia> NaN == NaN
false

```

, which is just regular floating point behavior. What I find more disturbing is this:

```julia
julia> unique(A, dims=2)
3×4 Matrix{Float64}:
 1.0 NaN NaN 3.0
 1.0 NaN NaN 3.0
 1.0 NaN NaN 3.0

```

That should definitely not happen. Do you mind opening an issue for this on the [issue tracker](https://github.com/JuliaLang/julia/issues)?

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [October 21, 2021, 11:13am UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/3 "2021-10-21T11:13:33Z")

</div>

Without words;)

```julia
julia> NaN == NaN
false

```

---

<div class="post-metadata">

**Author:** ![woclass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/woclass/32/212699_2.png) [@woclass](https://discourse.julialang.org/u/woclass)\
**Post date:** [October 21, 2021, 11:14am UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/4 "2021-10-21T11:14:06Z")

</div>

```julia
julia> A = [1 NaN 3; 1 NaN 3; 1 NaN 3]
3×3 Matrix{Float64}:
 1.0 NaN 3.0
 1.0 NaN 3.0
 1.0 NaN 3.0

julia> unique(A)
3-element Vector{Float64}:
   1.0
 NaN
   3.0

```

`:rofl:`

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [October 21, 2021, 11:17am UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/5 "2021-10-21T11:17:29Z")

</div>

The single-argument version probably just iterates and uses `isequal`, while the `dims` version is more complicated and creates a hash per dimension - it’s unclear to me whether that is intended behavior for the multidimensional version, though it does make sense when you take `NaN != NaN` into account 🤔

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [October 21, 2021, 12:05pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/6 "2021-10-21T12:05:07Z")

</div>

Will do.

Thanks for pointing to that direction.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 21, 2021, 12:15pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/7 "2021-10-21T12:15:08Z")

</div>

That’s what the IEEE 754 standard mandates: [floating point - What is the rationale for all comparisons returning false for IEEE754 NaN values? - Stack Overflow](https://stackoverflow.com/q/1565164/2442087).

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [October 21, 2021, 12:17pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/8 "2021-10-21T12:17:01Z")

</div>

Sorry for being sarcastic, I knew. Similar problem to SQL NULL values.

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [October 21, 2021, 1:01pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/9 "2021-10-21T13:01:07Z")

</div>

I don’t think it is sarcastic. We’re doing Julia a favor after all. Anyone who truly loves Julia would want to help get these issues fixed.

> [@Sukera](#):
>
> ```julia
> julia> NaN == NaN
> false
> 
> ```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 21, 2021, 1:06pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/10 "2021-10-21T13:06:24Z")

</div>

this `(NaN != NaN)` is NOT a bug, IEEE standards require it.

Don’t use `NaN` for this, use `nothing` maybe

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [October 21, 2021, 1:07pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/11 "2021-10-21T13:07:33Z")

</div>

No, this is expected behavior for `NaN`. That `unique(A, dims=1)` does not use `isequal` for this may be an issue, but `NaN != NaN` is very much intended. It works the same in other languages that use IEEE floating points, though their `unique` may have a different interpretation.

> **[NaN | Comparison with NaN](https://en.wikipedia.org/wiki/NaN#Comparison_with_NaN)**
>
> Comparisons are specified by the IEEE 754 standard to take into account possible NaN operands. When comparing two real numbers, or extended real numbers (as in the IEEE 754 floating-point formats), the first number may be either less than, equal to, or greater than the second number. This gives three possible relations. But when at least one operand of a comparison is NaN, this trichotomy does not apply, and a fourth relation is needed: unordered. In particular, two NaN values compare as unord As...

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [October 21, 2021, 1:10pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/12 "2021-10-21T13:10:32Z")

</div>

I can confirm that Matlab does not have the same “unique” issue, despite the fact that NaN is also considered different from NaN:

```julia
>> NaN == NaN
ans =
  logical
   0

```

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [October 21, 2021, 1:14pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/13 "2021-10-21T13:14:06Z")

</div>

Right - the question is whether julia should change its behavior here and whether that change would be breaking (meaning it could be done in 2.0 at the earliest).

Until then, may I ask what you were using that `NaN` for/how you encountered this? Julia has seperate `missing` and `nothing` values, to model the absence of a value (though one should exist, we just don’t know it) and the knowledge of absence (i.e. there is no value to represent the result). It does not rely on having to use `NaN` for a purpose it was never meant to be used for. See the docs for more information:

[https://docs.julialang.org/en/v1/manual/missing/](https://docs.julialang.org/en/v1/manual/missing/)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 21, 2021, 1:15pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/14 "2021-10-21T13:15:40Z")

</div>

That ` NaN == NaN` is false is maybe logical?  
_I.e., not being something does not imply being the same thing. Example_:

```julia
NaN == NaN # false 
x = 0/0 # NaN
y = Inf/Inf # NaN
x == y # false --> nice

```

---

<div class="post-metadata">

**Author:** ![leon](https://avatars.discourse-cdn.com/v4/letter/l/dc4da7/32.png) [@leon](https://discourse.julialang.org/u/leon)\
**Post date:** [October 21, 2021, 1:34pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/15 "2021-10-21T13:34:42Z")

</div>

I’m processing some oceanographic data right now. When people do a CTD cast at a particular sampling station, there is a Cast number associated with it. Sometimes, people leave it blank when there is only a cast.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 21, 2021, 1:37pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/16 "2021-10-21T13:37:33Z")

</div>

So you’re saying that you’re using `NaN` to represent “blank” values?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 21, 2021, 1:39pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/17 "2021-10-21T13:39:45Z")

</div>

It is a bug because `unique` is documented to use `isequal` and

```julia
julia> isequal([1, NaN, 3], [1, NaN, 3])
true

```

[https://github.com/JuliaLang/julia/pull/42737](https://github.com/JuliaLang/julia/pull/42737)

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [October 21, 2021, 1:42pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/18 "2021-10-21T13:42:11Z")

</div>

Great! Then I’ll also open an issue about the `unique(A, dims=2)` oddity.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [October 21, 2021, 1:42pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/19 "2021-10-21T13:42:29Z")

</div>

> [@Sukera](#):
>
> `unique(A, dims=2)`

Isn’t that also fixed by the PR?

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [October 21, 2021, 1:44pm UTC](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141/20 "2021-10-21T13:44:10Z")

</div>

> [@kristoffer.carlsson](#):
>
> Isn’t that also fixed by the PR?

I haven’t checked, but the array _grows_ for that case:

```julia
julia> A = [1 NaN 3; 1 NaN 3; 1 NaN 3]
3×3 Matrix{Float64}:
 1.0 NaN 3.0
 1.0 NaN 3.0
 1.0 NaN 3.0

julia> unique(A, dims=2)
3×4 Matrix{Float64}:
 1.0 NaN NaN 3.0
 1.0 NaN NaN 3.0
 1.0 NaN NaN 3.0

```

and the unit test doesn’t cover that 🤷‍♂️ I guess I don’t see why `==` vs `isequal` should produce _more_ values per row than existed previously.

[Next page](https://discourse.julialang.org/t/is-this-a-bug-of-the-julia-function-unique/70141.md?page=2)