# Delete duplicate rows in a DataFrame

**URL:** <https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487>\
**Category:** New to Julia\
**Tags:** dataframes\
**Created:** [July 29, 2021, 10:06am UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487 "2021-07-29T10:06:52Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nash](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nash/32/14482_2.png) [@Nash](https://discourse.julialang.org/u/Nash)\
**Post date:** [July 29, 2021, 10:06am UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/1 "2021-07-29T10:06:52Z")

</div>

In a post on Stackoverflow about identifying duplicate rows in a DataFrame [[Stackoverflow Post]](https://stackoverflow.com/questions/45021431/julia-find-duplicate-rows-in-dataframes), Dominykas Mostauskis made a side-comment about how to remove the rows identified. Unfortunately, the comment wasn’t that instructive for beginners.

So, how exactly does one go about removing identified unique rows from a DataFrame permanently?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [July 29, 2021, 10:10am UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/2 "2021-07-29T10:10:24Z")

</div>

Assuming you don’t actually want to _identify_ the rows which are duplicates and therefore removed, you can just use `unique`?

```julia

julia> using DataFrames

julia> df = DataFrame(a = rand(1:3, 10), b = rand(1:3, 10))
10×2 DataFrame
 Row │ a b     
     │ Int64 Int64 
─────┼──────────────
   1 │ 1 3
   2 │ 3 1
   3 │ 2 1
   4 │ 1 3
   5 │ 1 1
   6 │ 1 1
   7 │ 3 1
   8 │ 2 2
   9 │ 3 1
  10 │ 1 2

julia> unique(df)
6×2 DataFrame
 Row │ a b     
     │ Int64 Int64 
─────┼──────────────
   1 │ 1 3
   2 │ 3 1
   3 │ 2 1
   4 │ 1 1
   5 │ 2 2
   6 │ 1 2

```

---

<div class="post-metadata">

**Author:** ![sai\_matcha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sai_matcha/32/35801_2.png) [@sai\_matcha](https://discourse.julialang.org/u/sai_matcha)\
**Post date:** [February 14, 2022, 5:50pm UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/3 "2022-02-14T17:50:01Z")

</div>

is there a way to get non unique dataframe ? thank you.

---

<div class="post-metadata">

**Author:** ![sai\_matcha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sai_matcha/32/35801_2.png) [@sai\_matcha](https://discourse.julialang.org/u/sai_matcha)\
**Post date:** [February 14, 2022, 6:26pm UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/4 "2022-02-14T18:26:45Z")

</div>

i have found a way to do this.

```julia
df2 = transform(df,nonunique)
df3 = filter(r -> r.x1 != 0, df2)

```

df3 gives data frame of non unique rows

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 14, 2022, 6:36pm UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/5 "2022-02-14T18:36:44Z")

</div>

This will give you exactly one instance of each duplicated row. So you won’t know if a row is duplicated 5 times or 500 times. This may or may not be what you want, though.

---

<div class="post-metadata">

**Author:** ![sai\_matcha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sai_matcha/32/35801_2.png) [@sai\_matcha](https://discourse.julialang.org/u/sai_matcha)\
**Post date:** [February 14, 2022, 6:43pm UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/6 "2022-02-14T18:43:59Z")

</div>

is there a way to get whole rows as dataframe ?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 14, 2022, 6:49pm UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/7 "2022-02-14T18:49:32Z")

</div>

This might be very slow for data sets with lots of columns, but you could try this:

```julia
julia> df = DataFrame(a = [1, 1, 2, 4], b = [10, 10, 25, 35]);

julia> combine(groupby(df, :)) do sdf
           if nrow(sdf) == 1 
               DataFrame()
           else
               sdf
           end
       end
2×2 DataFrame
 Row │ a b     
     │ Int64 Int64 
─────┼──────────────
   1 │ 1 10
   2 │ 1 10

```

---

<div class="post-metadata">

**Author:** ![sai\_matcha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sai_matcha/32/35801_2.png) [@sai\_matcha](https://discourse.julialang.org/u/sai_matcha)\
**Post date:** [February 14, 2022, 7:02pm UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/8 "2022-02-14T19:02:50Z")

</div>

thankyou @pdeffebach

---

<div class="post-metadata">

**Author:** ![bert](https://avatars.discourse-cdn.com/v4/letter/b/9e8a1a/32.png) [@bert](https://discourse.julialang.org/u/bert)\
**Post date:** [November 23, 2022, 1:25am UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/9 "2022-11-23T01:25:27Z")

</div>

Is there a way to keep either the first or last duplicate? Like Python’s pandas [drop\_duplicates](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html):

> **keep** {‘first’, ‘last’, False}, default ‘first’
> 
> Determines which duplicates (if any) to keep. - `first` : Drop duplicates except for the first occurrence. - `last` : Drop duplicates except for the last occurrence. - False : Drop all duplicates.

---

<div class="post-metadata">

**Author:** ![Dan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dan/32/42581_2.png) [@Dan](https://discourse.julialang.org/u/Dan)\
**Post date:** [November 23, 2022, 2:30am UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/10 "2022-11-23T02:30:10Z")

</div>

```julia
df = DataFrame(a = [1, 1, 2, 4], b = [10, 10, 25, 35], c = [2,3,5,7])

function dropduplicates(df, cols; keep = :first)
    keep in [:first, :last] || error("keep parameter should be :first or :last")
    combine(groupby(df, cols)) do sdf
        if nrow(sdf) == 1 
            DataFrame()
        else
            DataFrame(
              filter(
                r->rownumber(r)==(keep == :first ? 1 : nrow(sdf)), 
                eachrow(sdf)
              )
            )
        end
    end
end

```

Giving:

```julia
julia> df
4×3 DataFrame
 Row │ a b c     
     │ Int64 Int64 Int64 
─────┼─────────────────────
   1 │ 1 10 2
   2 │ 1 10 3
   3 │ 2 25 5
   4 │ 4 35 7

julia> dropduplicates(df, [:a, :b])
1×3 DataFrame
 Row │ a b c     
     │ Int64 Int64 Int64 
─────┼─────────────────────
   1 │ 1 10 2

julia> dropduplicates(df, [:a, :b]; keep = :last)
1×3 DataFrame
 Row │ a b c     
     │ Int64 Int64 Int64 
─────┼─────────────────────
   1 │ 1 10 3

```

(maybe this is not the most efficient way)

---

<div class="post-metadata">

**Author:** ![hhaensel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hhaensel/32/1207_2.png) [@hhaensel](https://discourse.julialang.org/u/hhaensel)\
**Post date:** [June 22, 2023, 6:11am UTC](https://discourse.julialang.org/t/delete-duplicate-rows-in-a-dataframe/65487/11 "2023-06-22T06:11:37Z")

</div>

Just to add on that, `unique()` supports - perhaps meanwhile - `cols` and `keep`, and it is very fast as it creates a view on rows by default.