# Set-like operations on DataFrames? **URL:** https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004 **Category:** Data **Tags:** question, dataframes **Created:** [December 8, 2019, 11:16am UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004 "2019-12-08T11:16:39Z") **Posts on this page:** 14 **Page:** 1

### Author: ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee) #### Post date: [December 8, 2019, 11:16am UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/1 "2019-12-08T11:16:39Z")

How to merge two dataframes with the same columns, similar to set operations? For example, whether there are functions like ```julia df = union(df1, df2, on=:ID) df = intersect(df1, df2, on=:ID) df = diff(df1, df2, on=:ID) ``` ---

### Author: ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins) #### Post date: [December 8, 2019, 4:14pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/2 "2019-12-08T16:14:33Z")

Can you please explain what you mean by “similar to set operations”? Can you please give some example and an expected output? ---

### Author: ![bashonubuntu](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bashonubuntu](https://discourse.julialang.org/u/bashonubuntu) #### Post date: [December 8, 2019, 4:48pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/3 "2019-12-08T16:48:01Z")

You can look up sql-styled joins in julia which will answer some of your questions. [https://juliadata.github.io/DataFrames.jl/stable/man/joins/](https://juliadata.github.io/DataFrames.jl/stable/man/joins/) An intersect is basically an `inner-join.` A union can be accomplished by `hcat().` What are you trying to accomplish by diff? ---

### Author: ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee) #### Post date: [December 10, 2019, 5:11am UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/4 "2019-12-10T05:11:15Z")

Treat each row as an element in a set. This is viable because we assume the data has unique keys. Then perform set like operations like ```julia julia> a = Set([1 2 3]) Set([2, 3, 1]) julia> b = Set([2 3 4]) Set([4, 2, 3]) julia> union(a, b) Set([4, 2, 3, 1]) julia> intersect(a, b) Set([2, 3]) julia> setdiff(a, b) Set([1]) ``` Note: replace the elements to rows of a dataframe. ---

### Author: ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee) #### Post date: [December 10, 2019, 5:13am UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/5 "2019-12-10T05:13:53Z")

Thanks but its not sql-styled joins. It’s set-styled operations. I gave examples in the above reply 🙂 ---

### Author: ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins) #### Post date: [December 10, 2019, 7:45am UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/6 "2019-12-10T07:45:57Z")

This is what I assumed but your definition has `on` keyword argument which I do not understand in the context of your answer. Without `on` argument you can do e.g.: ```julia DataFrame(union(eachrow.([df1, df2])...)) ``` or ```julia DataFrame(union(Tables.rowtable.([df1, df2])...)) ``` (and the same with `intersect` and `diff`) ---

### Author: ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee) #### Post date: [December 10, 2019, 12:28pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/7 "2019-12-10T12:28:47Z")

This is clever! Didn’t know the `union` can operate on iterators. Cool!! ---

### Author: ![bertulli](https://avatars.discourse-cdn.com/v4/letter/b/ecc23a/32.png) [@bertulli](https://discourse.julialang.org/u/bertulli) #### Post date: [January 24, 2023, 9:56pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/8 "2023-01-24T21:56:56Z")

This seems not to be working anymore: ```julia df_err = DataFrame(diff(eachrow.([df_diff_dest, df_same_dest]))) ``` (where `df_diff_dest` and `df_same_dest` are grouped and filtered DataFrames) seems to throw ```julia ERROR: MethodError: no method matching -(::DataFrameRow{DataFrame, DataFrames.Index}, ::DataFrameRow{DataFrame, DataFrames.Index}) Closest candidates are: -(::ChainRulesCore.AbstractThunk, ::Any) at ~/.julia/packages/ChainRulesCore/a4mIA/src/tangent_types/thunks.jl:34 -(::DataValues.DataValue{T1}, ::T2) where {T1, T2} at ~/.julia/packages/DataValues/N7oeL/src/scalar/operations.jl:65 -(::ChainRulesCore.ZeroTangent, ::Any) at ~/.julia/packages/ChainRulesCore/a4mIA/src/tangent_arithmetic.jl:101 ... Stacktrace: [1] _broadcast_getindex_evalf @ ./broadcast.jl:670 [inlined] [2] _broadcast_getindex @ ./broadcast.jl:643 [inlined] [3] getindex @ ./broadcast.jl:597 [inlined] [4] copy @ ./broadcast.jl:899 [inlined] [5] materialize @ ./broadcast.jl:860 [inlined] [6] broadcast_preserving_zero_d @ ./broadcast.jl:849 [inlined] [7] -(A::DataFrames.DataFrameRows{DataFrame}, B::DataFrames.DataFrameRows{DataFrame}) @ Base ./arraymath.jl:8 [8] _broadcast_getindex_evalf @ ./broadcast.jl:670 [inlined] [9] _broadcast_getindex @ ./broadcast.jl:643 [inlined] [10] getindex @ ./broadcast.jl:597 [inlined] [11] macro expansion @ ./broadcast.jl:961 [inlined] [12] macro expansion @ ./simdloop.jl:77 [inlined] [13] copyto! @ ./broadcast.jl:960 [inlined] [14] copyto! @ ./broadcast.jl:913 [inlined] [15] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Tuple{Base.OneTo{Int64}}, typeof(-), Tuple{SubArray{DataFrames.DataFrameRows{DataFrame}, 1, Vector{DataFrames.DataFrameRows{DataFrame}}, Tuple{UnitRange{Int64}}, true}, SubArray{DataFrames.DataFrameRows{DataFrame}, 1, Vector{DataFrames.DataFrameRows{DataFrame}}, Tuple{UnitRange{Int64}}, true}}}) @ Base.Broadcast ./broadcast.jl:885 [16] materialize @ ./broadcast.jl:860 [inlined] [17] diff(a::Vector{DataFrames.DataFrameRows{DataFrame}}; dims::Int64) @ Base ./multidimensional.jl:1005 [18] diff(a::Vector{DataFrames.DataFrameRows{DataFrame}}) @ Base ./multidimensional.jl:965 [19] top-level scope @ ./REPL[36]:29 ``` Has the API changed or did I do something wrong? ---

### Author: ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach) #### Post date: [January 24, 2023, 10:07pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/9 "2023-01-24T22:07:41Z")

you want `setdiff`, not `diff`. ---

### Author: ![bertulli](https://avatars.discourse-cdn.com/v4/letter/b/ecc23a/32.png) [@bertulli](https://discourse.julialang.org/u/bertulli) #### Post date: [January 24, 2023, 10:12pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/10 "2023-01-24T22:12:49Z")

It asked me to add the keyword `:auto` as second argument to the constructor `DataFrame`, but even like that I got errors (and moreover it doesn’t show **what** are the rows): ```julia Adc instruction 53×2 DataFrame Row │ x1 x2 │ DataFram… DataFram… ─────┼──────────────────────────────────────────────── 1 │ 3-element DataFrameRow 3-element DataFrameRow 2 │ 3-element DataFrameRow 3-element DataFrameRow 3 │ 3-element DataFrameRow 3-element DataFrameRow 4 │ 3-element DataFrameRow 3-element DataFrameRow 5 │ 3-element DataFrameRow 3-element DataFrameRow 6 │ 3-element DataFrameRow 3-element DataFrameRow 7 │ 3-element DataFrameRow 3-element DataFrameRow 8 │ 3-element DataFrameRow 3-element DataFrameRow 9 │ 3-element DataFrameRow 3-element DataFrameRow 10 │ 3-element DataFrameRow 3-element DataFrameRow 11 │ 3-element DataFrameRow 3-element DataFrameRow 12 │ 3-element DataFrameRow 3-element DataFrameRow 13 │ 3-element DataFrameRow 3-element DataFrameRow 14 │ 3-element DataFrameRow 3-element DataFrameRow 15 │ 3-element DataFrameRow 3-element DataFrameRow 16 │ 3-element DataFrameRow 3-element DataFrameRow 17 │ 3-element DataFrameRow 3-element DataFrameRow 18 │ 3-element DataFrameRow 3-element DataFrameRow 19 │ 3-element DataFrameRow 3-element DataFrameRow 20 │ 3-element DataFrameRow 3-element DataFrameRow 21 │ 3-element DataFrameRow 3-element DataFrameRow 22 │ 3-element DataFrameRow 3-element DataFrameRow 23 │ 3-element DataFrameRow 3-element DataFrameRow 24 │ 3-element DataFrameRow 3-element DataFrameRow 25 │ 3-element DataFrameRow 3-element DataFrameRow 26 │ 3-element DataFrameRow 3-element DataFrameRow 27 │ 3-element DataFrameRow 3-element DataFrameRow 28 │ 3-element DataFrameRow 3-element DataFrameRow 29 │ 3-element DataFrameRow 3-element DataFrameRow 30 │ 3-element DataFrameRow 3-element DataFrameRow 31 │ 3-element DataFrameRow 3-element DataFrameRow 32 │ 3-element DataFrameRow 3-element DataFrameRow 33 │ 3-element DataFrameRow 3-element DataFrameRow 34 │ 3-element DataFrameRow 3-element DataFrameRow 35 │ 3-element DataFrameRow 3-element DataFrameRow 36 │ 3-element DataFrameRow 3-element DataFrameRow 37 │ 3-element DataFrameRow 3-element DataFrameRow 38 │ 3-element DataFrameRow 3-element DataFrameRow 39 │ 3-element DataFrameRow 3-element DataFrameRow 40 │ 3-element DataFrameRow 3-element DataFrameRow 41 │ 3-element DataFrameRow 3-element DataFrameRow 42 │ 3-element DataFrameRow 3-element DataFrameRow 43 │ 3-element DataFrameRow 3-element DataFrameRow 44 │ 3-element DataFrameRow 3-element DataFrameRow 45 │ 3-element DataFrameRow 3-element DataFrameRow 46 │ 3-element DataFrameRow 3-element DataFrameRow 47 │ 3-element DataFrameRow 3-element DataFrameRow 48 │ 3-element DataFrameRow 3-element DataFrameRow 49 │ 3-element DataFrameRow 3-element DataFrameRow 50 │ 3-element DataFrameRow 3-element DataFrameRow 51 │ 3-element DataFrameRow 3-element DataFrameRow 52 │ 3-element DataFrameRow 3-element DataFrameRow 53 │ 3-element DataFrameRow 3-element DataFrameRow Adcs instruction 53×2 DataFrame Row │ x1 x2 │ DataFram… DataFram… ─────┼──────────────────────────────────────────────── 1 │ 3-element DataFrameRow 3-element DataFrameRow 2 │ 3-element DataFrameRow 3-element DataFrameRow 3 │ 3-element DataFrameRow 3-element DataFrameRow 4 │ 3-element DataFrameRow 3-element DataFrameRow 5 │ 3-element DataFrameRow 3-element DataFrameRow 6 │ 3-element DataFrameRow 3-element DataFrameRow 7 │ 3-element DataFrameRow 3-element DataFrameRow 8 │ 3-element DataFrameRow 3-element DataFrameRow 9 │ 3-element DataFrameRow 3-element DataFrameRow 10 │ 3-element DataFrameRow 3-element DataFrameRow 11 │ 3-element DataFrameRow 3-element DataFrameRow 12 │ 3-element DataFrameRow 3-element DataFrameRow 13 │ 3-element DataFrameRow 3-element DataFrameRow 14 │ 3-element DataFrameRow 3-element DataFrameRow 15 │ 3-element DataFrameRow 3-element DataFrameRow 16 │ 3-element DataFrameRow 3-element DataFrameRow 17 │ 3-element DataFrameRow 3-element DataFrameRow 18 │ 3-element DataFrameRow 3-element DataFrameRow 19 │ 3-element DataFrameRow 3-element DataFrameRow 20 │ 3-element DataFrameRow 3-element DataFrameRow 21 │ 3-element DataFrameRow 3-element DataFrameRow 22 │ 3-element DataFrameRow 3-element DataFrameRow 23 │ 3-element DataFrameRow 3-element DataFrameRow 24 │ 3-element DataFrameRow 3-element DataFrameRow 25 │ 3-element DataFrameRow 3-element DataFrameRow 26 │ 3-element DataFrameRow 3-element DataFrameRow 27 │ 3-element DataFrameRow 3-element DataFrameRow 28 │ 3-element DataFrameRow 3-element DataFrameRow 29 │ 3-element DataFrameRow 3-element DataFrameRow 30 │ 3-element DataFrameRow 3-element DataFrameRow 31 │ 3-element DataFrameRow 3-element DataFrameRow 32 │ 3-element DataFrameRow 3-element DataFrameRow 33 │ 3-element DataFrameRow 3-element DataFrameRow 34 │ 3-element DataFrameRow 3-element DataFrameRow 35 │ 3-element DataFrameRow 3-element DataFrameRow 36 │ 3-element DataFrameRow 3-element DataFrameRow 37 │ 3-element DataFrameRow 3-element DataFrameRow 38 │ 3-element DataFrameRow 3-element DataFrameRow 39 │ 3-element DataFrameRow 3-element DataFrameRow 40 │ 3-element DataFrameRow 3-element DataFrameRow 41 │ 3-element DataFrameRow 3-element DataFrameRow 42 │ 3-element DataFrameRow 3-element DataFrameRow 43 │ 3-element DataFrameRow 3-element DataFrameRow 44 │ 3-element DataFrameRow 3-element DataFrameRow 45 │ 3-element DataFrameRow 3-element DataFrameRow 46 │ 3-element DataFrameRow 3-element DataFrameRow 47 │ 3-element DataFrameRow 3-element DataFrameRow 48 │ 3-element DataFrameRow 3-element DataFrameRow 49 │ 3-element DataFrameRow 3-element DataFrameRow 50 │ 3-element DataFrameRow 3-element DataFrameRow 51 │ 3-element DataFrameRow 3-element DataFrameRow 52 │ 3-element DataFrameRow 3-element DataFrameRow 53 │ 3-element DataFrameRow 3-element DataFrameRow Add instruction ERROR: DimensionMismatch: column :x1 has length 661 and column :x2 has length 541Stacktrace: [1] DataFrame(columns::Vector{AbstractVector}, colindex::DataFrames.Index; copycols::Bool) @ DataFrames ~/.julia/packages/DataFrames/dgZn3/src/dataframe/dataframe.jl:206 [2] DataFrame(columns::Vector{DataFrames.DataFrameRows{DataFrame}}, cnames::Vector{Symbol}; makeunique::Bool, copycols::Bool) @ DataFrames ~/.julia/packages/DataFrames/dgZn3/src/dataframe/dataframe.jl:352 [3] #DataFrame#218 @ ~/.julia/packages/DataFrames/dgZn3/src/dataframe/dataframe.jl:368 [inlined] [4] DataFrame(columns::Vector{DataFrames.DataFrameRows{DataFrame}}, cnames::Symbol) @ DataFrames ~/.julia/packages/DataFrames/dgZn3/src/dataframe/dataframe.jl:361 [5] top-level scope @ ./REPL[8]:29 ``` ---

### Author: ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins) #### Post date: [January 24, 2023, 10:26pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/11 "2023-01-24T22:26:57Z")

> [@bertulli](#): > > where `df_diff_dest` and `df_same_dest` are grouped and filtered DataFrames Can you show a full reproducible example. The code works for me: ```julia julia> df1 = DataFrame(a=[1,2], b=[3,4]) 2×2 DataFrame Row │ a b │ Int64 Int64 ─────┼────────────── 1 │ 1 3 2 │ 2 4 julia> df2 = DataFrame(a=[1,3], b=[3,5]) 2×2 DataFrame Row │ a b │ Int64 Int64 ─────┼────────────── 1 │ 1 3 2 │ 3 5 julia> DataFrame(union(eachrow.([df1, df2])...)) 3×2 DataFrame Row │ a b │ Int64 Int64 ─────┼────────────── 1 │ 1 3 2 │ 2 4 3 │ 3 5 julia> DataFrame(setdiff(eachrow.([df1, df2])...)) 1×2 DataFrame Row │ a b │ Int64 Int64 ─────┼────────────── 1 │ 2 4 ``` (most likely you are not passing correct values) ---

### Author: ![bertulli](https://avatars.discourse-cdn.com/v4/letter/b/ecc23a/32.png) [@bertulli](https://discourse.julialang.org/u/bertulli) #### Post date: [January 26, 2023, 1:33pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/12 "2023-01-26T13:33:40Z")

You’re right, I didn’t put the ellipsis and instead trusted to REPL to add `:auto`. Now it works, sorry for the bother. Anyway, please confirm me something (that I couldn’t guess from the docs): when calling `DataFrame(setdiff(eachrow.([df1, df2, df3, df4, df5])...))`, what is computed is (set-like) DF1 \setminus DF2 \setminus DF3 \setminus DF4 \setminus DF5 = DF1 \setminus (DF2 \cup DF3 \cup DF4 \cup DF5) ? ---

### Author: ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins) #### Post date: [January 26, 2023, 2:41pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/13 "2023-01-26T14:41:26Z")

This is unrelated to DataFrames.jl, but is part of `setdiff` from Base Julia, where you have: > setdiff(s, itrs…) > > Construct the set of elements in s but not in any of the iterables in itrs. So you are right 😄: ```julia julia> setdiff(1:6, 2, 4, 6:10) 3-element Vector{Int64}: 1 3 5 ``` ---

### Author: ![bertulli](https://avatars.discourse-cdn.com/v4/letter/b/ecc23a/32.png) [@bertulli](https://discourse.julialang.org/u/bertulli) #### Post date: [February 1, 2023, 12:08pm UTC](https://discourse.julialang.org/t/set-like-operations-on-dataframes/32004/14 "2023-02-01T12:08:32Z")

Many thanks! 😄 🙏 🖖