Something that has more options than the == in Base for comparing dataframes? E.g. one where you can set the tolerance of difference in a float column. And also can compare columns by name even though they may be arranged in different orders? Something like proc compare in SAS.

Not aware of anything baked in, but seems easy enough to do using isapprox? You’d just have to decide whether you want to decide equality elementwise, columnwise or across the whole DataFrame (i.e. if say you have an absolute tolerance of 0.1, do you consider two dfs equal if all elements are withi…

[image] nilshg: isapprox.(df, df2, atol = 0.15) isapprox unfortunately fails if the column is for instance of DateTime type. It would be really great if there were a function which knows that DataFrames can have multiple types, some of which need isapprox, some of which need isequal. In bes…

I don’t see how this is related to DataFrames - if you are comparing collections of heterogeneous types, you need to decide what the right way to compare them is. You seem to want something like mycompare(x::Number, y::Number) = isapprox(x, y) mycompare(x, y) = isequal(x, y) which is simple enough…

that is impressively simple, thank you for your help. Still I think it would be a good addition to DataFrames.jl. There is more comparing two dataframes than comparing its values - also the columns needs to be compared. But I admit, this little helper is very compact and self-explaining.

DataFrames already has a definition of isapprox that does the right thing comparing columns by pairs: JuliaData/DataFrames.jl/blob/3e290274d3c201e8bfe903f0d326e78c38fc0fef/src/abstractdataframe/abstractdataframe.jl#L514-L533 isapprox(df1::AbstractDataFrame, df2:…

Ha, that function seems to have been added just when I proposed the isapprox solution above. But this also doesn’t help Stephan, who wants non-numerical types to be compared by isequal, which this definition also doesn’t do: julia> df1 = DataFrame(a = [1,2], b = ["a", "b"]); df2 = copy(df1); juli…

Indeed, I was just replying to this part: [image] schlichtanders: I think it would be a good addition to DataFrames.jl . There is more comparing two dataframes than comparing its values - also the columns needs to be compared. The scalar comparison issue remains but as you said above, that’s …

Cool. For me it is a DataFrame issue. DataFrames have different types of columns, by nature. Hence if someone shows me a method which is intended to work generically on DataFrames, I expect it to work on DataFrames with different kinds of columntypes. The isapprox on DataFrames is hence a bit odd …

The documentation says “isapprox with given keyword arguments applied to all pairs of columns” which implies that this method must be defined for all pairs of column types… I agree it could be made more explicit in the documentation. But this restriction makes sense: the column types can be anyth…

Yeah there is some danger of type-piracy in the air. two ways I see how DataFrames isapprox could be implemented to still support different column types: While restricting isapprox to only Number columns won’t work apparently, one could make this a configuration (which may even default to Number,…

Is there a package to compare if two DataFrames are the same?

New to Julia

Nathan_Boyer January 15, 2024, 3:16pm 12

I’m not sure it has the functionality you seek (didn’t look into it), but this package announcement seems to be aligned with your goal of better Data Frame comparisons. May want to add new features/requests there:

Topic		Replies	Views
Is there a package to do DataFrames comparisons? General Usage dataframes	4	1372	August 3, 2021
Compare dataframes regardless of column order General Usage dataframes	8	319	February 4, 2025
How to supplement a function for equality check when joining the DataFrames Data dataframes	2	122	July 5, 2024
How to compare non-missing elements of two DataFrames New to Julia	3	608	July 1, 2020
Various constructors and equality for DataFrame Data question	5	1186	January 18, 2017

Is there a package to compare if two DataFrames are the same?

Related topics