Dataframes subset w/ nullable logic

General Usage
,
1

Working on a subset call that filters on multiple columns in a dataframe, but with variable conditions set by an external dict. So if the dict has a value, it checks for equality in the corresponding column. If the dict has a “null”, in this case, an empty string, it allows all rows (but lets the other columns filter in their own right). This means that the call looks like:
subset(data, :X => x -> (x.=="") .|| (x.==criteria['X']), :Y => y-> (y.=="") .|| (y.==criteria["Y"]))
What am I doing wrong w/ this logic? The short-circuiting should allow a bunch more rows through, but instead I’m getting all rows filtered out.

2

There’s nothing obviously wrong there, try getting your problem down to a fully reproducible example with data. E.g. below works fine

julia> df = DataFrame(X=["", "A", "B"], Y=["A", "", "B"])
3×2 DataFrame
 Row │ X       Y
     │ String  String
─────┼────────────────
   1 │         A
   2 │ A
   3 │ B       B

julia> criteria = Dict("X" => "A", "Y" => "A")
Dict{String, String} with 2 entries:
  "Y" => "A"
  "X" => "A"

julia> subset(df, :X => x -> (x.=="") .|| (x.==criteria["X"]), :Y => y-> (y.=="") .|| (y.==criteria["Y"]))
2×2 DataFrame
 Row │ X       Y
     │ String  String
─────┼────────────────
   1 │         A
   2 │ A
3

I should clarify, the dataframe doesn’t have “” in it, the criteria might, so if there are any “” criteria, I want that arg to be True all the time.

4

In that case you’d need to make the conditional on the criteria - like this.

subset(df, :X => x -> criteria["X"] == "" ? trues(length(x)) : (x .== criteria["X"]),
           :Y => y -> criteria["Y"] == "" ? trues(length(y)) : (y .== criteria["Y"]))

Though, especially if the number of criteria is large or dynamic, I might write it more like this

julia> function make_subset(criteria)
           subsets = []
           for (name, value) in criteria
               if value != ""
                   push!(subsets, name => (x -> x .== value))
               end
           end
           return subsets
       end
make_subset (generic function with 1 method)

julia> subset(df, make_subset(criteria)...)
1×2 DataFrame
 Row │ X       Y
     │ String  String
─────┼────────────────
   1 │ C       A
5

Realized it myself while I was running errands. Thanks for the fresh eyeballs!!!