# Subsetting boolean columns and multiple assignments

**URL:** <https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [September 5, 2021, 12:30pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716 "2021-09-05T12:30:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jacobusmmsmit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jacobusmmsmit/32/217669_2.png) [@jacobusmmsmit](https://discourse.julialang.org/u/jacobusmmsmit)\
**Post date:** [September 5, 2021, 12:30pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716/1 "2021-09-05T12:30:23Z")

</div>

Hello, I have two related questions both related to the following code:

```julia
df = DataFrame(a = [true, true, false])
subset1 = subset(df, :a)
subset2 = subset(df, :a => ByRow(!=(true)))

```

I want to be able to do something like

```julia
subset1, subset2 = subset(df, :a, return_both = true)

```

in which the second one is just whichever rows are left over after the subset so I don’t have to do another subset. I’m not sure how the internals of `subset` are handled but is it possible to somehow return only the index of the rows that match the condition? In this case you could then just run a single subset function, get the indices of `subset1` and just do a set exclusion to get the indices of `subset2`.

As an extension, this would allow you to then perform another subset on `subset2` to get `subset3` in a similar way without having to look at the whole data frame again.

If this is possible I’d love to know how to do it, if not how would you recommend I perform these operations efficiently?

Finally, is there a nicer way of subsetting with the negation of a boolean column than this?

```julia
subset(df, :a => ByRow(!=(true)))

```

Perhaps something like this?

```julia
subset(df, !:a)

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 5, 2021, 1:35pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716/2 "2021-09-05T13:35:56Z")

</div>

> [@jacobusmmsmit](#):
>
> in which the second one is just whichever rows are left over after the subset so I don’t have to do another subset. I’m not sure how the internals of `subset` are handled but is it possible to somehow return only the index of the rows that match the condition? In this case you could then just run a single subset function, get the indices of `subset1` and just do a set exclusion to get the indices of `subset2` .

No, there is nothing built-in for subset with this. I think the way to do this would be `findall` and have a slightly more complicated expression outside of the chaining pipeline. DataFramesMeta.jl’s `@with` macro will help with this.

> [@jacobusmmsmit](#):
>
> `subset(df, !:a)`

Yes! Please see [DataFramesMeta](https://github.com/JuliaData/DataFramesMeta.jl) which provides exactly the syntax you describe, with the help of metaprogramming.

---

<div class="post-metadata">

**Author:** ![jacobusmmsmit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jacobusmmsmit/32/217669_2.png) [@jacobusmmsmit](https://discourse.julialang.org/u/jacobusmmsmit)\
**Post date:** [September 5, 2021, 1:39pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716/3 "2021-09-05T13:39:24Z")

</div>

These answers comes from @bkamins and @ericphanson from Slack, posting here for permanence:

For the first part, getting indices can be done with `parentindices` resulting in the following example function:

```julia
function subset_returning_both(df::AbstractDataFrame, args...; skipmissing::Bool = false, view::Bool = false)
    subtestdf = subset(df, args...; skipmissing, view)
    indices = 1:size(df)[1]
    matching_indices = parentindices(subtestdf)[1]
    nonmatching_indices = setdiff(indices, matching_indices)
    return df[matching_indices, :], df[nonmatching_indices, :]
end

```

Alternatively, one could `transform` to get a new column with the Boolean values and then perform a `groupby` to get the groups. This method is much faster.

For the second part, it should instead read

```julia
subset(df, :a => ByRow(!isequal(true)))

```

which I must say is still a little verbose but at least now it’s following best practices.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 5, 2021, 1:42pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716/4 "2021-09-05T13:42:16Z")

</div>

As mentioned in slack, I think `transform` and `groupby` is the best solution here. Grouped DataFrames seems like the most natural built-in object for this kind of thing.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [September 5, 2021, 5:40pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716/5 "2021-09-05T17:40:47Z")

</div>

> [@jacobusmmsmit](#):
>
> `subset(df, :a => ByRow(!isequal(true)))`

I think you can do just

```julia
subset(df, :a => ByRow(!))

```

if it really is Boolean and doesn’t contain `missing`s for example.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 5, 2021, 8:37pm UTC](https://discourse.julialang.org/t/subsetting-boolean-columns-and-multiple-assignments/67716/6 "2021-09-05T20:37:54Z")

</div>

Some small comments to the answers already given.

> [@jacobusmmsmit](#):
>
> ```julia
> function subset_returning_both(df::AbstractDataFrame, args...; skipmissing::Bool = false, view::Bool = false)
> 
> ```

`df` should strictly be a `DataFrame` here (`parentindices` always refers to source data frame).

This solution answers the question directly (as you ask if and how `subset` can produce the indices),  
however in general `subset` is designed to be used in piping, so it does not normally produce a bare vector of indices and what @pdeffebach suggests (i.e. to add a column with a condition and then `groupby` it) is preferable assuming that you can afford to add an additional column to a data frame (which usually should be OK).

Now regarding the difference between `ByRow(!isequal(true))` and `ByRow(!)` observe that `subset`, if `skipmissing=true` would treat `missing` as false, but just negating missing with `!` is still `missing`, so it would be dropped twice. To show what I mean consider:

```julia
julia> .!([true, false, missing])
3-element Vector{Union{Missing, Bool}}:
 false
  true
      missing

julia> (!isequal(true)).([true, false, missing])
3-element BitVector:
 0
 1
 1

```