# Applying group selection conditions using groupeddataframes

**URL:** <https://discourse.julialang.org/t/applying-group-selection-conditions-using-groupeddataframes/46060>\
**Category:** New to Julia\
**Tags:** dataframes\
**Created:** [September 4, 2020, 1:41pm UTC](https://discourse.julialang.org/t/applying-group-selection-conditions-using-groupeddataframes/46060 "2020-09-04T13:41:17Z")\
**Posts on this page:** 2\
**Page:** 2

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 19, 2020, 11:29pm UTC](https://discourse.julialang.org/t/applying-group-selection-conditions-using-groupeddataframes/46060/21 "2020-09-19T23:29:22Z")

</div>

We are aware of these issues but its always helpful to formalize feedback!

---

<div class="post-metadata">

**Author:** ![jmboehm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jmboehm/32/2263_2.png) [@jmboehm](https://discourse.julialang.org/u/jmboehm)\
**Post date:** [September 20, 2020, 11:40am UTC](https://discourse.julialang.org/t/applying-group-selection-conditions-using-groupeddataframes/46060/22 "2020-09-20T11:40:48Z")

</div>

This is a little bit off-topic since the original question was about performance. But here’s how I would solve the problem using the [Douglass.jl interface to DataFrames](https://github.com/jmboehm/Douglass.jl), which may be appealing to economists and those familiar with Stata:

```julia
> df
11×4 DataFrame
│ Row │ id │ age │ inc │ status │
│ │ Int64 │ Int64 │ Int64 │ Int64 │
├─────┼───────┼───────┼───────┼────────┤
│ 1 │ 1 │ 53 │ 5 │ 1 │
│ 2 │ 1 │ 52 │ 5 │ 2 │
│ 3 │ 1 │ 17 │ 0 │ 3 │
│ 4 │ 2 │ 30 │ 0 │ 1 │
│ 5 │ 2 │ 29 │ 20 │ 2 │
│ 6 │ 3 │ 22 │ 12 │ 1 │
│ 7 │ 4 │ 61 │ 15 │ 1 │
│ 8 │ 5 │ 55 │ 11 │ 1 │
│ 9 │ 5 │ 51 │ 0 │ 2 │
│ 10 │ 6 │ 67 │ 12 │ 1 │
│ 11 │ 6 │ 62 │ 12 │ 2 │

```

Construct the husband’s age and a group-level dummy that captures whether the husband age requirement is satisfied.

```julia
Douglass> gen :husband_age = :age if :status == 1
11×5 DataFrame
│ Row │ id │ age │ inc │ status │ husband_age │
│ │ Int64 │ Int64 │ Int64 │ Int64 │ Int64? │
├─────┼───────┼───────┼───────┼────────┼─────────────┤
│ 1 │ 1 │ 53 │ 5 │ 1 │ 53 │
│ 2 │ 1 │ 52 │ 5 │ 2 │ missing │
│ 3 │ 1 │ 17 │ 0 │ 3 │ missing │
│ 4 │ 2 │ 30 │ 0 │ 1 │ 30 │
│ 5 │ 2 │ 29 │ 20 │ 2 │ missing │
│ 6 │ 3 │ 22 │ 12 │ 1 │ 22 │
│ 7 │ 4 │ 61 │ 15 │ 1 │ 61 │
│ 8 │ 5 │ 55 │ 11 │ 1 │ 55 │
│ 9 │ 5 │ 51 │ 0 │ 2 │ missing │
│ 10 │ 6 │ 67 │ 12 │ 1 │ 67 │
│ 11 │ 6 │ 62 │ 12 │ 2 │ missing │
Douglass> bysort :id (:status): egen :husband_age_requirement = (mean(skipmissing(:husband_age)) > 30) & (mean(skipmissing(:husband_age)) < 65)
11×6 DataFrame
│ Row │ id │ age │ inc │ status │ husband_age │ husband_age_requirement │
│ │ Int64 │ Int64 │ Int64 │ Int64 │ Int64? │ Bool │
├─────┼───────┼───────┼───────┼────────┼─────────────┼─────────────────────────┤
│ 1 │ 1 │ 53 │ 5 │ 1 │ 53 │ 1 │
│ 2 │ 1 │ 52 │ 5 │ 2 │ missing │ 1 │
│ 3 │ 1 │ 17 │ 0 │ 3 │ missing │ 1 │
│ 4 │ 2 │ 30 │ 0 │ 1 │ 30 │ 0 │
│ 5 │ 2 │ 29 │ 20 │ 2 │ missing │ 0 │
│ 6 │ 3 │ 22 │ 12 │ 1 │ 22 │ 0 │
│ 7 │ 4 │ 61 │ 15 │ 1 │ 61 │ 1 │
│ 8 │ 5 │ 55 │ 11 │ 1 │ 55 │ 1 │
│ 9 │ 5 │ 51 │ 0 │ 2 │ missing │ 1 │
│ 10 │ 6 │ 67 │ 12 │ 1 │ 67 │ 0 │
│ 11 │ 6 │ 62 │ 12 │ 2 │ missing │ 0 │

```

The husband-and-wife condition. Note that we can use any vector-valued Julia function on the right-hand-side of the assignment operation (here, `any`).

```julia
Douglass> bysort :id (:status): egen :husband_and_wife = any(:status .== 1) & any(:status .== 2)
11×7 DataFrame
│ Row │ id │ age │ inc │ status │ husband_age │ husband_age_requirement │ husband_and_wife │
│ │ Int64 │ Int64 │ Int64 │ Int64 │ Int64? │ Bool │ Bool │
├─────┼───────┼───────┼───────┼────────┼─────────────┼─────────────────────────┼──────────────────┤
│ 1 │ 1 │ 53 │ 5 │ 1 │ 53 │ 1 │ 1 │
│ 2 │ 1 │ 52 │ 5 │ 2 │ missing │ 1 │ 1 │
│ 3 │ 1 │ 17 │ 0 │ 3 │ missing │ 1 │ 1 │
│ 4 │ 2 │ 30 │ 0 │ 1 │ 30 │ 0 │ 1 │
│ 5 │ 2 │ 29 │ 20 │ 2 │ missing │ 0 │ 1 │
│ 6 │ 3 │ 22 │ 12 │ 1 │ 22 │ 0 │ 0 │
│ 7 │ 4 │ 61 │ 15 │ 1 │ 61 │ 1 │ 0 │
│ 8 │ 5 │ 55 │ 11 │ 1 │ 55 │ 1 │ 1 │
│ 9 │ 5 │ 51 │ 0 │ 2 │ missing │ 1 │ 1 │
│ 10 │ 6 │ 67 │ 12 │ 1 │ 67 │ 0 │ 1 │
│ 11 │ 6 │ 62 │ 12 │ 2 │ missing │ 0 │ 1 │

```

Finally, the income requirement. This can be done in different ways, e.g.

```julia
Douglass> bysort :id (:status): egen :income_requirement = any((:status .== 1) .& (:inc .> 10)) | any((:status .== 2) .& (:inc .> 10))  
11×8 DataFrame
│ Row │ id │ age │ inc │ status │ husband_age │ husband_age_requirement │ husband_and_wife │ income_requirement │
│ │ Int64 │ Int64 │ Int64 │ Int64 │ Int64? │ Bool │ Bool │ Bool │
├─────┼───────┼───────┼───────┼────────┼─────────────┼─────────────────────────┼──────────────────┼────────────────────┤
│ 1 │ 1 │ 53 │ 5 │ 1 │ 53 │ 1 │ 1 │ 0 │
│ 2 │ 1 │ 52 │ 5 │ 2 │ missing │ 1 │ 1 │ 0 │
│ 3 │ 1 │ 17 │ 0 │ 3 │ missing │ 1 │ 1 │ 0 │
│ 4 │ 2 │ 30 │ 0 │ 1 │ 30 │ 0 │ 1 │ 1 │
│ 5 │ 2 │ 29 │ 20 │ 2 │ missing │ 0 │ 1 │ 1 │
│ 6 │ 3 │ 22 │ 12 │ 1 │ 22 │ 0 │ 0 │ 1 │
│ 7 │ 4 │ 61 │ 15 │ 1 │ 61 │ 1 │ 0 │ 1 │
│ 8 │ 5 │ 55 │ 11 │ 1 │ 55 │ 1 │ 1 │ 1 │
│ 9 │ 5 │ 51 │ 0 │ 2 │ missing │ 1 │ 1 │ 1 │
│ 10 │ 6 │ 67 │ 12 │ 1 │ 67 │ 0 │ 1 │ 1 │
│ 11 │ 6 │ 62 │ 12 │ 2 │ missing │ 0 │ 1 │ 1 │

```

Now keep only those observations that satisfy all three requirements.

```julia
Douglass> keep if :husband_age_requirement & :husband_and_wife & :income_requirement
2×8 DataFrame
│ Row │ id │ age │ inc │ status │ husband_age │ husband_age_requirement │ husband_and_wife │ income_requirement │
│ │ Int64 │ Int64 │ Int64 │ Int64 │ Int64? │ Bool │ Bool │ Bool │
├─────┼───────┼───────┼───────┼────────┼─────────────┼─────────────────────────┼──────────────────┼────────────────────┤
│ 1 │ 5 │ 55 │ 11 │ 1 │ 55 │ 1 │ 1 │ 1 │
│ 2 │ 5 │ 51 │ 0 │ 2 │ missing │ 1 │ 1 │ 1 │

```

I should say that while I’ve tried to implement these functions using DataFrames `combine` and `transform`, performance so far has not been my main concern, and I haven’t done a lot of benchmarking yet. Still, I would expect this to not take very long on 2.5m observations.

[Previous page](https://discourse.julialang.org/t/applying-group-selection-conditions-using-groupeddataframes/46060.md?page=1)
