# How to sample a clustered group from a DataFrame?

**URL:** <https://discourse.julialang.org/t/how-to-sample-a-clustered-group-from-a-dataframe/45777>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [August 30, 2020, 2:56am UTC](https://discourse.julialang.org/t/how-to-sample-a-clustered-group-from-a-dataframe/45777 "2020-08-30T02:56:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jaepil\_Lee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jaepil_lee/32/9146_2.png) [@Jaepil\_Lee](https://discourse.julialang.org/u/Jaepil_Lee)\
**Post date:** [August 30, 2020, 2:56am UTC](https://discourse.julialang.org/t/how-to-sample-a-clustered-group-from-a-dataframe/45777/1 "2020-08-30T02:56:10Z")

</div>

Hello Julia users!

I have been trying to implement a Bootstrap from a panel data, stored in DataFrames. I think I got it working, but I am doing it in a super inefficient way.  
For example, as soon as I draw a vector of keys, say `keys=[1,2,1,3,4]`, I can do `df[keys]` to make a DataFrame by automatically duplicating the first row. Is there any way to do this in a grouped DataFrame? Below is my attempt, without doing so:

```julia
using DataFrames, CSV, Pipe, Parameters
using LinearAlgebra, Statistics, Distributions, Random
using StatsBase:sample
dfex = DataFrame(pid=repeat([1:4;], inner = [4]), 
                  a = randn(16), 
                  b = rand(16))
groupedDF = groupby(dfex, :pid)
length(dfex.pid)
unique_pid = unique(dfex.pid)
n_pid = length(unique_pid)
# Withe each pid, pick all the rows with that pid.
Bstrap = unique_pid[sample(axes(unique_pid, 1), n_pid; replace = true, ordered = false), 1]
length(unique(Bstrap))

function GenBootstrapDF(groupedDF,Bstrap)
  BootstrapDf = DataFrame()
  for i in Bstrap
      Bootstrap_sample = groupedDF[(pid=i,)]
      BootstrapDf = vcat(BootstrapDf, Bootstrap_sample)
  end
  return BootstrapDf
end 
GenBootstrapDF(groupedDF,Bstrap)

```

This works in the way I wanted, i.e. sampling the pid’s with replacement and then collect the SubDataFrame of the sampled pid’s from the `dfex`. However, I would like to improve the performance. Can you give me any ideas, please?

Thank you for your input!

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [August 30, 2020, 9:35am UTC](https://discourse.julialang.org/t/how-to-sample-a-clustered-group-from-a-dataframe/45777/2 "2020-08-30T09:35:47Z")

</div>

The `BootstrapDf = vcat(BootstrapDf, Bootstrap_sample)` pattern is generally better written as `append!(BootstrapDf, Bootstrap_sample)`. That will avoid a lot of copying (as internally Julia will anticipate the next call by allocating more space than needed).

Generally it’s also faster to use `reduce(vcat, list_of_dfs)` as it allows allocating the final data frame upfront. That requires storing a temporary `list_of_dfs` though, but here they are `SubDataFrame` views so they are cheap.

Finally, it’s faster/simpler to draw a sample of indices in `groupedDF` than a sample of `pid`. In the end something like this should be enough:

```julia
reduce(vcat, [groupedDF[i] for i in sample(1:length(groupedDF), n_pid; replace = true, ordered = false)])

```

---

<div class="post-metadata">

**Author:** ![Jaepil\_Lee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jaepil_lee/32/9146_2.png) [@Jaepil\_Lee](https://discourse.julialang.org/u/Jaepil_Lee)\
**Post date:** [August 30, 2020, 3:37pm UTC](https://discourse.julialang.org/t/how-to-sample-a-clustered-group-from-a-dataframe/45777/3 "2020-08-30T15:37:46Z")

</div>

This is what I wanted. I was experimenting like `[something[i] for i in something]` and `reduce` does the job. Thank you so much! For those who read this later, please run the following:

```julia
using DataFrames, CSV, Pipe, Parameters
using LinearAlgebra, Statistics, Distributions, Random
using StatsBase:sample

dfex = DataFrame(pid=repeat([1:4;], inner = [4]), 
                  a = randn(16), 
                  b = rand(16))
groupedDF = groupby(dfex, :pid)
n_pid = 10000 # whatever the number of sample you want from the data
A = reduce(vcat, [groupedDF[i] for i in sample(1:length(groupedDF), n_pid; replace = true, ordered = false)])

```