# Proposal: Using a "mask" for DataFrames and ModelMatrices
**URL:** https://discourse.julialang.org/t/proposal-using-a-mask-for-dataframes-and-modelmatrices/5779
**Category:** Statistics
**Tags:** proposal
**Created:** [September 8, 2017, 2:37am UTC](https://discourse.julialang.org/t/proposal-using-a-mask-for-dataframes-and-modelmatrices/5779 "2017-09-08T02:37:56Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@jeffwong](https://discourse.julialang.org/u/jeffwong)
#### Post date: [September 8, 2017, 2:37am UTC](https://discourse.julialang.org/t/proposal-using-a-mask-for-dataframes-and-modelmatrices/5779/1 "2017-09-08T02:37:56Z")
In data science we always have a need to split data into a training set and a test set. Sometimes we will do this multiple times like in k fold cross validation, or leave one out cross validations.
In R: the typical paradigm is
```r
train.rows = sample(1 : nrow(df), 0.8 * nrow(df), replace = F)
train_df = df[train.rows,]
test_df = df[-train.rows,]
```
The process of subsetting the rows actually allocates a new dataframe. Ideally there would have been a way to flag the testing rows as “hidden” with a single bit. Then the dataframe could be passed to a modeling function like glm, without any extra allocations.
Other than cross validation, this could be really useful for general dataframe operations.
Perhaps one way we could implement this is for every dataframe type to carry around a `weight` integer vector. If the row is hidden, the weight would be 0, and otherwise 1. This might work well with other implementations in stats like weighted means and variances. If the weighted mean function sees a weight of 0, it would disregard that row anyway, which would be equivalent to the mask
---
### Author:  [@juliohm](https://discourse.julialang.org/u/juliohm)
#### Post date: [September 8, 2017, 3:42am UTC](https://discourse.julialang.org/t/proposal-using-a-mask-for-dataframes-and-modelmatrices/5779/2 "2017-09-08T03:42:25Z")
Something along the lines of array `views` would solve the issue without the need for carrying a bit for every row of the frame:
```julia
A = zeros(1000,1000) # big matrix
rows = view(A, 1:500, :) # view first 500 rows without making copies
@show sizeof(A) # 8000000
@show sizeof(rows) # 48
```
I don’t know if there is an equivalent for `DataFrames` already implemented though.
---
### Author:  [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)
#### Post date: [September 8, 2017, 8:30am UTC](https://discourse.julialang.org/t/proposal-using-a-mask-for-dataframes-and-modelmatrices/5779/3 "2017-09-08T08:30:29Z")
Yes, views are implemented for DataFrames. In fact, all internal split-apply-combine methods for DataFrames use views. You can simply do
```julia
trainrows = [rand()<0.8 for i in eachrow(df)]
traindf = @view df[trainrows]
testdf = view(df, .!(trainrows))
```
The macro- and function- syntax for specifying views are equivalent.
---
### Author:  [@jeffwong](https://discourse.julialang.org/u/jeffwong)
#### Post date: [September 8, 2017, 11:46pm UTC](https://discourse.julialang.org/t/proposal-using-a-mask-for-dataframes-and-modelmatrices/5779/4 "2017-09-08T23:46:00Z")
Wow Julia is so amazing! I’m looking for view for ModelMatrix, but I can’t seem to find it