# Lazy columns in dataframes?

**URL:** <https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599>\
**Category:** Data\
**Tags:** question, dataframes\
**Created:** [January 26, 2023, 10:35pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599 "2023-01-26T22:35:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ggggggggg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ggggggggg/32/265_2.png) [@ggggggggg](https://discourse.julialang.org/u/ggggggggg)\
**Post date:** [January 26, 2023, 10:35pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/1 "2023-01-26T22:35:54Z")

</div>

When I think about using `DataFrames` or similar data science packages, I always worry about potential to calculate column `C` from an existing dataset, then calculate column `D` from `C`, then I can recalculate `C` and have `D` be inconsistent. Here is one way to avoid that with `DataFrames`, and I wonder what people think of it. Are there any footguns here? Would this be useful for other workflows?

The idea is to make a `LazyColumn` with a reference to the dataframe and columns it’s calculated from and a function to calculate the values. The values aren’t instantiated until queried. I believe something like this is common in databases as “computed column”, but here we have the full power of Julia rather than a database specific set of functions to work with.

```julia
using DataFrames
struct LazyCol <: AbstractVector{Float64}
	parent
	arg_cols
	f
end
function Base.getindex(v::LazyCol, i)
	arg_df = v.parent[i, v.arg_cols]
	return float.(v.f.(eachcol(arg_df)...))
end
Base.size(v::LazyCol) = (size(v.parent)[1],)
DataFrames.eachcol(dfr::DataFrameRow) = values(NamedTuple(dfr))
df = DataFrame(A=1:4, B=["M", "F", "F", "M"], C=5:8);
df.D = LazyCol(df, ["A","C"], (A,C)->A+C);
df.E = LazyCol(df, ["D"], D->D*2);

```

output

```julia
julia> df
4×5 DataFrame
 Row │ A B C D E
     │ Int64 String Int64 Float64 Float64
─────┼────────────────────────────────────────
   1 │ 1 M 5 6.0 10.0
   2 │ 2 F 6 8.0 12.0
   3 │ 3 F 7 10.0 14.0
   4 │ 4 M 8 12.0 16.0

```

Then if I re-define `D`, `E` is still 2 times `D`’s value. Also, if I use a callable struct for the function I have access to a complete graph of the way each column was calculated.

```julia
julia> df.D = LazyCol(df, ["A","C"], (A,C)->A-C);

julia> df
4×5 DataFrame
 Row │ A B C D E
     │ Int64 String Int64 Float64 Float64
─────┼────────────────────────────────────────
   1 │ 1 M 5 -4.0 -8.0
   2 │ 2 F 6 -4.0 -8.0
   3 │ 3 F 7 -4.0 -8.0
   4 │ 4 M 8 -4.0 -8.0

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 26, 2023, 10:40pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/2 "2023-01-26T22:40:16Z")

</div>

This is a cool idea! It would make a great package that, of course, would not need to rely on DataFrames.jl at all.

I think one thing to consider, though, is what assumptions DataFrames.jl makes about vectors. There might be some methods you _need_ to implement or else DataFrames.jl might make a copy and solidify the array.

---

<div class="post-metadata">

**Author:** ![ggggggggg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ggggggggg/32/265_2.png) [@ggggggggg](https://discourse.julialang.org/u/ggggggggg)\
**Post date:** [January 26, 2023, 10:44pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/3 "2023-01-26T22:44:24Z")

</div>

> [@pdeffebach](#):
>
> of course, would not need to rely on DataFrames.jl at all.

Can you elaborate on that? Do you mean it could be generalized to the Tables.jl API or something?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 26, 2023, 10:57pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/4 "2023-01-26T22:57:04Z")

</div>

I was imagining something where `LazyCol` only has vectors in it’s object, and computations are computed on the vectors.

```julia
julia> struct LazyCol
           f
           vecs
       end;

julia> function Base.getindex(v::LazyCol, i)
           vecs = v.vecs
           inputs = ntuple(k -> vecs[k][i], length(vecs))
           v.f(inputs...)
       end;

julia> t = LazyCol(-, [[1, 2], [100, 200]]);

julia> t[1]
-99

```

But that idea doesn’t really work if the input arrays get copied some how. So you do need to know something about the containing data frame.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 26, 2023, 11:00pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/5 "2023-01-26T23:00:20Z")

</div>

[GitHub - JuliaArrays/MappedArrays.jl: Lazy in-place transformations of arrays](https://github.com/JuliaArrays/MappedArrays.jl) ?

notice once you made a mapped array, you can stick it into a DataFrame without problem since it’s still a `<: AbstractVector`

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 26, 2023, 11:03pm UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/6 "2023-01-26T23:03:53Z")

</div>

For any example, though, unless you are really careful with `copycols = false` or doing mutating `!` operations, any `select(df, ...)` which copies vectors is going to break a lazy scheme.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [January 27, 2023, 7:53am UTC](https://discourse.julialang.org/t/lazy-columns-in-dataframes/93599/7 "2023-01-27T07:53:28Z")

</div>

Also support for operations like `push!`, `deleteat!` etc. need to be considered (of course it also could be just not supported assuming user does not do such operations)