# Sparse DataFrame

**URL:** <https://discourse.julialang.org/t/sparse-dataframe/64096>\
**Category:** General Usage\
**Tags:** dataframes, sparse\
**Created:** [July 5, 2021, 6:13pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096 "2021-07-05T18:13:52Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexlenail](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alexlenail/32/26764_2.png) [@alexlenail](https://discourse.julialang.org/u/alexlenail)\
**Post date:** [July 5, 2021, 6:13pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/1 "2021-07-05T18:13:52Z")

</div>

DataFrame(sparse\_matrix) gives me:

```julia
ArgumentError: a 'SparseMatrixCSC{Float64,Int64}' is not a table; see `?Tables.table` for ways to treat an AbstractMatrix as a table

```

I can’t find documentation about how to load a sparse matrix into a DataFrame, without inflating my matrix (which I’d rather avoid).

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [July 5, 2021, 6:46pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/2 "2021-07-05T18:46:39Z")

</div>

What do you want your DataFrame to look like?

---

<div class="post-metadata">

**Author:** ![alexlenail](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alexlenail/32/26764_2.png) [@alexlenail](https://discourse.julialang.org/u/alexlenail)\
**Post date:** [July 5, 2021, 6:53pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/3 "2021-07-05T18:53:04Z")

</div>

Zeros where the matrix doesn’t have entries – but without occupying 30GB of RAM

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [July 6, 2021, 2:04am UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/4 "2021-07-06T02:04:30Z")

</div>

I was just going to reply that this probably won’t be possible, and asking for your use case (30GB implies quite a lot of rows and/or columns, which if sparse are probably just a pain to handle outside the `SparseArrays` infrastructure?) but then I thought ah well this is Julia after all so why wouldn’t two packages just work ™ ?

You can think of a DataFrame as a collection of columns, with each column being a (named) standard Julia vector, so constructing a DataFrame from the columns of a `SparseArray`:

```julia
julia> using DataFrames, SparseArrays

julia> I = [1, 4, 3, 10_000_000]; J = [4, 7, 200_000, 9]; V = [1, 2, -5, 6];

julia> S = sparse(I, J, V);

julia> df = DataFrame(["x$i" => c for (i, c) ∈ enumerate(eachcol(S))]...)
10000000×200000 DataFrame
(...)

```

and just to show that I’m not just using a machine with 2TB of RAM:

```julia
julia> rand(0:5, 10_000_000, 200_000)
ERROR: OutOfMemoryError()

```

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [July 6, 2021, 6:22am UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/5 "2021-07-06T06:22:00Z")

</div>

Here’s an example wrapping an existing sparse matrix in a data frame:

```julia
julia> A = sparse(2I(4))
4×4 SparseMatrixCSC{Int64, Int64} with 4 stored entries:
 2 ⋅ ⋅ ⋅
 ⋅ 2 ⋅ ⋅
 ⋅ ⋅ 2 ⋅
 ⋅ ⋅ ⋅ 2

julia> df = DataFrame([eachcol(A)...], :auto, copycols=false)
4×4 DataFrame
 Row │ x1 x2 x3 x4    
     │ Int64 Int64 Int64 Int64 
─────┼────────────────────────────
   1 │ 2 0 0 0
   2 │ 0 2 0 0
   3 │ 0 0 2 0
   4 │ 0 0 0 2

julia> df.x1
4-element view(::SparseMatrixCSC{Int64, Int64}, :, 1) with eltype Int64:
 2
 0
 0
 0

```

You can remove `copycols=false` to have the `DataFrame` constructor automatically copy each column to a new `SparseVector`.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 6, 2021, 7:02am UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/6 "2021-07-06T07:02:49Z")

</div>

This is exactly the way to do it. The only cost is that some functions defined in DataFrames.jl will not work correctly as they assume that you can resize the vectors.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [November 3, 2021, 12:51pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/7 "2021-11-03T12:51:14Z")

</div>

Hi! What do you mean by “not work correctly”: should I expect exceptions or sneaky silent bugs?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 3, 2021, 1:09pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/8 "2021-11-03T13:09:33Z")

</div>

> [@gdalle](#):
>
> “not work correctly”: should I expect exceptions or sneaky silent bugs?

Exceptions. If you e.g. do `push!` or `append!` you will get an exception that the vector is not resizable.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 3, 2021, 1:17pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/9 "2021-11-03T13:17:00Z")

</div>

Cross-posted from Slack, here is one instance where it doesn’t work.

```julia
julia> using DataFramesMeta, SparseArrays;

julia> df = DataFrame(x = sparse([1, 0, 0, 0, 1]))
5×1 DataFrame
 Row │ x     
     │ Int64 
─────┼───────
   1 │ 1
   2 │ 0
   3 │ 0
   4 │ 0
   5 │ 1

julia> @subset! df :x .== 1
ERROR: MethodError: no method matching deleteat!(::SparseVector{Int64, Int64}, ::Vector{Int64})
Closest candidates are:
  deleteat!(::Vector{T} where T, ::AbstractVector{T} where T) at array.jl:1377
  deleteat!(::Vector{T} where T, ::Any) at array.jl:1376
  deleteat!(::BitVector, ::Any) at bitarray.jl:981
  ...
Stacktrace:
 [1] _delete!_helper(df::DataFrame, drop::Vector{Int64})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:999
 [2] delete!(df::DataFrame, inds::Vector{Int64})
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/dataframe/dataframe.jl:976
 [3] subset!(df::DataFrame, args::Any; skipmissing::Bool)
   @ DataFrames ~/.julia/packages/DataFrames/vuMM8/src/abstractdataframe/subset.jl:292
 [4] top-level scope
   @ ~/.julia/packages/DataFramesMeta/lbAjC/src/macros.jl:681

```

So no, it won’t be a sneaky bug.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [November 3, 2021, 1:39pm UTC](https://discourse.julialang.org/t/sparse-dataframe/64096/10 "2021-11-03T13:39:07Z")

</div>

Just referencing the GitHub issue [https://github.com/JuliaData/CategoricalArrays.jl/issues/374](https://github.com/JuliaData/CategoricalArrays.jl/issues/374)