# Heatmap of a dataframe

**URL:** <https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272>\
**Category:** Visualization\
**Created:** [June 14, 2017, 1:58pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272 "2017-06-14T13:58:08Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![ko56](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ko56/32/2684_2.png) [@ko56](https://discourse.julialang.org/u/ko56)\
**Post date:** [June 14, 2017, 1:58pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/1 "2017-06-14T13:58:08Z")

</div>

It seems that StatPlots doesn’t support making a heatmap of an all-numeric dataframe. Would that be a reasonable addition to the package?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [June 14, 2017, 2:11pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/2 "2017-06-14T14:11:35Z")

</div>

At first, I would say that the heatmap plot fits better in PlotRecipes.jl instead of StatsRecipes.jl? I’ve asked @mkborregaard about it in this thread:

> [@Annotations and line widths in Plots.jl heatmaps](https://discourse.julialang.org/t/annotations-and-line-widths-in-plots-jl-heatmaps/4259):
>
> One of the nicest features of heatmaps in the Python’s Seaborn module is the ability to annotate heatmaps with the values in the cells, the axes labels, and the line style separating the cells as you can see in these examples: [seaborn.heatmap — seaborn 0.12.2 documentation](http://seaborn.pydata.org/generated/seaborn.heatmap.html) Given the following heatmap in Plots.jl: using Plots A = rand(10,10) heatmap(A) Is there an option to annotate the cells? What about the line styles?

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [June 14, 2017, 2:51pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/3 "2017-06-14T14:51:59Z")

</div>

What would be the use case for this?

---

<div class="post-metadata">

**Author:** ![ko56](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ko56/32/2684_2.png) [@ko56](https://discourse.julialang.org/u/ko56)\
**Post date:** [June 14, 2017, 3:08pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/4 "2017-06-14T15:08:21Z")

</div>

I’m not sure what kind of answer you expect, but the particular thing  
I’m trying to do is display the pattern of missing values in a large set  
of numeric data, which is in the form of a dataframe. It seems to me  
the heatmap is the best kind of display for this (although in this use  
case it would have only two levels).

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [June 14, 2017, 3:17pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/5 "2017-06-14T15:17:58Z")

</div>

Ah, I see. That would involve converting the DataFrame to a Matrix of Float64, replacing the NAs with NaN. For a Matrix, your specific use case would actually better fit `spy` better than `heatmap`. A dataframe\_spy recipe in StatPlots that ran `isna` and returned the resulting Matrix{Bool} wrapped in a Plots.Spy object of the result would be a nice addition I think.

---

<div class="post-metadata">

**Author:** ![ko56](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ko56/32/2684_2.png) [@ko56](https://discourse.julialang.org/u/ko56)\
**Post date:** [June 14, 2017, 3:38pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/6 "2017-06-14T15:38:11Z")

</div>

Ok, thanks, I found the spy function in Plot/recipes/. Before I posted  
my question, I had thought of converting the dataframe to a matrix.  
The only drawback I saw was that I would lose the column labels of the  
dataframe, which would be useful in understanding the pattern, be it  
NAs/NaNs or non-0 elements.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [June 14, 2017, 3:44pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/7 "2017-06-14T15:44:27Z")

</div>

Good point - the names could be used as xticks in the recipe. The conversion could be as simple as

```julia
co, ro = Int[], Int[]
for j in 1:ncols(df)
   for i in 1:nrows(df)
      if isna(df[i,j])
         push!(co, j)
         push!(ro, i)
      end
   end
end
sparse(ro, co, true)

```

---

<div class="post-metadata">

**Author:** ![ko56](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ko56/32/2684_2.png) [@ko56](https://discourse.julialang.org/u/ko56)\
**Post date:** [June 14, 2017, 6:09pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/8 "2017-06-14T18:09:20Z")

</div>

I just saw a post by you on StackExchange on the same subject, suggesting the use of ‘convert’:

Perfect! 🙂 Happy to help. One note: If you have Null / NA values, you can give convert a third argument to replace them (as they are not allowed in an Array), e.g. convert(Array, iris[1:3], 0.).  
– Michael K. Borregaard Mar 22 at 17:47

This seems easier, is there any reason not to use this solution?  
However, convert(), in 0.5.1 at least, does not accept a 3d argument.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [June 14, 2017, 6:51pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/9 "2017-06-14T18:51:31Z")

</div>

Because I thought it was faster to generate a sparse Boolean Matrix immediately, as that’s what spy needs. I just benchmarked, and it isn’t, in fact.

The three-argument conversion method only works for DataArrays, as the third argument always has a type (NaN is Float64), but you don’t expect different DataFrame columns to have the same type. So you have to convert the columns one by one:

```julia
ar = hcat([convert(Array, df[i], NaN) for i in 1:ncol(df)]...)
# then return the Spy object that will automatically dispatch to the spy recipe
Spy(ar)

```

---

<div class="post-metadata">

**Author:** ![ko56](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ko56/32/2684_2.png) [@ko56](https://discourse.julialang.org/u/ko56)\
**Post date:** [June 14, 2017, 7:02pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/10 "2017-06-14T19:02:29Z")

</div>

Am I doing something wrong here?

julia\> df  
5×3 DataFrames.DataFrame  
│ Row │ a │ b │ c │  
├─────┼────┼────┼───┤  
│ 1 │ 1 │ 1 │ 1 │  
│ 2 │ 2 │ 2 │ 2 │  
│ 3 │ NA │ NA │ 3 │  
│ 4 │ 4 │ 4 │ 4 │  
│ 5 │ 5 │ 5 │ 5 │

julia\> convert(Array, df[1], NaN)  
ERROR: InexactError()  
in convert(::Type{Array{Int64,1}}, ::DataArrays.DataArray{Int64,1}, ::Float64) at /usr/home/ko/.julia/v0.5/DataArrays/src/dataarray.jl:391  
in convert(::Type{Array}, ::DataArrays.DataArray{Int64,1}, ::Float64) at /usr/home/ko/.julia/v0.5/DataArrays/src/dataarray.jl:412

julia\>

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [June 14, 2017, 7:17pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/11 "2017-06-14T19:17:19Z")

</div>

No, that’s just a really annoying aspect of the code. NaN are Float64, but the `convert` function recognises the `eltype` of `df[:a]` as `Int`, and the `InexactError()` tells you you cannot convert `NaN` to `Int`. Sorry, when you said “all-numeric” I thought “all-floats”. The Bool method works though. Or you can do `convert(Array, DataArray{Float64}(dd), NaN)`.

---

<div class="post-metadata">

**Author:** ![ko56](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ko56/32/2684_2.png) [@ko56](https://discourse.julialang.org/u/ko56)\
**Post date:** [June 14, 2017, 9:53pm UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/12 "2017-06-14T21:53:13Z")

</div>

Ok, thanks for your help. My last problem is that I can’t eliminate the colorbar from the heatmap. Neither legend=false, nor colorbar=false, nor both do the trick.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [June 15, 2017, 5:43am UTC](https://discourse.julialang.org/t/heatmap-of-a-dataframe/4272/13 "2017-06-15T05:43:29Z")

</div>

Sounds like a bug. You could open an issue, specifying your backend and versioninfo