Is any way to print DataFrames using heat maps as background?

[Dan_Micsa]

I want to print in various notebooks dataframes setting the background of each cell. This is possible in Python. Here is an example using a heatmap.

Something that looks like:

Is some library or code out there to do this? I know how to do this using plotting, I need something different.

[nilshg]

If anything this would probably be a job for PrettyTables HTML backend

[bkamins]

It should be also possible with text/plain. @Ronis_BR is the best person to comment.

[Dan_Micsa]

It will be cripple by various consoles constrains on various platforms. I’m more interested by decent printing in various notebooks around like in Python.

[pfitzseb]

Well, the

PrettyTables HTML backend

should work just fine in Jupyter notebooks.

[Dan_Micsa]

Very good idea! At least in VSCode works great.

using PrettyTables

data = Any[1 false 1.0 0x01
    2 true 2.0 0x02
    3 false 3.0 0x03
    4 true 4.0 0x04
    5 false 5.0 0x05
    6 true 6.0 0x06]

h1 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 1, bold = true, background = 0x220044)
h2 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 2, bold = true, background = 0x440088)
h3 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 3, bold = true, background = 0x6600AA)
h4 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 4, bold = true, background = 0x8800CC)

pretty_table(data, highlighters = (h1, h2, h3, h4))

[Ronis_BR]

Hi @Dan_Micsa !

You can do this using PrettyTables with the following code:

using Colors
using ColorSchemes
using DataFrames
using PrettyTables

# Function that defines the decoration in each cell.
function colormap_decoration(h, data, i, j)
    max = maximum(eachcol(df)) |> maximum
    min = minimum(eachcol(df)) |> minimum

    # Get the color of the current cell.
    color = get(
        colorschemes[:coolwarm],
        data[i, j],
        (min, max)
    )

    # Convert to hexadecimal.
    hex_color = color |> hex

    # Add to the decoration.
    return HTMLDecoration(background = "#" * hex_color)
end

# Create a ramdom DataFrames.
df = DataFrame(rand(20, 5), :auto)

# Define the highlighter that will add the color to each cell.
hl = HTMLHighlighter(
    (data, i, j) -> true,
    colormap_decoration
)

# Print the table.
pretty_table(df, highlighters = (hl,), backend = :html)

Leading to:

I used the master branch of PrettyTables.jl, but it should work in the released version. Notice that PrettyTables is passing thought a huge rewrite and next version should be some breaking changes.

Side note: this code is not optimized (max and min values are being computed when analyzing each cell), but it works

[Dan_Micsa]

I close this case thank you all for answers!

[Dan_Micsa]

thank you very much for this example.

[Dan_Micsa]

max and min values are being computed when analyzing each cell

I saw that “attrocity” but I was a gentleman.

[Ronis_BR]

[Dan_Micsa]

Another question, if you have a bit of time, how can I pass min and max for color map in your example w/o the ugly globals?

function colormap_decoration(h, data, i, j)
    max = maximum(eachcol(df)) |> maximum
    min = minimum(eachcol(df)) |> minimum
...

[Ronis_BR]

The first and easiest is to create a closure:

using Colors
using ColorSchemes
using DataFrames
using PrettyTables

function print_df(df)
    max = maximum(eachcol(df)) |> maximum
    min = minimum(eachcol(df)) |> minimum

    # Function that defines the decoration in each cell.
    function colormap_decoration(h, data, i, j)
        # Get the color of the current cell.
        color = get(
            colorschemes[:coolwarm],
            data[i, j],
            (min, max)
        )

        # Convert to hexadecimal.
        hex_color = color |> hex

        # Add to the decoration.
        return HTMLDecoration(background = "#" * hex_color)
    end

    # Define the highlighter that will add the color to each cell.
    hl = HTMLHighlighter(
        (data, i, j) -> true,
        colormap_decoration
    )

    # Print the table.
    pretty_table(df, highlighters = (hl,), backend = :html)
end

# Create a ramdom DataFrames.
df = DataFrame(rand(20, 5), :auto)
print_df(df)

The second is to define your own highlighter structure:

using Colors
using ColorSchemes
using DataFrames
using PrettyTables

struct MyHighlighter{T}
    f::Function
    fd::Function
    min::T
    max::T
end

# Function that defines the decoration in each cell.
function colormap_decoration(h, data, i, j)
    # Get the color of the current cell.
    color = get(
        colorschemes[:coolwarm],
        data[i, j],
        (h.min, h.max)
    )

    # Convert to hexadecimal.
    hex_color = color |> hex

    # Add to the decoration.
    return HTMLDecoration(background = "#" * hex_color)
end

function print_df(df)
    max = maximum(eachcol(df)) |> maximum
    min = minimum(eachcol(df)) |> minimum

    # Define the highlighter that will add the color to each cell.
    hl = MyHighlighter(
        (data, i, j) -> true,
        colormap_decoration,
        min,
        max
    )

    # Print the table.
    pretty_table(df, highlighters = (hl,), backend = :html)
end

# Create a ramdom DataFrames.
df = DataFrame(rand(20, 5), :auto)
print_df(df)

[Dan_Micsa]

Thank you. I have over 40y of programming and 2 days of Julia.

I will play with it. New concepts for me. In C++ I have to write a local class, a function is not possible inside.

[rocco_sprmnt21]

This is what I get.
Do I need to set something on VS Code?

julia> print_df(df)
<!DOCTYPE html>
<html>
<meta charset="UTF-8">
<style>
  table, td, th {
      border-collapse: collapse;
      font-family: sans-serif;
  }

  td, th {
      border-bottom: 0;
      padding: 4px
  }

[Dan_Micsa]

I think this is using HTML back end in a console mode. HTML should be used for conversion to HTML or notebooks that can render it graphically.

try this if you using the text like method:

using PrettyTables

data = Any[1 false 1.0 0x01
    2 true 2.0 0x02
    3 false 3.0 0x03
    4 true 4.0 0x04
    5 false 5.0 0x05
    6 true 6.0 0x06]

h1 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 1, bold = true, background = 0x220044)
h2 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 2, bold = true, background = 0x440088)
h3 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 3, bold = true, background = 0x6600AA)
h4 = Highlighter((data, i, j) -> j in (1, 3, 4) && data[i, j] == 4, bold = true, background = 0x8800CC)

#pretty_table(data; highlighters = (h))
pretty_table(data; highlighters = (h1, h2, h3, h4))
#pretty_table(data; highlighters = (h1, h2, h3, h4), backend = Val(:html))

More information here: Home · Pretty Tables

[Dan_Micsa]

Ronis_BR,

This feels like a bug in the PrettyTables.Crayon. I can’t set both fc and bc. If i set only one of them it works.

using Colors
using ColorSchemes
using DataFrames
using PrettyTables

function pp(df, min_ = nothing, max_ = nothing, isCentered = true, gradientCentered = :coolwarm, gradient = :YlGn_9)    
    if isnothing(min_)
        min_ = minimum(eachcol(df)) |> minimum
    end
    
    if isnothing(max_)
        max_ = maximum(eachcol(df)) |> maximum
    end

    if isCentered && min_ < -0.00001
        min_ = min(min, -max)
        max_ = max(-min, max)
        gradient = gradientCentered
    end

    function colormapDecorationHTML(h, data, i, j)
        color = get(colorschemes[gradient], data[i, j], (min_, max_))
        return HTMLDecoration(color = "#" * (color |> hex), background = convert(Bool, i % 2) ? "#222222" : "#111111")
        #return HTMLDecoration(background = "#" * (color |> hex), color = convert(Bool, i % 2) ? "#222222" : "#111111")
    end

    function colormapDecorationText(h, data, i, j)
        color = get(colorschemes[gradient], data[i, j], (min_, max_))
        color2 = "0x" * (color |> hex)
        color = parse(UInt32, color2)
        #return Crayon(;background = color)
        #return Crayon(;foreground = color)
        #return Crayon(;foreground = 0x888888, background = color)
        return Crayon(;foreground = 0xFFFF00, background = 0x0000FF)#! I can't set both fc and bc !
    end

    # Define the highlighter that will add the color to each cell.
    hlHTML = HTMLHighlighter(f = (data, i, j) -> true, fd = colormapDecorationHTML)
    hlText = Highlighter(f = (data, i, j) -> true, fd = colormapDecorationText)
    
    # Print the table.
    pretty_table(df, highlighters = (hlHTML), backend = :html, formatters = ft_printf("%.4f"))
    pretty_table(df, highlighters = (hlText), formatters = ft_printf("%.4f"))
end

# Create a random DataFrames.
df = DataFrame(rand(10, 5), :auto)
pp(df)

[Ronis_BR]

Hi @Dan_Micsa !

Sorry for the delay, I missed the notification…

I run this code and everything seems to be working correctly:

What is the problem in your case?

[Dan_Micsa]

This code looks OK in console but not in VS Code. I believe I disturb you with things that are not from the PrettyTable library.

[Ronis_BR]

This code looks OK in console but not in VS Code. I believe i disturb you with thinks that are not from the PrettyTable library.

No problem! Please, feel free to contact me anytime

It seems to me it is a problem of neither PrettyTables nor Crayons. I believe it might be something related to the console in VSCode. Can you execute the following code in Julia inside the VSCode console please:

julia> println("\e[38;2;255;255;0;48;2;0;0;255mThis must have blue background\e[0m")
julia> println("\e[33;44mThis must have blue background\e[0m")