Help testing PrettyTables v2

Ronis_BR · February 19, 2022, 2:29pm

Hi!

I have been working for a while in a huge rewrite of PrettyTables.jl internals. When I first developed this package, it was thought to be something straightforward: take a matrix and print it in a formatted table. I did not think about cropping, highlighting, multiple back-ends, etc. The result was in a suboptimal code since added features without tweaking how information flows.

I take some time and fix the concept. The code is now in master. All tests are passing, but I really need help from people who normally uses PrettyTables.jl to check if I miss something. The more immediate advantage is the performance. Take a look at the time to print the first table between master and v1.3:

Notice that we have breaking changes in HTML and LaTeX environments. I would love feedback on that too.

The idea is to tag v2.0 when we are ready to be the LaTeX printing backend of DataFrames.jl.

alusiani · February 20, 2022, 9:04pm

Thanks for doing this work! I have very little spare time but will be glad to test these changes.

eliascarv · February 20, 2022, 11:34pm

PrettyTables.jl is amazing! I love this packege.
I will definitely test the v2.

Ronis_BR · February 21, 2022, 12:32am

Thank you very much!!

mcreel · February 21, 2022, 8:41am

Working as expected for me, after making slight adjustments. I only use headers and highlighters, nothing else. Thanks!

Ronis_BR · February 21, 2022, 11:25am

Perfect! Thanks!

carstenbauer · February 21, 2022, 12:57pm

Just gave it a quick try with JuliaCon.jl. Seems to work just fine without any changes!

Ronis_BR · February 21, 2022, 4:08pm

Thanks @carstenbauer ! You gave me a very good idea. I will do the same with DataFrames (which uses a lot of options) to check if I broke something

alusiani · March 1, 2022, 9:32pm

Hi,

I have run a notebook I created some time ago to use PrettyTables to print matrices too large to fit on the screen width following what R does, and using today PrettyTables#master. I had to use the “header” keyword instead of setting the header with the 2nd argument, but other than that everything worked. I attach my notebook in the following:

functions for printing labelled vectors and matrices

uses PrettyTables.jl
simpler and more symmetric arguments
provides ability to wrap large matrices to fit limited screen area

## using Formatting
## pyfmt(tfmt, tstr) = fmt(tfmt, tstr)
using Format
using PrettyTables
using DataFrames
using Measurements
using Random

Random.seed!(1234);

##
## split matrix in submatrices of defined number of rows and columns
## for printing in a device with limited amount of rows and columns
##
function splitmatrix(
    matr::AbstractMatrix,
    nrow::Int, ncol::Int;
    col_names::Array{String, 1}=Array{String, 1}(),
    row_names::Array{String, 1}=Array{String, 1}(),
    topleft::String=""
  )
  
  if length(row_names) > 0 && length(row_names) != size(matr, 1)
    throw(DomainError(row_names, "row names number must match number of rows of matrix"))
  end
  if length(col_names) > 0 && length(col_names) != size(matr, 2)
    throw(DomainError(col_names, "column names number must match number of columns of matrix"))
  end
  col_beg = range(1, size(matr, 2), step=ncol)
  col_end = vcat(col_beg[2:end] .- 1, size(matr, 2))
  row_beg = range(1, size(matr, 1), step=nrow)
  row_end = vcat(row_beg[2:end] .- 1, size(matr, 1))
  [
    begin
      rc = matr[rbeg:rend, cbeg:cend]
      if length(col_names) > 0
        rc = vcat(permutedims(col_names[cbeg:cend]), rc)
      end
      if length(row_names) > 0
        if length(col_names) > 0
          row_names1 = vcat(topleft, row_names[rbeg:rend])
        else
          row_names1 = row_names[rbeg:rend]
        end
        rc = hcat(row_names1, rc)
      end
      rc
    end
    for (cbeg, cend) in zip(col_beg, col_end),
      (rbeg, rend) in zip(row_beg, row_end)
  ]
end

function splitmatrix(
    matr::AbstractVector,
    nrow::Int, ncol::Int;
    kwargs...
  )
  splitmatrix(reshape(matr, 1, length(matr)), nrow, ncol; kwargs...)
end

nothing

##
## formatter for PrettyTables
## - formats numbers
##

function pt_format1(fmtstr::String, v)
  return typeof(v) <: Number ? pyfmt(fmtstr, v) : v
end

pt_format(fmtstr::String) = pt_format([fmtstr])
pt_format(fmtstr::String, column::Int) = pt_format(fmtstr, [column])
pt_format(fmtstr::String, columns::AbstractVector{Int}) =
  pt_format([fmtstr for i = 1:length(columns)], columns)

function pt_format(fmtstr::Vector{String}, columns::AbstractVector{Int}=Int[])
  lc = length(columns)
  
  lc == 0 && (length(fmtstr) != 1) &&
  error("If columns is empty, then fmtstr must have only one element.")
  
  lc > 0 && (length(fmtstr) != lc) &&
  error("The vector columns must have the same number of elements of the vector fmtstr.")
  
  if lc == 0
    return (v, i, j) -> pt_format1(fmtstr[1], v)
  else
    return (v, i, j) -> 
    begin
      @inbounds for k = 1:length(columns)
        if j == columns[k]
          return pt_format1(fmtstr[k], v)
        end
      end
      return v
    end
  end
end

##
## formatter for PrettyTables
## - formats numbers and Measurements
##

function pt_format_meas1(fmtstr::String, v)
  if typeof(v) <: Measurement
    return pyfmt(fmtstr, v.val) * " ± " * pyfmt(fmtstr, v.err)
  else
    return typeof(v) <: Number ? pyfmt(fmtstr, v) : v
  end
end

pt_format_meas(fmtstr::String) = pt_format_meas([fmtstr])
pt_format_meas(fmtstr::String, column::Int) = pt_format_meas(fmtstr, [column])
pt_format_meas(fmtstr::String, columns::AbstractVector{Int}) =
  pt_format([fmtstr for i = 1:length(columns)], columns)

function pt_format_meas(fmtstr::Vector{String}, columns::AbstractVector{Int}=Int[])
  lc = length(columns)
  
  lc == 0 && (length(fmtstr) != 1) &&
  error("If columns is empty, then fmtstr must have only one element.")
  
  lc > 0 && (length(fmtstr) != lc) &&
  error("The vector columns must have the same number of elements of the vector fmtstr.")
  
  if lc == 0
    return (v, i, j) -> pt_format_meas1(fmtstr[1], v)
  else
    return (v, i, j) -> 
    begin
      @inbounds for k = 1:length(columns)
        if j == columns[k]
          return pt_format_meas1(fmtstr[k], v)
        end
      end
      return v
    end
  end
end

##--- hilight 1st column for PrettyTables
const pt_hili_col1 = Highlighter((data, i, j) -> j==1, crayon"bold")

##--- hilight 1st column for PrettyTables
const pt_hili_row1 = Highlighter((data, i, j) -> i==1, crayon"bold")

##--- simple latex format for PrettyTables
const latex_simple2 = LatexTableFormat(
  top_line       = "\\toprule",
  header_line    = "\\midrule",
  mid_line       = "\\midrule",
  bottom_line    = "\\bottomrule",
  left_vline     = "|",
  mid_vline      = "|",
  right_vline    = "|",
  header_envs    = [],
  subheader_envs = []
)

nothing

##
## alupt()
##
## print vector or matrix with optional row and column names
## using PrettyTables with alternative simpler arguments
## defaults to table sandwitched between just top-bottom horizontal lines
##
## arguments:
## - row_names, col_names
## - fmt: <C printf format string, e.g. ".4f">
## - hlines, vlines: PrettyTables definitions
## - topleft: label for top-left corner
## - other args passed to PrettyTables
##
## note, good example for PrettyTables functionality
## https://discourse.julialang.org/t/help-testing-new-features-of-prettytables-jl/36283
##

alupt(data::Number, kwargs...) = alupt(stdout, [data]; kwargs...)

alupt(data::AbstractVecOrMat; kwargs...) = alupt(stdout, data; kwargs...)

function alupt(::Type{String}, data::AbstractVecOrMat; kwargs...)
  io = IOBuffer()
  alupt(io, data; kwargs...)
  String( take!(io) )
end

function alupt(io::IO, data::AbstractVecOrMat; kwargs...)
  kwargs_dict = Dict{Symbol, Any}(kwargs)
  
  topleft = get(kwargs, :row_name_column_title, "")
  delete!(kwargs_dict, :row_name_column_title)
  topleft = get(kwargs, :topleft, topleft)
  delete!(kwargs_dict, :topleft)
  
  delete!(kwargs_dict, :noheader)
  header = []
  row_names = []

  if haskey(kwargs, :col_names)
    delete!(kwargs_dict, :col_names)
    header = kwargs[:col_names]
    !(header isa AbstractVector) && (header = [header])
  end
  
  if haskey(kwargs, :row_names)
    delete!(kwargs_dict, :row_names)
    row_names = kwargs[:row_names]
    !(row_names isa AbstractVector) && (row_names = [row_names])
  end
  
  if length(header) != 0 && length(row_names) != 0
    if length(header) == size(data, 2)+1
      topleft = header[begin]
      header = header[begin+1:end]
    end
    if length(row_names) == size(data, 1)
      row_names = vcat(topleft, row_names)
    end
    data = vcat(reshape(header, 1, length(header)), data)
    data = hcat(row_names, data)
  elseif length(header) != 0 && length(row_names) == 0
    if length(header) == (size(data, 2)-1)
      header = vcat(topleft, header)
    end
    (data isa AbstractVector) && (data = reshape(data, 1, length(data)))
    data = vcat(reshape(header, 1, length(header)), data)
  elseif length(header) == 0 && length(row_names) != 0
    if length(row_names) == size(data, 1)-1
      row_names = vcat(topleft, row_names)
    end
    data = hcat(row_names, data)
  end
  
  hlines = get(kwargs, :hlines, Array{Symbol,1}([]) )
  delete!(kwargs_dict, :hlines)
  if hlines != :none
    !(hlines isa AbstractVector) && (hlines = [hlines])
    hlines = unique(vcat(hlines, [:begin, :end]) )
  end

  vlines = get(kwargs, :vlines, Array{Symbol,1}([]) )
  delete!(kwargs_dict, :vlines)
  !(vlines isa AbstractVector) && (vlines = [vlines])
    
  highlighters = get(kwargs, :highlighters, tuple())
  !(highlighters isa Tuple) && (highlighters = (highlighters,) )
  delete!(kwargs_dict, :highlighters)
  if length(header) != 0 || get(kwargs, :auto_colnames, false) != false
    highlighters = (highlighters..., pt_hili_row1)
    if !isnothing(findfirst(x -> x == :header, hlines))
      hlines = filter(x -> x != :header, hlines)
      if size(data, 1) > 1
        hlines = vcat(hlines, 1)
      end
    end
  end
  if length(row_names) != 0 || get(kwargs, :auto_rownames, false) != false
    highlighters = (highlighters..., pt_hili_col1)
    if !isnothing(findfirst(x -> x == :header, vlines))
      vlines = filter(x -> x != :header, vlines)
      if size(data, 2) > 1
        vlines = vcat(vlines, 1)
      end
    end
  end
  push!(kwargs_dict, :highlighters => highlighters)
  push!(kwargs_dict, :hlines => hlines)
  push!(kwargs_dict, :vlines => vlines)
  
  formatters = get(kwargs, :formatters, tuple())
  delete!(kwargs_dict, :formatters)
  if haskey(kwargs, :fmt)
    formatters = (formatters..., pt_format(kwargs[:fmt]))
    delete!(kwargs_dict, :fmt)
  end
  push!(kwargs_dict, :formatters => formatters)

  push!(kwargs_dict, :noheader => true)
  delete!(kwargs_dict, :auto_rownames)
  delete!(kwargs_dict, :auto_colnames)
  pretty_table(io, data; header=header, kwargs_dict...)
end
    
nothing

##
## aluptmatr()
##
## print matrix with no frills and default fmt
##
aluptmatr(matr; kwargs...) = aluptmatr(stdout, matr; kwargs...)

function aluptmatr(::Type{String}, matr; kwargs...)
  io = IOBuffer()
  aluptmatr(io, matr; kwargs...)
  return String(take!(io) )
end

function aluptmatr(io::IO, matr::AbstractVecOrMat; screenrow=132, screencol=132, topleft="", kwargs...)
  kwargs_dict = Dict{Symbol, Any}(kwargs)
  delete!(kwargs_dict, :screenrow)
  delete!(kwargs_dict, :screencol)
  delete!(kwargs_dict, :topleft)
  alupt(matr; fmt=".4f", topleft=topleft, display_size=(screenrow, screencol), kwargs_dict...)
end

function aluptmatr(io::IO, matr::DataFrame; kwargs...)
  aluptmatr(io, convert(Matrix, matr);  kwargs...)
end

##
## aluptmatrwr()
##
## print large matrix wrapping elements as specified
## in small blocks that can be read on a limited size screen
##
## maxcol: maximum number of columns printed horizontally
## maxrow: maximum number of rows printed vertically
##
aluptmatrwr(matr; kwargs...) = aluptmatrwr(stdout, matr; kwargs...)

function aluptmatrwr(::Type{String}, matr; kwargs...)
  io = IOBuffer()
  aluptmatrwr(io, matr; kwargs...)
  return String(take!(io) )
end

function aluptmatrwr(io::IO, matr;
    fmt=".4f", rowsep="\n", maxrow=11, maxcol=11, screenrow=132, screencol=132, topleft="",
    row_names::Array{String, 1}=Array{String,1}(), col_names::Array{String, 1}=Array{String,1}()
  )
  print(io, join(alupt.(
        String, splitmatrix(matr, maxrow, maxcol, row_names=row_names, col_names=col_names, topleft=topleft),
        formatters = pt_format(fmt), display_size=(screenrow, screencol), tf=tf_borderless, hlines=:none),
      rowsep))
end
    
nothing

##
## examples
##

##
## print vector
##
alupt(rand(4), fmt=".4f")

##
## print matrix
##
alupt(rand(4,4), fmt=".4f")

##
## print matrix
##
alupt(
  rand(4, 4),
  fmt = ".4f",
  topleft = "example",
  row_names = "row_" .* ["a", "b", "c", "d"],
  col_names = "col_" .* ["a", "b", "c", "d"]
)

────────
 0.5908
 0.7668
 0.5662
 0.4601
────────
────────────────────────────────
 0.7940  0.2468  0.0664  0.2760
 0.8541  0.5797  0.9568  0.6517
 0.2006  0.6489  0.6467  0.0566
 0.2986  0.0109  0.1125  0.8427
────────────────────────────────
─────────────────────────────────────────
e[1m example e[0me[1m  col_a e[0me[1m  col_b e[0me[1m  col_c e[0me[1m  col_d e[0m
e[1m   row_a e[0m 0.9505  0.8212  0.1278  0.2469
e[1m   row_b e[0m 0.9647  0.0342  0.3742  0.0118
e[1m   row_c e[0m 0.9458  0.0945  0.9311  0.0460
e[1m   row_d e[0m 0.7899  0.3149  0.4389  0.4962
─────────────────────────────────────────

##
## example print large matrix
##
labels = vcat([s1*s2 for s1 in ["a", "b", "c", "d"], s2 in["1", "2", "3"]]...)
aluptmatrwr(
  rand(12, 12),
  maxcol = 8,
  maxrow = 8,
  topleft = "example",
  row_names = "row_" .* labels,
  col_names = "col_" .* labels
)

 example  col_a1  col_b1  col_c1  col_d1  col_a2  col_b2  col_c2  col_d2
  row_a1  0.7320  0.4050  0.6437  0.8273  0.4987  0.3866  0.9521  0.3542
  row_b1  0.2991  0.4995  0.4014  0.0993  0.0940  0.3306  0.7950  0.1329
  row_c1  0.4492  0.6588  0.5251  0.6343  0.5251  0.7480  0.4681  0.3008
  row_d1  0.8751  0.5156  0.6120  0.1327  0.2655  0.2656  0.0952  0.1945
  row_a2  0.0463  0.2607  0.4326  0.7752  0.1101  0.2911  0.7278  0.8837
  row_b2  0.6984  0.5955  0.0822  0.8692  0.8344  0.6126  0.9825  0.0678
  row_c2  0.3651  0.2925  0.1991  0.0396  0.6334  0.7058  0.4270  0.0288
  row_d2  0.3025  0.2886  0.5761  0.7904  0.3379  0.5084  0.4679  0.5646

 example  col_a3  col_b3  col_c3  col_d3
  row_a1  0.1463  0.2949  0.4699  0.3412
  row_b1  0.6471  0.6963  0.9940  0.5238
  row_c1  0.9985  0.3324  0.3558  0.6618
  row_d1  0.4954  0.4151  0.1651  0.8330
  row_a2  0.6543  0.6036  0.6826  0.4989
  row_b2  0.8349  0.0444  0.8066  0.9379
  row_c2  0.4157  0.9157  0.7080  0.9569
  row_d2  0.1828  0.3723  0.5154  0.6103

 example  col_a1  col_b1  col_c1  col_d1  col_a2  col_b2  col_c2  col_d2
  row_a3  0.3726  0.6182  0.2182  0.4312  0.1130  0.4726  0.8489  0.5989
  row_b3  0.1505  0.6643  0.3620  0.1377  0.7830  0.6126  0.5586  0.3037
  row_c3  0.1473  0.7535  0.2047  0.6081  0.8380  0.1926  0.8074  0.5007
  row_d3  0.2834  0.0369  0.9330  0.2551  0.0879  0.8511  0.0134  0.4100

 example  col_a3  col_b3  col_c3  col_d3
  row_a3  0.8365  0.1130  0.6086  0.0296
  row_b3  0.4383  0.6702  0.8880  0.6854
  row_c3  0.6900  0.1405  0.4948  0.6771
  row_d3  0.3723  0.0813  0.0895  0.6958

brianguenter · March 2, 2022, 3:03am

Is there a way to pretty print all the rows in an array that has more rows than can fit on the screen? I tried various keyword arguments that seemed like plausible ways to do this but nothing worked. Any rows that would have flowed off the bottom of the screen were removed.

Ronis_BR · March 2, 2022, 7:30am

Hi @alusiani !

Awesome! Thanks for the information:)

Hi @brianguenter !

Is the option crop = :horizontal what you are looking for?

oheil · March 2, 2022, 9:32am

Fiddling around a bit with that and the idea to combine PrettyTables with TerminalPager (also by @Ronis_BR ) I found the following solution, but it’s a bit painful, because I needed a third package.

Anyways I show you and perhaps there is a easier, more straight forward way to do it:

julia> using Suppressor, PrettyTables, TerminalPager

julia> df=rand(100,100);

julia> ( @capture_out pretty_table(stdout,df) ) |> pager

If pretty_table could be told to return the pretty table as a string it would be much easier.

Ronis_BR · March 2, 2022, 11:58am

Hi @oheil !

You can return a string using pretty_table(String, rand(100, 100)).

oheil · March 2, 2022, 12:07pm

Great, which makes:

julia> using PrettyTables, TerminalPager

julia> df=rand(100,100);

julia> pretty_table(String,df) |> pager

Very satisfying (except that I missed the obvious)

oheil · March 2, 2022, 1:31pm

actually that leads to an improvement of the help string, which you get when

help?> pretty_table

What you get is quite long, and the (in my case) beginner information is scrolled far away.
Adding, for example, some usage examples at the end of the doc string, would help, I guess, for the most use cases, without the need to scroll to the top.

Ronis_BR · March 2, 2022, 1:45pm

Yeah, the help string for pretty_table is quite long. If I add examples at the end, you would still need to scroll a lot because there are too many options. That’s one of the reasons why I created TerminalPager.jl. Using it, you can type |? to go to the help pager mode, and then you get a good experience navigating through the help as if you were using Linux less command.

oheil · March 2, 2022, 1:49pm

Genius!

Ronis_BR · March 2, 2022, 2:00pm

Thanks! I hope it will be useful for you.

pfitzseb · March 2, 2022, 2:09pm

Base’s “Extended help” functionality might also be worth a look:

julia> """
       foo
       bar

       # Extended help
       baz
       """
       f(x) = x
f

help?> f
search: f fd for fma fld fld1 fill fdio frexp foldr foldl flush floor float first fill! fetch fldmod filter falses finally

  foo bar

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Extended help is available with `??`

help?> ?f
search: f fd for fma fld fld1 fill fdio frexp foldr foldl flush floor float first fill! fetch fldmod filter falses finally

  foo bar

  Extended help
  ≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡

  baz

Ronis_BR · March 2, 2022, 6:16pm

I agree, we might be able to organize better the help of the function pretty_table.