Invisible fonts in DataFrame.display() with solarized dark theme in Terminal

When I show a DataFrame object in the REPL, under each row its supposed to list the Type for each series.

using DataFrames
DataFrame(((1,2,3), (4,5,6)))

However, in my terminal, the font color for the Types is the same as the background, making them effectively invisible. I know the values are there, as when I highlight the text, the color inversion makes them visible. See attached screenshot.

12

This is pretty problematic for me, since my IDE is terminal based (nvim) and I’m far too in love with solarized dark to switch to some other color scheme.

Has anyone else dealt with this issue? Since solarized-dark is a pretty common theme, I’m tempted to raise this as an issue in DataFrames.jl package. Should I report this to DataFrames.jl, or is there maybe some other reason for this behaviour?

This happens when I’m running the REPL in the terminal by itself, and also when I use an embedded terminal in nvim.

[edit: I’ll mention I’m on PopOS 21.04, using Tilix… in case others with different terminals/IDEs aren’t running into this issue.]

4 Likes

Open an issue in DataFrames.jl , and if it’s necessary you will be directed to the right place.

DataFrames relies on PrettyTables for printing, so I think it’s probably an issue for there.

1 Like

It can be reported either in DataFrames.jl or PrettyTables.jl, anyway I will ask @Ronis_BR to kindly have look at it :).

However, note that we use dark gray color here. So this is a standard color picked from a set of 16 standard terminal colors. Personally (but it is subjective), I would say that color theme you use should show such a color legibly (what I want to say is that we are not using some fancy and strange color - it is a basic color that normally should be displayed in a legible way). How hard would be to fix your color scheme to display dark gray in a legible way?

B.t.w. how is this table displayed in your terminal?:

julia> DataFrame(x=[missing, DataFrame(x=1)])
2×1 DataFrame
 Row │ x
     │ DataFrame?
─────┼───────────────
   1 │ missing
   2 │ 1×1 DataFrame
2 Likes

It’s known that Solarized’s light_black color is broken (i.e. the same as the default background color). I’m still inclined to tell people to fix their themes so that devs can use all colors fine…

4 Likes

I think the problem is with the theme itself. PrettyTables.jl uses, by default, the color dark_gray when printing the subheader (the type in case of DataFrames.jl). I am not sure what we can do here because your theme is apparently using dark_gray as the background also. You have two options, the first one is to disable colors at all by starting Julia with --color=no. The second is to show DataFrames.jl using the option subheader_crayon. Something like this:

julia> show(df, subheader_crayon = DataFrames.crayon"white")

@pfitzseb good to know. I had no idea solarized dark had issues… it’s a default theme in gnome terminal (on Ubuntu), so I was trusting that it was a safe choice :confused:

@bkamins Solarized is truly aweful in your example.
3

Ya, so its clear this is a solarized issue. Its trivial for me to change the values for that one color to get around the issue, so I can work around this.

The very first version of PrettyTables.jl integration with DataFrames.jl used a global state to configure default options when printing. Unfortunately, it added a huge amount of time to print the first table. I heard that things got much better in Julia 1.7. Maybe we can revisit this to help this kind of issue. What do you think @bkamins ?

@pfitzseb thanks for the info! I just realized that dark_gray and light_black are supposed to be the same :smiley:

The workaround was really effortless on my end. I just changed that single color value to a lighter tone in my terminal, and all is legible and fine now.

Thanks all for the input, and for enlightening me about this silly palette choice in my much loved solarized them :slight_smile: .

2 Likes

Awesome @RTbecard ! Feel free to ping me if you have any other question related to this printing system.

The question is if you can make DataFrames.jl/prettytables.jl at 2b9f6673547259bab9fb3bf3b5224eebc7b11ecd · JuliaData/DataFrames.jl · GitHub a mutable struct (and if doing so would have performance impact). Then we could allow users to tweak it at run time.

There is two problem. The first one is that you mentioned, which highlights the missing and nothing. The second is the keyword argument subheader_crayon that is passed to pretty_tables. In this case, we will need a global variable to store the default options and let the user change them. PrettyTables.jl already have this mechanism of printing with pre-defined, global options (see @pt_conf). However, it leads to a HUGE degradation in time to print the first table. Last time I checked, I can print a rand(2,2) the first time in less than 1s. If I use the global options, then it increases to more than 4s.

1 Like

If you think it is worth it I think you can open an issue in DataFrames.jl to track it and discuss if this is needed. However, I think it is a low priority. A priority should be to have sensible defaults that work well.

1 Like

light_black can be changed to light_green, since it is ok in other colorschemes and gives nice gray color in solarized.

I really did not like this here (black background). It does not make sense to change the default inside PrettyTables.jl since it was design to be easily changed. We can indeed change the default in DataFrames.jl. However, @bkamins must approve that. My personal opinion is that it is not good to change the behavior for everyone because of one theme. The new version does not look good IMHO:

I think it adds too much contrast to an information that is not the principal.

1 Like

Exactly - dark gray was selected exactly to get the opposite. The information should be less not more visible (of course still visible).

1 Like

After a bit more digging, it seems that solarized-dark caused more issues in Julia’s REPL I wasn’t aware of.

Parts of the stack traces were also invisible (line numbers)… :confused:

See below for more extended discussions on the limitations of solarized.

1 Like

Yes, this is known bug (Unreadable stacktrace in solarized theme · Issue #38730 · JuliaLang/julia · GitHub), current solution is to redefine Base.text_color at startup.jl:

Base.text_colors[:light_black] = Base.text_colors[:light_green]

Unfortunately, solarized theme is considered to be part of the “lowest common denominator”, so this state of things is going to stay for a long time (probably forever).

With regard to DataFrame, there is an alternative solution, as discussed in Provide entry point for color customization · Issue #2831 · JuliaData/DataFrames.jl · GitHub

You can add following lines in each script, where you are using DataFrames.jl

using DataFrames, Crayons

Base.display(df::AbstractDataFrame) = show(df, header_crayon = crayon"light_green")

This is piracy, but this is piracy for the greater cause, so it is fine.