GR build problems: "installation is incomplete"

[compleat]

@carstenbauer

(And I also offered a last-resort alternative like mybinder.org were everything was guaranteed (because tested) to work.)

That is good to know! Having a backup can be a lifesaver - for teacher as well as student.

[mkitti]

Background: GR Binaries

There are two ways to provide binaries for GR which hinge on the environmental variable JULIA_GR_PROVIDER. You can evaluate that environmental variable by checking the value of ENV["JULIA_GR_PROVIDER"] within Julia or use operating system and shell specific means outside of Julia. The two methods are “BinaryBuilder” and “GR” with “BinaryBuilder” being preferred.

https://github.com/jheinen/GR.jl/blob/master/deps/build.jl#L70-L92

The BinaryBuilder binary provider method uses GR_jll which are binaries built via cross-compilation by the Julia’s BinaryBuilder.org ecosystem: GitHub - JuliaBinaryWrappers/GR_jll.jl

The GR method binary provider downloads tarballs from https://gr-framework.org/downloads/

Problems can occur if the binaries and the Julia package get out of sync. You may be trying to update GR_jll, but at the end of the day it will not matter because you are actually use the tarball. Going the other way, you might be trying to redownload the tarball, but perhaps the Julia package is using GR_jll.

Source of the Error

@compleat

The initial error message many of them are getting is:

Your GR installation is incomplete.   Rerun build step for GR package

The error comes from GR.jl’s __init__ function (nb: GR.jl has an __init__ and an init function) when it cannot locate the directory of the GR install via the Julia variable grdir in that function. That Julia variable can modified via the environmental variable GRDIR.

jheinen/GR.jl/blob/813fba7c2731751701628f3c366990182c7718b2/src/GR.jl#L259-L286

jheinen/GR.jl/blob/1e6d3d526ed8ca5757f371c618fd41f83a1012c1/src/GR.jl#L277-L280

Troubleshooting

I recommend working backwards in situations like this.

1. What is the value of the local variable grdir in GR.__init__ ?
2. What is the value of GR.gr_provider?

julia> GR.gr_provider
"BinaryBuilder"

BinaryBuilder

GR_jll.jl is a binary package created by Julia’s binary builder ecosystem. It is distinct from GR.jl. If you are having trouble with GR_jll.jl please make sure to indicate that when seeking help.

If the value of GR.gr_provider is “BinaryBuilder”, the continue troubleshooting down this path:

3. What is the value of joinpath(dirname(GR_jll.libGR_path), "..")? You may need to manually load GR_jll via using GR_jll to evaluate this and may need to explicitly add it to your environment.
4. What is located at the value of the expression above? Is it valid?
5. What is the version of GR_jll in your manifest?

If you determine that GR_jll is the problem, please indicate that when seeking help.

GR or other values of GR.gr_provider

If the value of GR.gr_provider is not “BinaryBuilder”, then we may need to investigate how that came to be.

jheinen/GR.jl/blob/813fba7c2731751701628f3c366990182c7718b2/src/GR.jl#L13-L32

GR.gr_provider is a compile time constant which is contingent on deps.jl. Locate “deps.jl” by obtaining the value of the compile time constant GR.depsfile. You should then examine the content of deps.jl. deps.jl is not located in the GR.jl source tree, because it is generated by build.jl:

https://github.com/jheinen/GR.jl/blob/master/deps/build.jl#L78-L86

build.jl is invoked when you do “] build GR” in the package interface or do using Pkg; Pkg.build("GR").

Looking through build.jl there are two environmental variables that influence the selection of a binary provider:

1. GRDIR
2. JULIA_GR_PROVIDER

Further troubleshooting would thus involve clearing or manipulating these environmental variables and then invoking ] build GR. That should in turn influence the existence or contents of “deps.jl” and GR.gr_provider.

Troubleshooting Steps for non-BinaryBuilder binary provider install

The first step to fix this is just to do

Tiny Plots in jupyter-notebook

Please update using ENV["GRDIR"]=""; ] build GR.

That should look like this, first checking what the current value of the GRDIR environmental variable:

julia> ENV["GRDIR"]
"C:\\Users\\kittisopikulm\\.julia\\artifacts\\7d224ea2f9a0db864ae942e168f53343cb07aaa6\\bin\\.."

julia> ENV["GRDIR"] = ""
""

(@v1.6) pkg> build GR
    Building GR → `C:\Users\kittisopikulm\.julia\dev\GR\deps\build.log`
...

What does ENV["GRDIR"]=""; ] build GR do?

The first thing it does is looks for a “gr” folder in your home directory, “/opt”, “/usr/local”, or “/usr”. If you’re on Windows, you probably only have a home directory.

https://github.com/jheinen/GR.jl/blob/master/deps/build.jl#L16-L22

julia> homedir()
"C:\\Users\\kittisopikulm"

If that fails get_grdir returns Nothing. (It returns the type Nothing as opposed to an instance of nothing.)

Later, this in turn results in provider being set to "BinaryBuilder":

https://github.com/jheinen/GR.jl/blob/master/deps/build.jl#L72-L73

It’s important to check this, since a couple of weeks ago, that was not the case:

Use GR as binary provider

jheinen/GR.jl/blob/1e6d3d526ed8ca5757f371c618fd41f83a1012c1/deps/build.jl#L70-L76

More directly you can also set the environmental variable ENV["JULIA_GR_PROVIDER"] explicitly to either “BinaryBuilder” or “GR” before rebuilding via “] build GR”.

Summary

1. Collect information based on tracing the source of the error:
2. What is the version of the GR and GR_jll packages?
3. What is the value of GR.gr_provider?
4. What are the values of the environmental variables JULIA_GR_PROVIDER and GRDIR?
5. What happens when you clearr the environmental variable GRDIR?

julia> ENV["GRDIR"]
"C:\\Users\\kittisopikulm\\.julia\\artifacts\\7d224ea2f9a0db864ae942e168f53343cb07aaa6\\bin\\.."

julia> ENV["GRDIR"] = ""
""

(@v1.6) pkg> build GR
    Building GR → `C:\Users\kittisopikulm\.julia\dev\GR\deps\build.log`
...

Does any of the above variables and constants change?
6. What happens when you do ENV["JULIA_GR_PROVIDER"] = "BinaryBuilder" and then do using Pkg; pkg"build GR" ?
7. What happens when you do ENV["JULIA_GR_PROVIDER"] = "GR" and then do using Pkg; pkg"build GR" ?

I hope this expanded guide to troubleshooting GR helps you and your students. More generally when you get stuck like this I suggest tracing the source of the error and then walking through the process to debug. When you discover constants or environmental variables try evaluating to see what they are and then later see if modifying them changes the situation.

[compleat]

Thanks for taking the time, but at the end of the day, my students and I just need a platform where plotting works. Maybe your comments will help someone make it work reliably.

Thanks again.

[mbauman]

@compleat

Maybe your comments will help someone make it work reliably.

I understand your frustration here, but the real goal would be for someone to give us a bit more information about how it broke so we can fix it properly: once and for all.

[compleat]

Thank you, but (with respect) when I set Julia to my class, I thought it would have developed far enough along to be able to reliably plot (even in Windows).

[mkitti]

It’s being actively developed, and there are moving pieces. The situation today is different that it was two weeks ago. Part of the current situation is the package is transitioning to using BinaryBuilder provided binaries.

We do have a way of version pinning it to make it work as some have described above. pkg> add GR@0.54 or using a Manifest file which let’s you grab everything at exactly the same version.

The gist of the extra information I added is that you then need to do:

julia> ENV["GRDIR"] = ""
""

(@v1.6) pkg> build GR

It’s the first thing that @jheinen tells everyone.

If you’re trying a new version of GR, such as GR v0.57.1, then they key piece of information we need is

julia> GR.gr_provider
"BinaryBuilder"

If the above value is still “GR” rather than “BinaryBuilder”, then it may be necessary to force GR to recompile itself.

For example, rather than deleting the entire .julia, you could focus on the the location of dirname(dirname(pathof(GR))).

Here, I’ve artificially created a situation with the latest GR.

julia> using GR
Your GR installation is incomplete. Rerun build step for GR package.
    Building GR → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/f74b42150042d11a1d94badf27c6125867c7e9bd/build.log`

julia> GR.gr_provider # This should be "BinaryBuilder" in normal circumstances
"GR"

julia> ENV["GRDIR"]
"/home/mkitti/.julia/packages/GR/f111O/src/../deps/gr"

julia> isfile(ENV["GRDIR"]) # This is trying to point at something that doesn't exist, not good
false

julia> ENV["GRDIR"] = "" # Clear the environmental variable

julia> GR_path = dirname(dirname(pathof(GR))) # Figure out where GR lives
"/home/mkitti/.julia/packages/GR/f111O"

(@v1.6) pkg> rm GR # Remove it from your environment
    Updating `~/.julia/environments/v1.6/Project.toml`
  [28b8d3ca] - GR v0.57.1
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`

julia> rm(GR_path; force = true, recursive = true) # Remove the GR package files to force recompilation

julia> ENV["GRDIR"] # Double check the environment is clean
""

(@v1.6) pkg> add GR
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
   Installed GR ─ v0.57.1
    Updating `~/.julia/environments/v1.6/Project.toml`
  [28b8d3ca] + GR v0.57.1
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`
    Building GR → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/f74b42150042d11a1d94badf27c6125867c7e9bd/build.log`
Precompiling project...
  Progress [========================================>]  2/2
  ? Plots
1 dependency successfully precompiled in 12 seconds (119 already precompiled)
1 dependency failed but may be precompilable after restarting julia

julia> exit() # Restart Julia
$ bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using GR

julia> GR.gr_provider # Better
"BinaryBuilder"

julia> x = [1.0, 2.0, 3.0]; y = rand(Float64,3) # This works
3-element Vector{Float64}:
 0.1945528819219915
 0.2910514639410986
 0.6935992470239583

julia> plot(x,y)

If you’re not interested in knowing why you need to do that, that’s fine with me.

[mbauman]

My hunch is that this is the result of a readonly package depot, and thus the deps/deps.jl file isn’t being created.

Did the build step error?

1.6’s Preferences.jl should help packages avoid this issue.

[mkitti]

Looking closely at what is happening with the latest GR, these constants are getting set during the first precompilation:

https://github.com/jheinen/GR.jl/blob/master/src/GR.jl#L13-L32

If depsfile did not exist when the package was compiled, but now exists, the package might not know since it has not been recompiled. It may be necessary to move more of this into GR.__init__.

[jheinen]

Thanks @mkitti for clearifying this.

From several sides I have been asked for a BinaryBuilder integration. Unfortunately I cannot remove the “old” installation mechanisms. Besides the previous method, there are also many users who prefer a single GR installation for different languages (C, Python, Julia, Ruby) - by the way, this is how I do it in our environment(s).

Problems like those described by several people here when updating GR and/or Plots must not occur. I fully agree that the current state is not sustainable.

At the moment, however, I don’t see how I can simplify this. Maybe I am doing something wrong in my build script. Any help is appreciated.

[mkitti]

I’ve got a PR on the way. It’s a bit heavy handed though because it changes the library constants to constant Refs.

https://github.com/jheinen/GR.jl/pull/386

One trick is I used is to do touch(pathof(GR)) in order to force GR.jl to recompile after rebuilding it.

[compleat]

@mkitti

The gist of the extra information I added is that you then need to do:

julia> ENV["GRDIR"] = ""
""

(@v1.6) pkg> build GR

I remember now… When I had this problem before (about 6 months ago), after searching for several days, I finally came upon this solution, and it worked brilliantly.
I had just plain forgotten this.

I wonder if this (setting the directory to “”) might be added to the error message? At the moment, it says to rebuild GR, but I think if this were added, it would save lots of headaches. Thanks for that!

[jdad]

To switch to Plotly backend in a transparent way, just add the following to your .julia/config/startup.jl

ENV["PLOTS_DEFAULT_BACKEND"] = "pyplot"

[Alexander-Barth]

Thanks Carsten for sharing this, which is very useful! We also use jupyter for teaching and most problems are related to the installation of Conda + IJulia. For some students we could narrow it down to the presence of special characters in the user name (https://github.com/JuliaPy/Conda.jl/issues/199). I guess umlauts in user names should be quite common too in your place . As you advice to install anaconda manually, if IJulia fails, I am wondering if your students have been bitten by this issue too.

[compleat]

You may wish to have a look at:

Issues encountered teaching classes with Julia for the first time

This semester, for the first time, I have assigned Julia as a required language in my Data Science classes, alongside R. While I am not a beginner to Julia, I am posting this here because I am a novice at trying to use it in teaching. By and large, there is some good work getting done, but I am facing a huge problem in student forums. To students (especially the less technical ones), Julia seems ‘buggy’ and I seem to be the target of resentment for my having chosen it. Students initially fo…

[carstenbauer]

No, I don’t think we’ve run into the umlaut problem often (at all?). I think that umlauts are not super common in German names and also that (French) accents and such are leading to issues more often than plain umlauts. But of course, we could have just been lucky.

The vast majority of issues that students had with installation were not related to Julia or a Julia package at all. Instead is was almost always a python dependency problem: we are using Jupyter / IJulia, matplotlib (through PyPlot.jl) for plotting, and, for some courses, SymPy.jl. Typically, the student was unaware of an old broken/incompatible python installation - which he might not have even consciously installed - which made the Julia packages think “let’s use this system python” and made things fail for various individual reasons. That’s also why I’m super curious about Symbolics.jl (to replace our modest use of SymPy.jl), Makie.jl as a super promising (hopefully soon stabilising) awesome pure-Julia plotting solution (matplotlib and 3D or animations is something else…) and, most importantly, Pluto.jl as a pure-Julia replacement for Jupyter (I would need a switch to turn off the reactivity feature though, which might never exist… I’m aware of Neptune.jl.).

Otherwise, we faced a few networking issues (pkg operations behind a firewall or similar) or students executing Julia commands that they found on the internet to “fix” a problem but instead making things worse (like mangling with the toml files without understanding what they are doing). But this the minority (compared to python problems) within the minority of cases where significant issues occurred at all.

[Alexander-Barth]

Typically, the student was unaware of an old broken/incompatible python installation

Yes, I had precisely this problem too with some students. I believe that if a user chooses to add anaconda to the PATH (on windows), you will run into issues with Conda.jl.

There is so much useful information in this thread and related threads, I am wondering if it would be useful to have a common wiki with troubleshooting and best practice for teaching (for different software stacks), maybe under a new JuliaEducation or JuliaTeaching github organization? Or directly adding this troubleshooting information directly to the respective Julia packages (documentation of more helpful error message)?

[antoine-levitt]

@carstenbauer

The vast majority of issues that students had with installation were not related to Julia or a Julia package at all. Instead is was almost always a python dependency problem: we are using Jupyter / IJulia, matplotlib (through PyPlot.jl) for plotting, and, for some courses, SymPy.jl. Typically, the student was unaware of an old broken/incompatible python installation - which he might not have even consciously installed - which made the Julia packages think “let’s use this system python” and made things fail for various individual reasons.

I tell students who are not comfortable managing their python installation to do ENV["PYTHON"]="" before installing PyCall. It does take a while because julia has to reinstall python+packages, but that way I don’t have to deal with the 100 different python versions and package managers they have on their systems. The biggest problem is that there is no indication of what’s going on and so the students panic before the installation completes. There are firewall issues, but in that case students just make their laptop go through their phones’ 4G.

[mkitti]

We merged some PRs in the last few hours. v0.57.2 as released has an issue with touch for the reason that @mbauman brought up, but should succeed if you manually build it.

We shouldn’t need that trick anymore since we always try to include deps.jl now. On #master with Using Libdl.dlsym to dynamically load function pointers by mkitti · Pull Request #388 · jheinen/GR.jl · GitHub merged, things are really robust and no longer has the touch issue.

If you need the fix now, do

] add https://github.com/jheinen/GR.jl#master

Otherwise, I expect that @jheinen will release current master as v0.57.3.

I added a fallback from GR_jll to the GR provided tarball which seems to be working out pretty well. GR_jll is failing on Mac right now due to a problem with QtBase5_jll, but the fallback allows GR to succeed.

Travis CI - Test and Deploy with Confidence

Using low-level GR I can achieve a time to first plot in 0.73 seconds on Julia 1.6.0 on Linux.

$ julia-1.6.0/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time begin
           using GR
           x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
           y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
           GR.polyline(x,y)
           GR.axes(0.2, 0.2, 0, 0, 1, 1, -0.01)
           GR.updatews()
       end
  0.738510 seconds (378.15 k allocations: 27.209 MiB, 2.06% gc time, 0.71% compilation time)

[jheinen]

@mkitti : Many thanks for the active support.

GR.jl v0.57.3 will be merged in the next round …

[mkitti]

On Windows, the GKS QtTerm takes a few seconds longer to load for some reason. If you need a quick plot on Windows, try ENV["GKSwstype"] = 41. I got it down to 0.854989 seconds there.

julia> ENV["GKSwstype"] = 41
41

julia>  @time begin
                  @time using GR
                  x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
                  y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
                  @time GR.polyline(x,y)
                  @time GR.axes(0.2, 0.2, 0, 0, 1, 1, -0.01)
                  @time GR.updatews()
              end
  0.736812 seconds (358.68 k allocations: 24.924 MiB, 0.44% compilation time)
  0.027033 seconds (4.20 k allocations: 239.321 KiB, 16.08% compilation time)
  0.012760 seconds (7.61 k allocations: 439.718 KiB, 49.23% compilation time)
  0.004378 seconds (498 allocations: 36.640 KiB, 99.48% compilation time)
  0.854989 seconds (488.09 k allocations: 33.029 MiB, 2.13% compilation time)

Also thank you @jheinen for indulging me for the last few hours. I’m really excited where this is heading.