GR build problems: "installation is incomplete"

General Usage
#26

It’s being actively developed, and there are moving pieces. The situation today is different that it was two weeks ago. Part of the current situation is the package is transitioning to using BinaryBuilder provided binaries.

We do have a way of version pinning it to make it work as some have described above. pkg> add GR@0.54 or using a Manifest file which let’s you grab everything at exactly the same version.

The gist of the extra information I added is that you then need to do:

julia> ENV["GRDIR"] = ""
""

(@v1.6) pkg> build GR

It’s the first thing that @jheinen tells everyone.

If you’re trying a new version of GR, such as GR v0.57.1, then they key piece of information we need is

julia> GR.gr_provider
"BinaryBuilder"

If the above value is still “GR” rather than “BinaryBuilder”, then it may be necessary to force GR to recompile itself.

For example, rather than deleting the entire .julia, you could focus on the the location of dirname(dirname(pathof(GR))).

Here, I’ve artificially created a situation with the latest GR.

julia> using GR
Your GR installation is incomplete. Rerun build step for GR package.
    Building GR → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/f74b42150042d11a1d94badf27c6125867c7e9bd/build.log`

julia> GR.gr_provider # This should be "BinaryBuilder" in normal circumstances
"GR"

julia> ENV["GRDIR"]
"/home/mkitti/.julia/packages/GR/f111O/src/../deps/gr"

julia> isfile(ENV["GRDIR"]) # This is trying to point at something that doesn't exist, not good
false

julia> ENV["GRDIR"] = "" # Clear the environmental variable

julia> GR_path = dirname(dirname(pathof(GR))) # Figure out where GR lives
"/home/mkitti/.julia/packages/GR/f111O"

(@v1.6) pkg> rm GR # Remove it from your environment
    Updating `~/.julia/environments/v1.6/Project.toml`
  [28b8d3ca] - GR v0.57.1
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`

julia> rm(GR_path; force = true, recursive = true) # Remove the GR package files to force recompilation

julia> ENV["GRDIR"] # Double check the environment is clean
""

(@v1.6) pkg> add GR
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
   Installed GR ─ v0.57.1
    Updating `~/.julia/environments/v1.6/Project.toml`
  [28b8d3ca] + GR v0.57.1
  No Changes to `~/.julia/environments/v1.6/Manifest.toml`
    Building GR → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/f74b42150042d11a1d94badf27c6125867c7e9bd/build.log`
Precompiling project...
  Progress [========================================>]  2/2
  ? Plots
1 dependency successfully precompiled in 12 seconds (119 already precompiled)
1 dependency failed but may be precompilable after restarting julia

julia> exit() # Restart Julia
$ bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using GR

julia> GR.gr_provider # Better
"BinaryBuilder"

julia> x = [1.0, 2.0, 3.0]; y = rand(Float64,3) # This works
3-element Vector{Float64}:
 0.1945528819219915
 0.2910514639410986
 0.6935992470239583

julia> plot(x,y)

If you’re not interested in knowing why you need to do that, that’s fine with me.

6 Likes
Build Julia on NixOS
#27

My hunch is that this is the result of a readonly package depot, and thus the deps/deps.jl file isn’t being created.

Did the build step error?

1.6’s Preferences.jl should help packages avoid this issue.

#28

Looking closely at what is happening with the latest GR, these constants are getting set during the first precompilation:

If depsfile did not exist when the package was compiled, but now exists, the package might not know since it has not been recompiled. It may be necessary to move more of this into GR.__init__.

1 Like
#29

Thanks @mkitti for clearifying this.

From several sides I have been asked for a BinaryBuilder integration. Unfortunately I cannot remove the “old” installation mechanisms. Besides the previous method, there are also many users who prefer a single GR installation for different languages (C, Python, Julia, Ruby) - by the way, this is how I do it in our environment(s).

Problems like those described by several people here when updating GR and/or Plots must not occur. I fully agree that the current state is not sustainable.

At the moment, however, I don’t see how I can simplify this. Maybe I am doing something wrong in my build script. Any help is appreciated.

5 Likes
#30

I’ve got a PR on the way. It’s a bit heavy handed though because it changes the library constants to constant Refs.

One trick is I used is to do touch(pathof(GR)) in order to force GR.jl to recompile after rebuilding it.

4 Likes
#31

I remember now… When I had this problem before (about 6 months ago), after searching for several days, I finally came upon this solution, and it worked brilliantly.
I had just plain forgotten this.

I wonder if this (setting the directory to “”) might be added to the error message? At the moment, it says to rebuild GR, but I think if this were added, it would save lots of headaches. Thanks for that!

4 Likes
#32

To switch to Plotly backend in a transparent way, just add the following to your .julia/config/startup.jl

ENV["PLOTS_DEFAULT_BACKEND"] = "pyplot"
1 Like
#33

Thanks Carsten for sharing this, which is very useful! We also use jupyter for teaching and most problems are related to the installation of Conda + IJulia. For some students we could narrow it down to the presence of special characters in the user name (installation failled with accent in user name · Issue #199 · JuliaPy/Conda.jl · GitHub). I guess umlauts in user names should be quite common too in your place :slight_smile:. As you advice to install anaconda manually, if IJulia fails, I am wondering if your students have been bitten by this issue too.

#34

You may wish to have a look at:

#35

No, I don’t think we’ve run into the umlaut problem often (at all?). I think that umlauts are not super common in German names and also that (French) accents and such are leading to issues more often than plain umlauts. But of course, we could have just been lucky.

The vast majority of issues that students had with installation were not related to Julia or a Julia package at all. Instead is was almost always a python dependency problem: we are using Jupyter / IJulia, matplotlib (through PyPlot.jl) for plotting, and, for some courses, SymPy.jl. Typically, the student was unaware of an old broken/incompatible python installation - which he might not have even consciously installed - which made the Julia packages think “let’s use this system python” and made things fail for various individual reasons. That’s also why I’m super curious about Symbolics.jl (to replace our modest use of SymPy.jl), Makie.jl as a super promising (hopefully soon stabilising) awesome pure-Julia plotting solution (matplotlib and 3D or animations is something else…) and, most importantly, Pluto.jl as a pure-Julia replacement for Jupyter (I would need a switch to turn off the reactivity feature though, which might never exist… I’m aware of Neptune.jl.).

Otherwise, we faced a few networking issues (pkg operations behind a firewall or similar) or students executing Julia commands that they found on the internet to “fix” a problem but instead making things worse (like mangling with the toml files without understanding what they are doing). But this the minority (compared to python problems) within the minority of cases where significant issues occurred at all.

2 Likes
#36

Typically, the student was unaware of an old broken/incompatible python installation

Yes, I had precisely this problem too with some students. I believe that if a user chooses to add anaconda to the PATH (on windows), you will run into issues with Conda.jl.

There is so much useful information in this thread and related threads, I am wondering if it would be useful to have a common wiki with troubleshooting and best practice for teaching (for different software stacks), maybe under a new JuliaEducation or JuliaTeaching github organization? Or directly adding this troubleshooting information directly to the respective Julia packages (documentation of more helpful error message)?

2 Likes
#37

I tell students who are not comfortable managing their python installation to do ENV["PYTHON"]="" before installing PyCall. It does take a while because julia has to reinstall python+packages, but that way I don’t have to deal with the 100 different python versions and package managers they have on their systems. The biggest problem is that there is no indication of what’s going on and so the students panic before the installation completes. There are firewall issues, but in that case students just make their laptop go through their phones’ 4G.

#38

We merged some PRs in the last few hours. v0.57.2 as released has an issue with touch for the reason that @mbauman brought up, but should succeed if you manually build it.

We shouldn’t need that trick anymore since we always try to include deps.jl now. On #master with Using Libdl.dlsym to dynamically load function pointers by mkitti · Pull Request #388 · jheinen/GR.jl · GitHub merged, things are really robust and no longer has the touch issue.

If you need the fix now, do

] add https://github.com/jheinen/GR.jl#master

Otherwise, I expect that @jheinen will release current master as v0.57.3.

I added a fallback from GR_jll to the GR provided tarball which seems to be working out pretty well. GR_jll is failing on Mac right now due to a problem with QtBase5_jll, but the fallback allows GR to succeed.
https://travis-ci.org/github/jheinen/GR.jl/builds/766334229

Using low-level GR I can achieve a time to first plot in 0.73 seconds on Julia 1.6.0 on Linux.

$ julia-1.6.0/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time begin
           using GR
           x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
           y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
           GR.polyline(x,y)
           GR.axes(0.2, 0.2, 0, 0, 1, 1, -0.01)
           GR.updatews()
       end
  0.738510 seconds (378.15 k allocations: 27.209 MiB, 2.06% gc time, 0.71% compilation time)

image
image1232×488 51 KB

4 Likes
#39

@mkitti : Many thanks for the active support. :clap: :100:

GR.jl v0.57.3 will be merged in the next round …

4 Likes
#40

On Windows, the GKS QtTerm takes a few seconds longer to load for some reason. If you need a quick plot on Windows, try ENV["GKSwstype"] = 41. I got it down to 0.854989 seconds there.

julia> ENV["GKSwstype"] = 41
41

julia>  @time begin
                  @time using GR
                  x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
                  y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
                  @time GR.polyline(x,y)
                  @time GR.axes(0.2, 0.2, 0, 0, 1, 1, -0.01)
                  @time GR.updatews()
              end
  0.736812 seconds (358.68 k allocations: 24.924 MiB, 0.44% compilation time)
  0.027033 seconds (4.20 k allocations: 239.321 KiB, 16.08% compilation time)
  0.012760 seconds (7.61 k allocations: 439.718 KiB, 49.23% compilation time)
  0.004378 seconds (498 allocations: 36.640 KiB, 99.48% compilation time)
  0.854989 seconds (488.09 k allocations: 33.029 MiB, 2.13% compilation time)

image
image1476×1011 61.4 KB

Also thank you @jheinen for indulging me for the last few hours. I’m really excited where this is heading.

What does GKSwstype do?
#41

Thanks for your work on this. Is it possible to include the suggestion above about the error message - that in the case of the ‘incomplete build’ issue (which gave rise to this thread) the error message would include the suggestion to set GRDIR to “”

Thanks!

#42

If what I did was successful, GR.jl should just do that automatically for you:

I can simulate the problem by doing using GR; rm(GR.depsfile) and then starting a new Julia session:

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>  @time begin
                  using GR
                  ENV["GKSWSTYPE"] = 41
                  x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]; y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
                  GR.polyline(x,y)
                  GR.axes(0.2, 0.2, 0, 0, 1, 1, -0.01)
                  GR.updatews()
              end
[ Info: Precompiling GR [28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71]
Your GR installation is incomplete. Rerunning build step for GR package.
    Building GR → `C:\Users\kittisopikulm\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\82a03d9c331d181bf33c99c9af04202cc4533d48\build.log`
Precompiling project...
  Progress [========================================>]  3/3
  ? GRUtils
  ? Plots
1 dependency successfully precompiled in 12 seconds (198 already precompiled)
2 dependencies failed but may be precompilable after restarting julia
[ Info: GR was successfully rebuilt
 16.719066 seconds (10.48 M allocations: 641.204 MiB, 1.10% gc time, 1.57% compilation time)

As you can see we encounter the “incomplete build” issue here, but GR.jl rebuilds itself automatically and even still makes the plot for you!

3 Likes
#43

If it still cannot recover, then it indeed does provide your suggestion:

3 Likes
#44

Awesome! Thank you!

1 Like
#45

To the OP, I feel your pain. FWIW, I just ran into a problem caused by the intersection of GR, Plots, and PyPlot (GR issue 392). After hours of troubleshooting, I discovered that the most recent GR version, 0.57.3, was picking up multiple Qt libraries on my system. Restricting the version to 0.55.0 has resolved it for me. Maybe not directly related to the direct issue discussed here, but it speaks to the larger problems of the Julia packaging ecosystem. Hopefully it continues to improve.

1 Like