Reverting back to avoid precompile errors?

Interesting. Maybe @quinnj knows more about why that particular combination doesn’t work?

There is no version 1.2.1 (I suspect that was a typo for 1.1.2), but @mike_k seems to have it working with an earlier version of Julia than I’m using. I guess I’ll just have to wait.

1 Like

sorry for my impreciseness,… yes, there is no version 1.2.1 (this is what I have meant)

1 Like

Thanks so much for your efforts, however neither the combination
Parsers v1.1.2 + CSV v0.8.0 nor Parsers v1.1.2 + CSV v0.8.5
work for me. So I am also in waiting mode.

In the mean time:
Is some kind of time-capsule functionality (as hinted at in the original post) feasible? Or can we (and by that I mean someone much more competent than me) come up with other ideas for how to mitigate situations like this in the future?

Also, for future robustness: Does the whole package need to fail to load, if one part of it (or one dependency) fails to precompile?

That functionality already exists: If you keep your old Manifest.toml around and only instantiate it, pinning all versions to their original ones and don’t update, old code will still work as it originally did (provided you’re using the same julia version). Manifest.toml has as its sole purpose to record package versions.

A dependency is a dependency for a reason: The package requires functionality provided by it. Since the smallest unit of encapsulation is a julia package, there is no smaller part that could be reasonably swapped out for something else - it’s a whole version or no version. There’s no mixing between versions of the same package.

If all dependencies are properly versioned, package versions should only be able installable with working versions. If some sub-dependency fails to precompile, something has gone wrong during that version tagging process.

I don’t understand though why neither combination works for you. Are you certain those two are the versions actually installed & pinned? Do you have any other packages in those environments? Can you try two empty environments with both combinations of packages?

1 Like

For my earlier post,

I started with a clean environment, pinning Parsers to version 1.1.2. I separately tried both the current CSV (0.8.5) which has Parsers v2 its Project.toml and an older version (0.8.0) which has Parsers v1. The first failed immediately, saying there was no compatible version. The second failed on using CSVwith the error I reported above.

And what about explicitly asking for CSV@v0.8.5 while having Parsers pinned to 1.1.2?

(@martin.demare please also try all these versions in a clean environment!)

Hi Sukera,
Unfortunately it doesn’t work for me.

PS. My project file looks as follows:

name = “mardm_project”
uuid = “f215c453-d48a-4ef3-91ee-74b828a867ac”
authors = [“martin”]
version = “0.1.0”

[deps]
CSV = “336ed68f-0bac-5ca0-87d4-7b16caf5d00b”
Parsers = “69de0a69-1ddd-5017-9359-2bf0b02dc9f0”

And the manifest contains:

[[CSV]]
deps = [“Dates”, “Mmap”, “Parsers”, “PooledArrays”, “SentinelArrays”, “Tables”, “Unicode”]
git-tree-sha1 = “b83aa3f513be680454437a0eee21001607e5d983”
pinned = true
uuid = “336ed68f-0bac-5ca0-87d4-7b16caf5d00b”
version = “0.8.5”

[[Parsers]]
deps = [“Dates”]
git-tree-sha1 = “bfd7d8c7fd87f04543810d9cbd3995972236ba1b”
pinned = true
uuid = “69de0a69-1ddd-5017-9359-2bf0b02dc9f0”
version = “1.1.2”

fyi: just to exclude the following: I gave it a try with Julia-1.7.0-beta4 and it still works on my machine (Ubuntu 18.04). Maybe the problem is related to Windows (?). Sorry, I have no more ideas.

Actually it seems to work outside Atom (terribly sorry that I didn’t check that until now!). I will repost when I have updated/downgraded the atom related packages to make it work also inside atom (possibly it is just a simple update that is required). Regardless it means that I can work again, so: good times! :slight_smile:

1 Like

Same for me. Starting in an empty directory, activating that directory, then adding Parsers and pinning to 1.1.2, any of
add CSV, add CSV@0.8.0, add CSV@0.8.5 does the initial precompilation, but fails with

...expected UnionAll, got Type{Parsers.Options}

when trying using CSV.
I thought I didn’t understand the dependency chain. Now I’m sure of it :frowning_face:

EDIT: I should have said that this is all in the REPL, using Windows Terminal.

I can’t repro this on Windows 10 (in Hyper, but I very much doubt that the terminal plays any role here):

(@v1.7) pkg> activate --temp
  Activating new project at `C:\Users\ngudat\AppData\Local\Temp\jl_Lapatm`

(jl_Lapatm) pkg> add Parsers@1.1.2 CSV
    Updating registry at `C:\Users\ngudat\.julia\registries\General.toml`
   Resolving package versions...
   Installed Parsers ────── v1.1.2
   Installed PooledArrays ─ v1.3.0
    Updating `C:\Users\ngudat\AppData\Local\Temp\jl_Lapatm\Project.toml` 
  [336ed68f] + CSV v0.8.5
  [69de0a69] + Parsers v1.1.2
    Updating `C:\Users\ngudat\AppData\Local\Temp\jl_Lapatm\Manifest.toml`
  [336ed68f] + CSV v0.8.5
  [9a962f9c] + DataAPI v1.7.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [69de0a69] + Parsers v1.1.2
  [2dfb63ee] + PooledArrays v1.3.0
  [91c51154] + SentinelArrays v1.3.7
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.5.0
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [8dfed614] + Test
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [4536629a] + OpenBLAS_jll
  [8e850b90] + libblastrampoline_jll       
Precompiling project...
  4 dependencies successfully precompiled in 11 seconds (8 already precompiled)

julia> using CSV

julia> CSV.read
read (generic function with 2 methods)

This made me wonder what might be different between your setup and mine. That suggested I try running Julia without my usual startup file. When I did that, I got exactly the same results that you did.

Then I did a series of runs, repeating what I had just done, but adding one of the packages I usually load in my startup before activate/add/using. Everything ran smoothly until I added JuliaFormatter first. Then I had the compilation error.

It might be that other packages that require some sort of parsing produce similar results (Revise is one I thought could be a problem, but it isn’t). If nothing else, I nowe understand that an environment is not necessarily as ‘clean’ as I think it is.

To be clear, the problem here is as follows:

  1. Julia is started and some packages are imported as part of an editing environment (e.g. IJulia, Atom etc.) or from the startup file. These imports happen in some (Julia) environment (possibly the default) with its set of package versions for direct and indirect dependencies.
  2. A different environment is activated. This might have different package versions in its manifest than the first environment had because the compat requirements makes it necessary to resolve to a different set of versions.
  3. Some package is imported in the new environment. In case this package or one of its dependencies was already loaded in step 1, that will be reused since Julia can only have one version of a given package loaded in the same process. However, if that was of a different version than we wanted to load this time, we won’t have the code we intended and if there were breaking changes between the versions things might, well, break.

Actually Revise has very few non-stdlib indirect dependencies but if Requires or OrderedCollections ever get breaking updates, things might get ugly.

2 Likes

There’s the problem then:

(@v1.7) pkg> activate --temp                
  Activating new project at `/tmp/jl_R7mGad`

(jl_R7mGad) pkg> add JuliaFormatter                                   
    Updating registry at `~/.julia/registries/General`                
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...                                      
    Updating `/tmp/jl_R7mGad/Project.toml`                            
  [98e50ef6] + JuliaFormatter v0.15.10                                
    Updating `/tmp/jl_R7mGad/Manifest.toml`                           
...

julia> using JuliaFormatter                                                                       
[ Info: Precompiling JuliaFormatter [98e50ef6-434e-11e9-1051-2b60c6c9e899]                        
                                                                                                  
(jl_R7mGad) pkg> add CSV                                                                          
   Resolving package versions...                                                                  
    Updating `/tmp/jl_R7mGad/Project.toml`                                                        
  [336ed68f] + CSV v0.8.5                                                                         
    Updating `/tmp/jl_R7mGad/Manifest.toml`                                                       
  [336ed68f] + CSV v0.8.5                                                                         
  [9a962f9c] + DataAPI v1.7.0                                                                     
  [e2d170a0] + DataValueInterfaces v1.0.0                                                         
  [82899510] + IteratorInterfaceExtensions v1.0.0                                                 
  [69de0a69] ↓ Parsers v2.0.3 ⇒ v1.1.2    # note `Parsers` being downgraded a major version!
  [2dfb63ee] + PooledArrays v1.3.0                                                                
  [91c51154] + SentinelArrays v1.3.7                                                              
  [3783bdb8] + TableTraits v1.0.1                                                                 
  [bd369af6] + Tables v1.5.0                                                                      
  [9fa8497b] + Future                                                                             
Precompiling project...                                                                           
  1 dependency successfully precompiled in 8 seconds (24 already precompiled)                     
                                                                                                  
(jl_R7mGad) pkg> st                                                                               
      Status `/tmp/jl_R7mGad/Project.toml`                                                        
  [336ed68f] CSV v0.8.5                                                                           
  [98e50ef6] JuliaFormatter v0.15.10                                                              
                                                                                                  
julia> using CSV                                                                                  
[ Info: Precompiling CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]                                   
ERROR: LoadError: TypeError: in Type{...} expression, expected UnionAll, got Type{Parsers.Options}

The using JuliaFormatter step at the start is crucial. Since JuliaFormatter depends on CommonMark, which depends on JSON which claims to be and truly is compatible with Parsers versions 1 and 2, I’m not sure where to lay blame here. Truthfully, JuliaFormatters and Parsers should have to be recompiled once Parsers was downgraded or CSV should see a different Parsers to JuliaFormatter (which is stated non-policy, as far as I remember).

Is there documentation/precedent for how this is handled?

There’s a related issue at least: Warn when a package loads a dependencies that already got loaded and that dependency has a different version than in the current manifest. · Issue #32906 · JuliaLang/julia · GitHub

3 Likes

Nice, I’ll crosslink this thread to github so we have an example “out in the wild” - thanks for finding that issue!

1 Like

Dear All, I can’t understand any of this… I just can’t load CSV. I don’t need/want to know why it is failing, but I will not be able to use Julia until this is fixed and I can run

]add CSV
using CSV

Does anyone have any idea when that might be?

Thanks

Impossible to say without knowing more details but you can try it from the Julia REPL after starting Julia with julia --startup-file=no.

1 Like

The solution suggested by @GunnarFarneback is more general, however a very Atom-specific work-around is to add CSV to the base environment and then restarting Atom. The dependencies of CSV are then incorporated early enough in the initiation process not to cause problems. After that it works fine to activate other environments and adding CSV to them as needed.

1 Like