[ANN] Introducing Upgradathon Fridays

Hi all,

we’re going to try an experiment and designate every Friday between now and JuliaCon an Upgradathon Friday, i.e. one dedicated day each week to try to get as many people together in one place and upgrade packages to 0.7. We’ll be gathering in the #upgradathon channel on slack. We hope that by getting people in one place on one specified day, we’ll be able to make focused progress on upgrading the ecosystem to 0.7. We should have enough people around to answer questions and help out with any obstacles in upgrading. Even if you’re not a package author, please feel free to join in the fun. There’s 1736 packages still to be upgraded to 0.7, so I’m convinced we’ll be able to find you one :slight_smile: .

We’ll run the first of these tomorrow, Friday June 29th, and then every Friday thereafter, unless it turns out that another day of the week would work better for most people (or we run out of packages to upgrade). I’ll be hanging out in the #upgradathon slack channel starting around noon EST, though please do get started without me!

Keno

P.S.: There’s nothing wrong with upgrading packages on other days of the week also of course :wink:

37 Likes

That describes this last week for me pretty well :smile:

And here is how you do it!

First off, the process will be much simpler if you choose to drop support for Julia 0.6 on the master branch. Unless you have strong reason to create another release compatible with Julia 0.6 I would strongly recommend doing this, several packages, like StaticArrays and DataStructures, have already done so. If you need to release a 0.6 compatible bugfix release of your package you can do this from a branch later.

If your package are dependent on other packages, I suggest that you start with upgrading those first.

Before you start, check open pull request to the package to see if someone else has already made progress!

1. Obtain a copy of Julia v0.7

You can either do this by compiling from source (see the Julia README) but unless you want to wait, or are note comfortable with compiling yourself I suggest that you just download the latest nightly build from https://julialang.org/downloads/nightlies.html

2. Obtain the package source code

The first step is to put the source code that you want to upgrade in some local directory. I propose you take the chance to try out the new awesome package manager for this. In the Pkg REPL we can use the develop command to download the source code. Enter the REPL (with ] from the Julia REPL) and run

pkg> develop Foo

where Foo is the package we want to upgrade. By default, Pkg will now clone the repo, and put it in ~/.julia/dev/Foo. This is where we will make all our changes. I recommend that you make a branch before doing any edits to the source code (for example git checkout -b julia07fixes).

3. Run FemtoCleaner

The first step is to run FemtoCleaner which can do some, but not all upgrades automatically. You can use FemtoCleaner from GitHub (see FemtoCleaner usage HOWTO) but I will describe how to run it locally, since it is likely that you need to make other adjustments as well.

  1. First we need to update the Julia requirements for the package. This is what FemtoCleaner looks at when deciding which code rewrites that applies. Do this by changing julia 0.6 (or whatever is in there) to julia 0.7-alpha, then commit that change.

  2. Next we need to install FemtoCleaner. It is easier to run it from Julia 0.6. To install, open a Julia v0.6 prompt and run

    Pkg.clone("https://github.com/Keno/AbstractTrees.jl")
    Pkg.checkout("AbstractTrees", "kf/for06")
    Pkg.clone("https://github.com/JuliaComputing/Deprecations.jl")
    Pkg.clone("https://github.com/JuliaComputing/FemtoCleaner.jl")
    
  3. Once the installation is done it is time to clean! We can run it with the following commands

    import FemtoCleaner
    FemtoCleaner.cleanrepo("/home/fredrik/.julia/dev/Foo"; show_diff=true, delete_local=false)
    

    FemtoCleaner will now clone the repo to a temporary directory (the directory is printed to the screen), and upgrades whats in its power do to. When it is done the diff will be printed to the terminal. You can skip this, but it is good to have a look at what replacements FemtoCleaner did, in case you need to do some manual adjustments. After this you should go to the temporary directory, make a branch (git checkout -b femtocleaner), commit the changes (git commit -m "femtocleaning"), and push them back to your local repo (git push -u origin femtocleaner). In ~.julia/dev/Foo we can now pull in FemtoCleaners changes; either git cherry-pick the commit, or simply reset our branch to the femtocleaner branch that we just pushed (git reset --hard femtocleaner).

4. Run package tests

Now we can begin to test our package with Julia v0.7. We can test the package from the Pkg REPL:

pkg> test Foo

which will run the tests for Foo. It is likely that there will be deprecation warnings, and/or test errors. Look at the output and try to figure out what’s wrong. Fix them in the source code, and run the tests again. This is an iterative process, and hopefully the warnings are less the next time you pkg> test the package.

5. Push the changes and make a PR

When you are happy with the result, simply push the changes to the remote repo (git push -u origin julia07fixes where you might have to replace origin with your own fork unless you have commit priviliges to the original repo.) When the changes are pushed, create a pull request, and wait for CI to test your changes (don’t forget to remove Julia v0.6 from the CI configurations scripts!). If tests pass; Congratulations!, if not; try looking at the build logs to see what went wrong and fix up the last changes.

5. Some other tips

  • Line numbers from warnings and errors are not always 100% accurate, to improve the accuracy you can turn off inlining by starting julia with julia --inline=no.
  • Sometimes the deprecation warnings does not have any clue to where the error is, which makes it very difficult to find them. To get a full stacktrace to the warning you can start julia with julia --depwarn=error which will turn the deprecation warnings to errors. This prints a stacktrace and makes it much simpler to find.
  • Need help? Check out the Slack channel Keno mentioned in the top post, there are lots of willing people there to help you get rid of Julia 0.6!

Happy Upgradathon!

41 Likes

Thank you for this!

I apologize if this has already been explained but rather be clear on this:

  1. It seems like you are suggesting that the whole Julia ecosystem should drop v0.6 and move on to v0.7 (obviously in preparation to v1).
  2. I thought, following Stefan Karpinski’s explanation here, that we want to avoid such uniform migrations because they might not work so well (see Stefan’s explanation on the Python 3 disaster).

Have I misunderstood you?

Yes! To be clear; just because the master branch moves on it does not mean that the package will be useless on julia 0.6, it just won’t be the latest release, since those only support julia v0.7 and above.

:man_shrugging: I think it is possible, we already have 0.7-only compatible packages, and if we start at the bottom it should be fine, but lets see!

3 Likes

I think that the package manager is now sophisticated enough to handle this. At any moment, a package exists in multiple versions (basically all commits, but in practice the tagged releases + branches). Making a new release just introduces another version, the old releases do not disappear.

It is possible though that package authors have to pay more attention to versions though. The Pkg(3) compatibility specifiers, closely related to semantic versioning, should facilitate this.

But just to be extra clear, this specific migration, v0.6->v0.7, is not gonna be part of this mechanism you are describing. So iff everyone does this all will be fine. Then, after we have Pkg(3) in place we can do what ever we want.

No, I think that people who wish to remain on v0.6 for a while should be fine, as Pkg(2) can handle this, too; upper bounds on some package versions may be required though, but this is a small change.

1 Like

Thanks for the nice writeup. What is the recommendation about REQUIRE and Project.toml? If a package wants to support just v0.7 from now on, should it remove REQUIRE and add a project file?

The REQUIRE file is still used when releaseing to METADATA, and hence also when converting to Uncurated. I think it is possible to have both Project.toml and REQUIRE, but personally I would wait until something like WIP: add package converter (creating Project.toml from REQUIRE) by KristofferC · Pull Request #392 · JuliaLang/Pkg.jl · GitHub have landed with that change. Perhaps Kristoffer can shed some more light on this.

1 Like

How do I install an unofficial package? Is there anything like Pkg.clone?

1 Like

] dev https://github.com/you/repo.jl

should work

3 Likes

if it is just installation, use add.

But if you want to modify its source after installation, use dev.

julia> Pkg.checkout("AbstractTrees", "kf/for06")
INFO: Checking out AbstractTrees kf/for06...
ERROR: GitError(Code:ERROR, Class:Merge, There is no tracking information for the current branch.)
Stacktrace:
 [1] (::Base.LibGit2.##117#125{Base.LibGit2.GitRepo})(::Base.LibGit2.GitReference) at ./libgit2/libgit2.jl:709
 [2] with(::Base.LibGit2.##117#125{Base.LibGit2.GitRepo}, ::Base.LibGit2.GitReference) at ./libgit2/types.jl:608
 [3] #merge!#109(::String, ::String, ::Bool, ::Base.LibGit2.MergeOptions, ::Base.LibGit2.CheckoutOptions, ::Function, ::Base.LibGit2.GitRepo) at ./libgit2/libgit2.jl:706
 [4] (::Base.#kw##merge!)(::Array{Any,1}, ::Base.#merge!, ::Base.LibGit2.GitRepo) at ./<missing>:0
 [5] (::Base.Pkg.Entry.##16#18{String,String,Bool,Bool})(::Base.LibGit2.GitRepo) at ./pkg/entry.jl:230
 [6] transact(::Base.Pkg.Entry.##16#18{String,String,Bool,Bool}, ::Base.LibGit2.GitRepo) at ./libgit2/libgit2.jl:882
 [7] with(::Base.Pkg.Entry.##15#17{String,String,Bool,Bool}, ::Base.LibGit2.GitRepo) at ./libgit2/types.jl:608
 [8] checkout(::String, ::String, ::Bool, ::Bool) at ./pkg/entry.jl:226
 [9] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#checkout,Tuple{String,String,Bool,Bool}})() at ./pkg/dir.jl:36
 [10] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#checkout,Tuple{String,String,Bool,Bool}}, ::String) at ./file.jl:70
 [11] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N} where N) at ./pkg/dir.jl:36
 [12] #checkout#1(::Bool, ::Bool, ::Function, ::String, ::String) at ./pkg/pkg.jl:188
 [13] checkout(::String, ::String) at ./pkg/pkg.jl:188

Maybe just go into the directory (~/.julia/v0.6/AbstractTrees) and check it out manually git checkout kf/for06.

This is odd…

$ cd ~/.julia/v0.6/AbstractTrees/
$ git checkout kf/for06
Already on 'kf/for06'
Your branch is based on 'origin/kf/for06', but the upstream is gone.
  (use "git branch --unset-upstream" to fixup)

Ahh well, we’ll see if it works!

EDIT: I had to manually add your repo as the origin, then pull the latest version (it was 4 commits behind for some reason). But it works!

A minor questions: how to go about using something from StdLib without Project.toml? Specifically, if I am

using LinearAlgebra: UpperTriangular

then I get an error message like

[ Info: Precompiling module TransformVariables
ERROR: LoadError: ArgumentError: Package TransformVariables does not have LinearAlgebra in its dependencies:
 - If you have TransformVariables checked out for development and have
   added LinearAlgebra as a dependency but haven't updated your primary
   environment's manifest file, try `Pkg.resolve()`.
 - Otherwise you may need to report an issue with TransformVariables.

Have you tried:

Yes, of course. In the end I solved the problem with free and develop again.