Recommended way to develop/fix with packages

What’s the easiest way to provide patches to packages. I guess I should for the project on github but how do I make then sure that Julia uses my branch instead of the one I installed?

1 Like

This is my workflow (for patching a hypothetical Example package):

  1. Get a local copy of the package, this can be done with pkg> develop Example. This will clone the package to ~.julia/dev/Example by default. It will also make sure that the local copy is the one that is “installed” in the sense that using Example will use the files in that local folder.
  2. Edit the package in ~.julia/dev/Example and make sure your changes work as you expect.
  3. Fork the package on GitHub and add your fork as a remote. Then use git to commit your changes, and push to your fork.
  4. Open a pull request.
21 Likes

Probably typo, should be “remote”.

1 Like

Why does this have to be so complicated? :thinking:

Like “add your fork as a remote.” Not a one click action, not even a one command line action.

1 Like

Can you explain what you find so complicated?

Forking and requesting a PR are done on GitHub’s side. The only phase Julia is involved in is locating the git project, which is a single command: pkg> develop https://github.com/some/package.

1 Like

The order of operations for easy pull requests is to:

  1. Fork the repo you’re trying to fix (on github)
  2. Clone the fork to your computer
  3. Make changes to the local copy
  4. Push changes to your forked repo
  5. Open pull request via the github app

If you happen to start developing a package you already have locally (i.e. if you find a bug and want to fix it), then you have to follow this mode of operations.

  1. Checkout new branch (because you’ll be at HEAD)
  2. Add remote branch using:
git remote add upstream <my_fork_url>
  1. Make changes to the local copy
  2. Push changes to the upstream remote (i.e. your fork)
  3. Open pull request via the github app
2 Likes

Because you don’t want anyone to be able to automatically add their code to your repo without review. You want them to work on their own remote and give a PR, not push to your repo. I think that’s a very very very valid concern to have.

5 Likes

Thanks for the command line for adding the remote branch. :slight_smile:

But Julia already has git integration. And I guess that github has a public api.
So wouldn’t it possible to ad Pkg-commands for most of these steps?

3 Likes

In Julia 0.6 with PkgDev it was just one command

PkgDev.publish()

it automatically made a fork, within that a branch with the fix and opened the web browser with the prepared pull request. Once PkgDev is ready for 1.0 we hopefully have that workflow again.

4 Likes

Sounds promising! :hugs:

This is a one-line command,

hub fork

Using the “hub” utility for working with GitHub repos.

1 Like

To expand a little on previous answers, these are the steps:

  1. In the Julia REPL, ‘Develop’ the package. This will clone the package to ~/.julia/dev/Example
pkg> develop Example
  1. Fork the package on GitHub
  2. On the command line, add your fork as a remote:
cd ~/.julia/dev/Example
git remote add upstream git@github.com:<your github username>/Example.jl.git
  1. Create a new branch, giving it a name reflecting the change
git checkout -b new-feature
  1. Make your changes
  2. Use git commit to commit your changes. Several commits can be made.
  3. On the command line, push the changes to your fork:
git push upstream new-feature
  1. Open a pull request on Github
10 Likes

It looks like the GitHub CLI provides a fork operation.

Here’s my experiment:

mkdir tmp.
cd tmp
tmp> 
julia> ]
pkg> activate .
(tmp) pkg> add Revise Example
(tmp) pkg> develop Example
   Resolving package versions...
    Updating `~/development/Julia/tmp/Project.toml`
  [7876af07] ~ Example v0.5.3 ⇒ v0.5.4 `~/.julia/dev/Example`
    Updating `~/development/Julia/tmp/Manifest.toml`
  [7876af07] ~ Example v0.5.3 ⇒ v0.5.4 `~/.julia/dev/Example`

in 'nother terminal:

cd  ~/.julia/dev/Example
[0] % git remote -v
origin	https://github.com/JuliaLang/Example.jl.git (fetch)
origin	https://github.com/JuliaLang/Example.jl.git (push)

# you'll have to set up the GitHub CLI authentication

› : ~/.julia/dev/Example
[0] % gh repo fork 
✓ Created fork <whatsisname>/Example.jl
? Would you like to add a remote for the fork? Yes
✓ Added remote origin

% git remote -vv
origin	https://github.com/<whatsisname>/Example.jl.git (fetch)
origin	https://github.com/<whatsisname>/Example.jl.git (push)
upstream	https://github.com/JuliaLang/Example.jl.git (fetch)
upstream	https://github.com/JuliaLang/Example.jl.git (push)

Then you’d make a branch for your changes - and go through the normal process.

I think that has a reasonably easy workflow with a small set-up cost.

2 Likes

I found an interesting weirdness in GitHub.
I was working on building a lab for playing with forking repositories and merging pull-requests - and found that a user cannot fork their own repository. The GitHub WebUI explicitly disallows it. The GitHub cli gives a misleading error message.

It can be done by making a local clone C of the eventual upstream repository - U. Also create a new repository on github - A.

In the local directory of C:

git remote rm origin
git remote add `A`
git remote add upstream `U`
git checkout -b <feature-branch-name>

This creates something that walks and quacks like a fork. I’m still trying to get who is doing what to whom in push, checkout, merge and fetch with the players in the fork.

I’m not sure yet whether the semantics of pull-requests work. I’m having to dive into what a pull-request really is - I think we call this at the plumbing level. I’m going to need more time to dissect.

There could be some identity crisis within forking repos under the same user. Something seems to stink when doing pull-requests. I’m comparing data-models with fork of Example.jl

Hi,
I’m working with bitbucket repository where pull request should probably done using feature branches which are then merged to develop branch when accepted. How to I work with such feature branch? Do I add the feature branch (if it is possible) or do I add the “main” branch in develop and then checkout my feature branch at ~/.julia/dev/? How is the update then done after feature is accepted?
-p

You can either add specific branches or the main branch and then checkout locally.
See e.g. 12. API Reference · Pkg.jl for how add/develop can be called
Pkg.develop(url="https://github.com/JuliaLang/Example.jl", rev="master")

You can also just clone the git repo to anywhere you want, and checkout any branch you want.
Starting julia with julia --project in the folder will run julia in that packages environment and allow you to test it.

Updating the main branch based on the feature branch would just follow the standard bitbucket procedure I guess, not sure if that was the question?

1 Like

Yes, standard bitbucket after changes done and tested, thanks! Only problem was that I got error from develop:

julia> Pkg.develop(url="ssh://git@bitbucket.<domain>.<cc>:<port>/<repo>/<package>.jl.git", rev="feature/<branch>")
ERROR: rev argument not supported by `develop`; consider using `add` instead

Julia Version 1.8.2 (2022-09-29). I think https access will not work, I could try but I prefer ssh.
-p

EDITED:
Pkg.add(…, rev=“feature/”) and (@1.8) pkg> develop works but fetches the “main”.

Oh, my bad. I had assumed that develop and add supported the same arguments.

Does ssh work without the rev argument?

If you want to change things locally you don’t want add since it snapshots a specific version and will not update to changes. So the solution is probably to develop the main branch and then just go into the .julia/dev/YourPackage and checkout the feature branch.

Otherwise just clone the package normally to wherever you want to have it, checkout the feature branch and then develop there. Either activate the packages environment for testing, or add it to some other environment by doing develop on the path to the package.

1 Like

Yes it works that way. I’ll do “manual” checkout, seems to be viable option in this case, and I can work this way too :+1:.