# Github contingency planning — should Julia be backing up issues/PRs?
**URL:** https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086
**Category:** Tooling
**Created:** [July 2, 2023, 4:55pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086 "2023-07-02T16:55:51Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 2, 2023, 4:55pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/1 "2023-07-02T16:55:51Z")
Reading Cory Doctorow’s [article on the inevitable en-💩-ification of for-profit websites](https://www.wired.com/story/tiktok-platforms-cory-doctorow/) made me think about how closely the development of Julia (and many other free/open-source projects) is currently tied to [github.com](http://github.com). What is the contingency plan if Microsoft (_gasp_) turns evil?
Yes, people can move their git repos elsewhere, but that wouldn’t preserve the issues and PR discussions, which are an invaluable record of development history. And if Github were, hypothetically, to suddenly change its APIs to lock in developers and make migration harder, it might be too late to easily grab everything.
Even for a project the size of JuliaLang/julia, the entire archive of issues and PRs couldn’t possibly be more than a GB or so. Wouldn’t it be prudent to institute regular backups? What’s the best tool for this, [python-github-backup](https://github.com/josegonzalez/python-github-backup)?
---
### Author:  [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [July 3, 2023, 1:58am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/2 "2023-07-03T01:58:39Z")
> [@stevengj](#):
>
> Reading Cory Doctorow’s [article on the inevitable en-💩-ification of for-profit websites](https://www.wired.com/story/tiktok-platforms-cory-doctorow/)
💯💯 This concept has really changed my perspective since I read that. It’s nothing I didn’t know, but having the framework to slot all of the different examples into really makes it concrete.
> [@stevengj](#):
>
> Yes, people can move their git repos elsewhere, but that wouldn’t preserve the issues and PR discussions,
At least at the moment, these things _are_ preserved if migrating to GitLab. Not sure about Gitea and codeberg and the like, but it seems plausible that they would. [It’s also possible to export](https://dev.to/github/export-github-issues-commit-history-and-more-github-artifact-exporter-2ok6) basically everything to JSON. Could be a good idea to make this part the release process or something (perhaps just minor releases and not every patch)?
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 3, 2023, 3:28am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/3 "2023-07-03T03:28:40Z")
> [@stevengj](#):
>
> Yes, people can move their git repos elsewhere, but that wouldn’t preserve the issues and PR discussions, which are an invaluable record of development history
Tools like Gitea have forge-specific importers
which (usually with an API access token) allow for importing of issues, PRs, and more
I’ve recently used this to turn a bunch of my GitHub repos into GitHub mirrors of the canonical selfhosted repo. You can see an example of the final result here: [https://git.tecosaur.net/tec/emacs-everywhere](https://git.tecosaur.net/tec/emacs-everywhere).
That said, it appears this may be harder with huge repos. For instance, see [Gitea hosted Gitea · Issue #1029 · go-gitea/gitea · GitHub](https://github.com/go-gitea/gitea/issues/1029) where the last comment contains:
> The alternative export API that Github provides is sadly not working for us, it always returns that it fails to export the data. My guess is that the amount of data we have is too much.
* * *
It’s also worth noting there’s a soon-ish-to-be-merged PR for syncing issues/PRs/etc. from a _pull mirror_ [Sync issues/PRs etc from pull mirror by harryzcy · Pull Request #20311 · go-gitea/gitea · GitHub](https://github.com/go-gitea/gitea/pull/20311), which should help lead to bidirectional syncing in the future.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 3, 2023, 3:34am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/4 "2023-07-03T03:34:05Z")
> [@kevbonham](#):
>
> [It’s also possible to export](https://dev.to/github/export-github-issues-commit-history-and-more-github-artifact-exporter-2ok6) basically everything to JSON.
I know it’s possible to export everything _now_. But it seems unwise to rely on that being possible indefinitely. A process for regular complete backups seems like something it would be good to make routine well _before_ there seems to be any concrete reason to worry.
> [@tecosaur](#):
>
> Tools like Gitea have forge-specific importers …
For the purposes here, I don’t think relying on yet another commercial service (gitlab, gitea, …) is a good idea. You want a regular backup, using free/open-source tools, to open formats like JSON etc.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 3, 2023, 3:42am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/5 "2023-07-03T03:42:15Z")
> I don’t think relying on yet another commercial service (gitlab, gitea, …)
Indeed, that’s why I suggested Gitea, because it’s _not_ a commercial service. It’s a self-hostable FOSS forge, and once the ForgeFed effort comes to fruition you’ll be able to get an F3 (Friendly Forge Format) form of the repos, which is a relatively new open standard for forge data (see [https://forum.forgefriends.org/t/about-the-friendly-forge-format-f3/681](https://forum.forgefriends.org/t/about-the-friendly-forge-format-f3/681) / [https://f3.forgefriends.org](https://f3.forgefriends.org)), and the sync PR I linked to is effectively “a regular backup, using free/open-source tools, to open formats like JSON etc.”.
Is there a bit of “soon™” going on here? Yes, but it’s the closest thing to a good solution that I’m currently aware of, and the ForgeFed/F3 effort is moving along.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 3, 2023, 1:53pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/6 "2023-07-03T13:53:04Z")
> [@tecosaur](#):
>
> Indeed, that’s why I suggested Gitea, because it’s _not_ a commercial service. It’s a self-hostable FOSS forge
Thanks for the clarification — I was confused by the `.com`. They offer commercial support for a FOSS platform, which is fine.
But I don’t think we have to move immediately to self-hosting. Just have a regular backup of data so that we _could_ switch to self-hosting at any time without a huge loss of information.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 3, 2023, 2:46pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/7 "2023-07-03T14:46:40Z")
> But I don’t think we have to move immediately to self-hosting. Just have a regular backup of data so that we could switch to self-hosting at any time without a huge loss of information.
Yea, it would be a bit premature in some respects. To be safe in the knowledge you actually have the backup in a format that _could_ be used for self-hosting in the future though, I think you’d either want to have:
- A live pull sync with everything you want backed up — in a way it’s the ultimate way of being sure you _can_ self-host at any time … already be quietly doing so
- A F3 archive, but the standard is still setting down so that’s probably a bit hard ATM (IETF draft is in the works: [Submission status of draft-f3-format-00](https://datatracker.ietf.org/submit/status/133552/))
- Some other format that can latter be massaged to F3 or otherwise imported into some self-hosted forge
> Thanks for the clarification — I was confused by the `.com`. They offer commercial support for a FOSS platform, which is fine.
NP. Last year a company for commercial support was formed, and there was a bit of community uproar because all of the project branding IP (name, logo, etc.) was transferred to the commercial entity instead of a non-profit which the commercial entity was affiliated with. As a result there’s an active fork now called Forgejo which the largest Gitea site ([codeberg.org](http://codeberg.org)) has switched to and the ForgeFed team is as I understand it working on that fork ATM.
> **[Forgejo – Beyond coding. We forge.](https://forgejo.org/)**
>
> Forgejo is a self-hosted lightweight software forge. Easy to install and low maintenance, it just does the job.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 3, 2023, 2:57pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/8 "2023-07-03T14:57:12Z")
> [@tecosaur](#):
>
> That said, it appears this may be harder with huge repos.
`python-github-backup` (linked above) has a GitHub API throttling feature that may help with this.
> [@tecosaur](#):
>
> Some other format that can latter be massaged to F3
In practice, I think the backup tool’s robustness and completeness is more important than the format. My feeling is that any reasonably structured open format can be massaged into any other open format if needed.
---
### Author:  [@gdalle](https://discourse.julialang.org/u/gdalle)
#### Post date: [July 4, 2023, 6:02am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/9 "2023-07-04T06:02:05Z")
Just popping by to say that it would sure be nice to back up the packages from the general registry in addition to Julia itself, but idk how feasible / legal it is licensewise?
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 4, 2023, 8:15am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/10 "2023-07-04T08:15:15Z")
Most of them probably already are in terms of Git history at least, see [https://www.softwareheritage.org/](https://www.softwareheritage.org/).
---
### Author:  [@gdalle](https://discourse.julialang.org/u/gdalle)
#### Post date: [July 4, 2023, 8:17am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/11 "2023-07-04T08:17:42Z")
I quickly checked a few packages I registered this year or the last, and they’re not on there, so presumably the backup is partial
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [July 4, 2023, 12:06pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/12 "2023-07-04T12:06:23Z")
I share your concerns, but I don’t think that a backup is the best solution in this case.
Backups without verified restore and integrity compared to the original can easily turn out to be useless. For a filesystem backup, this is easy to do, but for Github metadata, this is more or less equivalent to doing the full migration to a self-hosted solution.
So, I would propose that effort is spent on finding a self-hosted solution instead, and migrating while we can. For a typical project, CI would be a pain point (Github provides it for free), but my understanding is that Julia already moved that out of Github.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 4, 2023, 12:46pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/13 "2023-07-04T12:46:57Z")
> [@Tamas\_Papp](#):
>
> So, I would propose that effort is spent on finding a self-hosted solution instead
Have you heard the phrase “the perfect is the enemy of the good”?
Moving to a self-hosted solution requires a _lot_ more resources than simply running a few-line `python-github-backup` script (or similar) occasionally and occupying a bit of disk space. Even if the backup is not perfect — even if it loses 5% of the issue/PR information — it would still be a _huge_ improvement over having nothing. (The only backup that needs to be _perfect_ is the code, but `git` already gives us that.) And insisting on a self-hosted solution or nothing means that most projects will have nothing.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [July 4, 2023, 1:00pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/14 "2023-07-04T13:00:32Z")
> [@stevengj](#):
>
> Even if the backup is not perfect
While you make a valid point, the concern is not about the backed up data being perfect, but about it being being garbage. That can easily happen down the line with an automated solution.
Nevertheless, you are right, even a backup from a few months ago or similar would be valuable.
---
### Author:  [@tecosaur](https://discourse.julialang.org/u/tecosaur)
#### Post date: [July 4, 2023, 2:49pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/15 "2023-07-04T14:49:21Z")
> [@stevengj](#):
>
> Moving to a self-hosted solution requires a _lot_ more resources than simply running a few-line `python-github-backup` script (or similar) occasionally and occupying a bit of disk space.
FWIW I don’t think a `gitea` self-hosted mirror will take nearly as much as you seem to think. I’d think a €5/month hetzner VM would be sufficient, and low-maintenance.
---
### Author:  [@Jake](https://discourse.julialang.org/u/Jake)
#### Post date: [July 4, 2023, 7:04pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/16 "2023-07-04T19:04:43Z")
And then we trust the ransomware specialists won’t encrypt everything! I have seen how important backups are.
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [July 5, 2023, 3:20am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/17 "2023-07-05T03:20:37Z")
> [@gdalle](#):
>
> Just popping by to say that it would sure be nice to back up the packages from the general registry in addition to Julia itself, but idk how feasible / legal it is licensewise?
This is already done, I believe. A JuliaHub package server contains a copy of all packages in the Julia General registry. Search this forum for a response to the “leftpad” incident.
---
### Author:  [@GregVernon](https://discourse.julialang.org/u/GregVernon)
#### Post date: [July 5, 2023, 8:19am UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/18 "2023-07-05T08:19:46Z")
My completely naive and probably stupid thought: could such a backup be “hosted” via a distributed (and possibly partitioned) torrent? Community members provide the storage _au gratis_, multiple copies of the backups help ensure redundancy, etc.
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [July 5, 2023, 1:09pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/19 "2023-07-05T13:09:35Z")
I second Gitea. I have been running it for a few months on Hetzner and it has been very reliable. Noteworthy point is that the source is also hosted on GitHub.
---
### Author:  [@fgerick](https://discourse.julialang.org/u/fgerick)
#### Post date: [July 5, 2023, 1:34pm UTC](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086/20 "2023-07-05T13:34:47Z")
Quite ironic in some way, but I believe here lies also the problem with self-hosting. The barrier to contribution is increased and general visibility decreased. Maybe a centralized platform based on donations, such as Wikipedia or the Signal messenger could be an option. ~~Not sure if that exists already and how the financing works out~~. I guess I should look at what people have posted earlier (codeberg exists).
[Next page](https://discourse.julialang.org/t/github-contingency-planning-should-julia-be-backing-up-issues-prs/101086.md?page=2)