Q: automatically updating/detecting missed JLL packages updates

Tooling
, , ,
1

context:

  • Some JLL packages are not updated to the latest version
  • so important (security) patches not used.
  • hard to find these packages in the local environments.

example:

  • LibPQ_jll ( PostgreSQL client) - based on v14.1 : LibPQ_jll v14.1.0+1

Question:

  • Is it possible to automate the detection of these missed package updates? ( and update )
    • in the local environment
    • in the package libraries ( globally )
    • any tools? packages ( or built in solution ; bots ) ?
  • What is the best practice?

Side note:

Related problem:

  • using Julia in the Enterprise world - I need to check the vulnerability risks of the Julia packages.
3 Likes
2

There is a problem with automatic updates: packages can introduce API/ABI-breaking changes in a completely unpredictable and often non-trivial to check way. Linux distributions don’t have this problem because most of them have a single version of a package at any given time and they rebuild the entire universe (well, the affected subset) when a package is updated. But we have to support multiple versions at the same time, and retroactively fix compat bounds if necessary, which, again, can’t really be automated. The risk is that automatically updating packages will cause breakage elsewhere, with zero possibility of automatically guarding against that.

With this I’m not saying that we shouldn’t update packages (we should! And pull requests are always welcome) but only that automated mass updates are greatly non-trivial. If someone wants to take the lead on that task, be welcome.

3 Likes
3

ok. automatic update is not easy …

First, a small step.

Is it possible to add some (new) metadata to the binarybuilder scripts?

so

  • I can detect the missing (minor/patch) releases …
  • I can write a (half) automatic update scripts - as I like …
4

What do you mean exactly? I’m not very happy about adding ad-hoc random metadata. In what format? How that can be used? How that can be generalised? In the meantime it’s much easier to do a pull request to update the package before getting solid answers to all these questions

5

still researching / thinking :thinking:

1.) “DEBIAN” best practice
the debian using a watch_file format and uscan tool

wget https://udd.debian.org/cgi-bin/upstream-status.json.cgi
cat upstream-status.json.cgi  | grep '"status"' | sort | uniq -c
   3344     "status": "error",
   7195     "status": "newer package available",
    898     "status": "only older package available",
     12     "status": "package available",
  19710     "status": "up to date",

2.) Fedora - https://release-monitoring.org

“Anitya is a release monitoring project.
Its goal is to regularly check if a project has made a new release. When Anitya discovers a new release for a project, it publishes a RabbitMQ message via fedora messaging. This makes it easy to integrate with Anitya and perform actions when a new release is created for a project. For example, the Fedora project runs a service called the-new-hotness which files a Bugzilla bug against a package when the upstream project makes a new release.”

example geos: GEOS · Anitya

3.) Repology https://repology.org/

example - “geos” package"

connect to udd
psql "postgresql://udd-mirror:udd-mirror@udd-mirror.debian.net/udd"

and check the watch_file columns

psql (12.10 (Ubuntu 12.10-0ubuntu0.20.04.1), server 11.14 (Debian 11.14-0+deb10u1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

udd=> \dt+ upstream*
                         List of relations
 Schema |       Name        | Type  | Owner |  Size   | Description 
--------+-------------------+-------+-------+---------+-------------
 public | upstream          | table | udd   | 12 MB   | 
 public | upstream_metadata | table | udd   | 4768 kB | 
 public | upstream_status   | table | udd   | 4928 kB | 
(3 rows)
  

udd=> select * from upstream  where source like 'geos';
-[ RECORD 1 ]-----------+----------------------------------------------------------------------------------
source                  | geos
version                 | 3.10.2-1
distribution            | debian
release                 | sid
component               | main
watch_file              | version=4                                                                        +
                        | opts=\                                                                           +
                        | dversionmangle=s/\+(debian|dfsg|ds|deb)\d*$//,\                                  +
                        | uversionmangle=s/(\d)[_\.\-\+]?((RC|rc|pre|dev|beta|alpha)\d*)$/$1~$2/;s/RC/rc/ \+
                        | https://download.osgeo.org/geos/ \                                               +
                        | (?:|.*/)geos-(?:[_\-]v?|)(\d\S*)\.(?:tar\.xz|txz|tar\.bz2|tbz2|tar\.gz|tgz)      +
                        | 
signing_key_pgp         | 
signing_key_asc         | 
debian_uversion         | 3.10.2
debian_mangled_uversion | 3.10.2
upstream_version        | 3.10.2
upstream_url            | https://download.osgeo.org/geos/geos-3.10.2.tar.bz2
errors                  | 
warnings                | 
status                  | up to date
last_check              | 2022-05-21 00:21:20.012495

fedora : geos: GEOS · Anitya Latest version : 3.10.2

the Julia geos_jll does not contain the patched versions.

side note:

Summary:

  • maybe we can link JLL packages to other repo metadata?
    • or just using uscan tool and metadata format.
  • setting “safe update policy” for every package
    • “patch” changes
    • “minor” changes: Postgresql using this … and safe to update.
    • “manual” - …
  • maybe we can add the “key-value” metadata
    • wikidata ID :
    • upstream bug reporting
    • upstream documentation

so for the metadata for geos_jll i.e. the format could be :

:thinking: