PSA: All CI services temporarily halted

About an hour ago, we noticed a compromise of the Julia CI infrastructure and were shortly thereafter contacted by a security researcher with further details. We have temporarily halted all CI services on JuliaLang and JuliaPackaging while we remediate this incident. At this time, we have no evidence of any malicious activity - however our review of relevant logs is still in progress. Unfortunately, recovery may take some time. We will keep you posted. Our apologies for the interruption.

We have partially restored CI services for Julia base. Yggdrasil CI services remain offline and will probably continue to be offline through tomorrow evening.

Hi, Is there going to be a write-up on this? I think there is a genuine interest in how this occurred and how it got solved. AI brings a lot of malicious power, which creeps up everywhere and fuels supply chain attacks. So having insight might help designing better Julia targeted risk controls.

We can certainly share more details. It may take a bit of time for us to properly write things up; presently our efforts are focused on restoring CI services.

Hi,

It’s been more than a week since the original announcement. Are you going to share more details soon? What is the impact of the compromise? Can I trust Julia binaries from the official website? What about binary/JLL packages?

We’re still working on this and will share more details upon completion. Automatic publication of nightly binaries is still paused (though we manually pushed a few in the past week). As stated in the original post, there is no indication of malicious activity.

Nightly binaries should start appearing automatically again for master. Release and PR binaries will need another couple days.

Is there a new version of juliareleases.asc I can use to verify the nightly binaries?

Ah yes, good point - all signing secrets have been rotated. We will update the website, etc, once the other pipelines are restored as well, but for now, here’s the updated pubkey:

-----BEGIN PGP PUBLIC KEY BLOCK-----

xsFNBGoseh4BEADemCYuKp7uLR304Eij5hYPXHhl0FuOsOvWBJJal89SYRp0r7ix
uxkotCtMeQLBjj3n/sDFRYORT7Hil0Aylp58J4ZInHJutmpEm2V/AijyLcrmeGXJ
QiCF3rJzFyTgr7MAXwzpq7KtmJlXZQzGUknsnNs9PzLFm81qvchpsmW1ZE8pOzz+
wOItlN7HnbZpdmQwRPRkp5X+FePqT4ICci/ZN88SkqE5gHStxPvUbm8Vlg4CJ6bp
TG7HnXda6X1CWCa/5+Z0PwsThHipZ1Yv/Snxm5V3BmI5LiqQbESyGezRs96wbBOI
o2VTqp6yvGCt85XNGGsmuH+L69lEzdN8SxQwPN8rS2OBocWilGabl/RtfIzKLH16
PmrfAuuNzA+TGFpztoFMZVt3jCL0CWNJALM9vd/vevuhuTlyXZ9JjbYplEQFy0e+
9IHvXCsYB1MKPmL8ZlbVqRJ0niTsfCDs8+H2dO2md/x5JJPBY58nk+wFnTEmV/sN
WX5f6CrZUfFeGmmXoRpdbodtx0p7zrYi9kC6hUYS+wffsNeAkz3wWQ9bOPRrWUga
xwPln8CVQcnlQF37NiEHgtg/xF6HOE15IkVR6x+JfrrmjHslhX5YLvRLAYCFUPmb
r87tVdFkCafwitTh58rvY3ZeF/WHdUuqmrtweFRRjICa3qQAvACh94QiuwARAQAB
zTJKdWxpYSBSZWxlYXNlIFNpZ25pbmcgS2V5IDxidWlsZGJvdEBqdWxpYWxhbmcu
b3JnPsLBdgQTAQgAIBYhBGS3eaVwly//e/wrVOrUceGh8sEKBQJqLHoeAhsDAAoJ
EOrUceGh8sEKe1YP/23Qp9jFyLhyK+OKkGvt+REgKSB4UKOTxpsc4HvUFvUUB4Mb
9AjBdzUPGavOSMbkDytZNz7yFrb/rWfwd5tgMP973xBYzipztIj3MsQC8X7Qg8sS
rxdh37+2G4di34/AIo0HuPIGwne9hJXxyPa+Opj8DR7K8dC+aeMOydewUVoNQhz4
Wnuir+LryM4Ubs0+pWYTTU51tj8ADgirWDz84ZajLwANAPkYNdgsfnk7eV05/gc3
GC8ygahSZgTfnwojwvKoFfLDsM6z0zjTryoJY68xe3NYh59kdEN0W8jKw3OJGMcX
T82zckNSwAEIJOdMaqauRAHeGJ/kPovyy3enxsq8VKA9eBAxlmlUcuodnLCZHI+d
g/VKMUPegMuvyaZnY2VAHBmONKX4jpeMZYbRYfyXa2InixN/hcaaA3g/90yevCH+
GNmN1MaXfkjISUdsMvLQMixVeQ0/tm+SHSopFwTJtuzEaAyyNAkLL4U1iRRSMwW1
BaJrc943PhBQ4ytkWJhm0ge1e2qRB3SSKYzoOTji98hA/hUPmt8ZO+yb8FngJXbw
+fa+onWVxbGp8AO8+TNvJb89iuCC/VoMtf9sdSnZ85Zj3+jJk+aVauQHohgfwVOX
eBjaNN1vJCy+1efZArpk6d6H94ZeCMLnEzxzVQz+3AF1BgZVQNF8hPD2qiK0
=uJpw
-----END PGP PUBLIC KEY BLOCK-----