# Package with large artifacts - CI runners out of disk space

**URL:** <https://discourse.julialang.org/t/package-with-large-artifacts-ci-runners-out-of-disk-space/108964>\
**Category:** Package Management\
**Tags:** artifacts\
**Created:** [January 18, 2024, 2:44pm UTC](https://discourse.julialang.org/t/package-with-large-artifacts-ci-runners-out-of-disk-space/108964 "2024-01-18T14:44:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pat-alt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pat-alt/32/34264_2.png) [@pat-alt](https://discourse.julialang.org/u/pat-alt)\
**Post date:** [January 18, 2024, 2:44pm UTC](https://discourse.julialang.org/t/package-with-large-artifacts-ci-runners-out-of-disk-space/108964/1 "2024-01-18T14:44:55Z")

</div>

Hi there,

I’m working on a [package](https://github.com/pat-alt/TrillionDollarWords.jl) that makes a new financial dataset, model and model activations available. The latter are quite big and I’m trying to make them available as Arrow files through artifacts (see [here](https://github.com/pat-alt/TrillionDollarWords.jl/releases/tag/activations_2024-01-17)).

Unfortunately, they are so big that the CI runners run out of disk space: [Activations to artifacts · pat-alt/TrillionDollarWords.jl@1a55e83 · GitHub](https://github.com/pat-alt/TrillionDollarWords.jl/actions/runs/7571107349). They are still small enough though to facilitate model probing in memory, which is quite neat and I’d really like to add that functionality to the package.

What’s the best way to go about this? Can I avoid downloading all artifacts during CI? Not ideal, but they are standardized, so as long as downloads work for one layer of activations, it’s reasonably safe to assume they will work for all layers.

Thanks!

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 18, 2024, 6:22pm UTC](https://discourse.julialang.org/t/package-with-large-artifacts-ci-runners-out-of-disk-space/108964/2 "2024-01-18T18:22:06Z")

</div>

> [@pat-alt](#):
>
> The latter are quite big

Quantify “quite big”.

> [@pat-alt](#):
>
> Unfortunately, they are so big that the CI runners run out of disk space

Github-hosted runners [now have 150 GiB of local storage](https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/), it’s quite remarkable if you managed to run out of space there because of artifacts (but see also the request above).

> [@pat-alt](#):
>
> Can I avoid downloading all artifacts during CI?

Artifacts can be [lazy](https://pkgdocs.julialang.org/v1/artifacts/#Artifact-types-and-properties), which means can be downloaded on-demand when you need them. Sounds like this is what you want to do?

---

<div class="post-metadata">

**Author:** ![pat-alt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pat-alt/32/34264_2.png) [@pat-alt](https://discourse.julialang.org/u/pat-alt)\
**Post date:** [January 19, 2024, 6:52am UTC](https://discourse.julialang.org/t/package-with-large-artifacts-ci-runners-out-of-disk-space/108964/3 "2024-01-19T06:52:38Z")

</div>

Thanks for the quick response @giordano.

> [@giordano](#):
>
> Quantify “quite big”.

They are about 300MB [each](https://github.com/pat-alt/TrillionDollarWords.jl/releases/tag/activations_2024-01-17) and there’s 24 of them, so it adds up but definitely not to 150gb.

> [@giordano](#):
>
> Artifacts can be [lazy](https://pkgdocs.julialang.org/v1/artifacts/#Artifact-types-and-properties), which means can be downloaded on-demand when you need them. Sounds like this is what you want to do?

This is what I had missed and really should have done in the first place, thanks!