I’m working on a package that makes a new financial dataset, model and model activations available. The latter are quite big and I’m trying to make them available as Arrow files through artifacts (see here).
Unfortunately, they are so big that the CI runners run out of disk space: Activations to artifacts · pat-alt/TrillionDollarWords.jl@1a55e83 · GitHub. They are still small enough though to facilitate model probing in memory, which is quite neat and I’d really like to add that functionality to the package.
What’s the best way to go about this? Can I avoid downloading all artifacts during CI? Not ideal, but they are standardized, so as long as downloads work for one layer of activations, it’s reasonably safe to assume they will work for all layers.