# Git LFS objects could be installed with a package

**URL:** <https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553>\
**Category:** Package Management\
**Created:** [January 26, 2023, 5:22am UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553 "2023-01-26T05:22:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![lazzarello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lazzarello/32/37769_2.png) [@lazzarello](https://discourse.julialang.org/u/lazzarello)\
**Post date:** [January 26, 2023, 5:22am UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553/1 "2023-01-26T05:22:50Z")

</div>

There’s an [open issue in Pkg.jl](https://github.com/JuliaLang/Pkg.jl/issues/3171) about some complexities when installing package dependencies that contain git LFS metadata for large objects. My use case is to require public NASA planetary data in a library that is a dependency in a number of orbital simulation projects that are private to my employer.

I worked around the issue (which I documented in a comment) but I think there could be a better solution than the one proposed. I just don’t know what it is yet. I’d like to open up a discussion about possible improvements for users of packages that depend on large objects. I feel like this use-case lies outside of what BinaryBuilder provides but I’m open to a dialog.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 26, 2023, 8:41am UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553/2 "2023-01-26T08:41:20Z")

</div>

Is the data in tarball format? Then use [GitHub - simeonschaub/ArtifactUtils.jl: A tool for package developers for working with artifacts.](https://github.com/simeonschaub/ArtifactUtils.jl) to fetch, otherwise look into [GitHub - oxinabox/DataDeps.jl: reproducible data setup for reproducible science](https://github.com/oxinabox/DataDeps.jl), but data doesn’t belong to the package repository itself.

---

<div class="post-metadata">

**Author:** ![lazzarello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lazzarello/32/37769_2.png) [@lazzarello](https://discourse.julialang.org/u/lazzarello)\
**Post date:** [January 26, 2023, 3:52pm UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553/3 "2023-01-26T15:52:50Z")

</div>

Downloading the data is not the problem, referencing the data across library installations is the problem. This is gonna dip into domain specific information but I think there’s enough in common for me to continue.

tl;dr DataDeps seems to address this problem, (thanks @giordano ) I speak about below. I still believe that “files” belong in the package repository. Continue reading if you want to know what I mean.

I am not an astrophysicist but I work with them, helping them build better software for GNC (Guidance Navigation and Controls) of spacecrafts. SPICE is a software toolkit for many things, in this case calculating geometry for mission planning, operations and analysis.

NASA provides what the toolkit calls “kernels”, best described by [the project’s own extensive documentation](https://naif.jpl.nasa.gov/pub/naif/toolkit_docs/Tutorials/pdf/individual_docs/12_intro_to_kernels.pdf) as

> “Kernel” means a file containing “low level” ancillary data that may be used,  
> along with other data and SPICE Toolkit software, to determine higher level  
> observation geometry parameters of use to scientists and engineers in planning  
> and carrying out space missions, and analyzing data returned from missions.

I suspect this toolkit was created in The Before Times when general purpose computers did not have (nearly) infinite resources for calculating planetary geometry, let alone cool high level languages like Julia to do it! Regardless, there is a large academic and professional community that knows how to use SPICE and depends on it for their results.

There are literally thousands of kernels available to the public from a NASA maintained database, searchable from a rather old-school looking website. For example, here are all the kernels (and future kernels because this mission is not over) for the [OSIRIS-REX spacecraft](https://pds.jpl.nasa.gov/ds-view/pds/viewCollection.jsp?identifier=urn%3Anasa%3Apds%3Aorex.spice%3Aspice_kernels&version=12.0) mission data.

These kernels can run into the hundreds of megabytes in size with published quantities in the thousands. This makes it problematic to “store them all” in our own project, so to speak. NASA employs people to publish new kernels as new spacecraft enter orbits.

So…rather than thinking of this binary information as a file download, I’m thinking of it more like program data which is dependent on source code. It’s not really _a shared library_ like that OpenSSL bug that I commented about not long ago. I believe if SPICE were rewritten, this information would be stored in some special purpose database and our Julia code would just…open a database network driver and ask for it. _(side note, build this database. that sounds useful)_

_whew, still with me?_

In the mean time, we have git LFS. It can store references to these “kernels” and allow us to commit them to source code just like any other file _because they are files._ Our source code can reference these files by path. Yea! But the size of the object is far greater than any reasonable person would want to include in the project history.

Since Julia uses git as the primary (exclusive?) interface for package management, it would make sense if…a SPICE.jl _package_ would _manage_ these large objects, so the user doesn’t have to. In some cases the user is a computer program running a ci pipeline with no human intervention.

Git LFS might not be the best solution but it’s one we have. Downloading kernels with custom code from a URL for every run (especially in CI) can be expensive for already expensive (hours long) simulation runs. In the mean time I’ll check out DataDeps and see if it solves this problem. Maybe I’ll also work on a micro-service setup between the SPICE functions that load big kernels and a network database that spits out results.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 26, 2023, 4:01pm UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553/4 "2023-01-26T16:01:47Z")

</div>

> [@lazzarello](#):
>
> Git LFS might not be the best solution but it’s one we have. Downloading kernels with custom code from a URL for every run (especially in CI) can be expensive for already expensive (hours long) simulation runs. In the mean time I’ll check out DataDeps and see if it solves this problem. Maybe I’ll also work on a micro-service setup between the SPICE functions that load big kernels and a network database that spits out results.

You’re looking into the wrong solution for a real problem. If you’re understandably concerned about saving CI cycles, the solution is to cache the data (see for example [GitHub - julia-actions/cache: A shortcut action to cache Julia artifacts, packages, and registries.](https://github.com/julia-actions/cache) if using artifacts), not to clog your repository. If using artifacts, the tarballs should also be duplicated by the PkgServer (at least gzipped tarballs, I’m not sure about the other compression formats), for persistency of the data in the juia ecosystem.

---

<div class="post-metadata">

**Author:** ![lazzarello](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lazzarello/32/37769_2.png) [@lazzarello](https://discourse.julialang.org/u/lazzarello)\
**Post date:** [January 26, 2023, 4:15pm UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553/5 "2023-01-26T16:15:53Z")

</div>

I’m not worried about clogging anything but thank you for your concern. [This video led by a nice person describes](https://www.youtube.com/watch?v=kSlQpzccRaI) this exact problem (it’s one of reproducible science) and how DataDeps solves it. Looking forward to pulling that into my project and giving it a try.

Still think git LFS in packages would (sort of) work the same way.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 26, 2023, 4:27pm UTC](https://discourse.julialang.org/t/git-lfs-objects-could-be-installed-with-a-package/93553/6 "2023-01-26T16:27:05Z")

</div>

> [@lazzarello](#):
>
> I’m not worried about clogging anything

I know you aren’t (clogging the repository is _your_ solution), _I_ am 🙂