# Should Julia Packages Allow Authors to Specify "Essential" Directories for Installation?
**URL:** https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238
**Category:** Package Management
**Tags:** question, pkg, installation
**Created:** [February 15, 2024, 7:04am UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238 "2024-02-15T07:04:14Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@singularitti](https://discourse.julialang.org/u/singularitti)
#### Post date: [February 15, 2024, 7:04am UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/1 "2024-02-15T07:04:14Z")
I’ve been pondering an idea to optimize the storage footprint of Julia packages and would love to get your insights and opinions on it.
### The Issue
While exploring various packages, I’ve noticed that directories like `test`, `docs`, `examples`, and `notebooks` often contain large files (images, binaries, etc.) that are not directly necessary for the functionality of the package itself. These files can significantly increase the total disk space used by installed packages, which might be an issue for users with limited storage or those who prefer a lean installation.
### The Proposal
What if package authors could specify which directories are essential for their package’s operation, and only those directories would be included by default when users install the package? This could potentially reduce the installation size of packages by excluding non-essential directories (such as `docs` and `examples`) unless explicitly requested by the user.
The idea is not to limit the availability of these resources but to give users the option to install a minimal version of the package, focusing on the essentials needed for its operation. Additional resources could still be accessed by those interested, for example, by cloning the repository or opting in through the package manager.
### Examples
Packages such as `GR.jl` and `ColorSchemes.jl` include substantial non-source files within their `docs`, `examples`, or `test` directories. While important for development purposes, these files may not be needed by all users, particularly those focused on using the packages’ functionalities rather than modifying or extending them.
### Benefits
- **Reduced Disk Space Usage:** This approach allows for more efficient use of disk space, catering to users with limited storage or those who prefer a minimal installation footprint.
- **Enhanced Installation Efficiency:** Streamlining the number of files to download can lead to quicker installation times and a more user-friendly experience.
### Your Thoughts?
I’m curious about your thoughts on this idea. Do you think it’s a feasible enhancement for the Julia package ecosystem? Would it be beneficial for both package authors and users? Are there potential downsides or challenges that might arise from implementing such a feature?
See [Feature Request: Option to Exclude Non-Essential Directories on Package Installation · Issue #3798 · JuliaLang/Pkg.jl (github.com)](https://github.com/JuliaLang/Pkg.jl/issues/3798)
---
### Author:  [@Elrod](https://discourse.julialang.org/u/Elrod)
#### Post date: [February 15, 2024, 8:13am UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/2 "2024-02-15T08:13:01Z")
LoopVectorization’s master branch shouldn’t include the logo, but it does currently: [LoopVectorization.jl/docs/src/assets at main · JuliaSIMD/LoopVectorization.jl · GitHub](https://github.com/JuliaSIMD/LoopVectorization.jl/tree/main/docs/src/assets)
That said, most of the large files for the docs are on the `docsassets` branch: [LoopVectorization.jl/docs/src/assets at docsassets · JuliaSIMD/LoopVectorization.jl · GitHub](https://github.com/JuliaSIMD/LoopVectorization.jl/tree/docsassets/docs/src/assets)
This is a pattern you can already use: do not commit any large files that are not needed to the release (or `main`) branches, but only to some special branch that you can link with the path to the branch
[LoopVectorization.jl/docs/src/examples/matrix\_multiplication.md at 9ea87422cd9ae0e11e7768be26153840e5f488c8 · JuliaSIMD/LoopVectorization.jl · GitHub](https://github.com/JuliaSIMD/LoopVectorization.jl/blob/9ea87422cd9ae0e11e7768be26153840e5f488c8/docs/src/examples/matrix_multiplication.md?plain=1#L35)
This handles docs, where all you need is a url. Examples might be similar?
Not sure what the best way to handle tests is.
But, ideally, only the `src` directory + `Project.toml` would be needed for using a package.
---
### Author:  [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [February 15, 2024, 8:26am UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/3 "2024-02-15T08:26:50Z")
> [@Elrod](#):
>
> But, ideally, only the `src` directory + `Project.toml` would be needed for using a package.
Plus `ext` if it is using extensions.
Plus `deps` (rarely used now days. but not never)
Plus some packages keep other stuff they use outside of `src` (PkgTemplates is one such), but probably shouldn’t be.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [February 15, 2024, 11:09am UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/4 "2024-02-15T11:09:34Z")
> [@singularitti](#):
>
> **Reduced Disk Space Usage:** This approach allows for more efficient use of disk space, catering to users with limited storage or those who prefer a minimal installation footprint.
Technically this is correct, but in practice we are talking about… megabytes, I guess. Sure, saving space is always nice, but is it worth the complication in practice?
> [@singularitti](#):
>
> **Greater Control for Package Authors:** Authors can tailor their package’s installation footprint, focusing on delivering essential components while offering additional resources as optional.
I don’t understand why you are phrasing this as a benefit to package authors, when it may just mean having to fix something that worked before. Eg if a package keeps some binary tables in `data/`, not in `src/`, it would have to specify that explicitly in order to continue working.
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [February 15, 2024, 1:20pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/5 "2024-02-15T13:20:04Z")
> [@Tamas\_Papp](#):
>
> in practice we are talking about… megabytes, I guess
Looking at ~/.julia/packages, I notice large unnecessary files in popular packages, for example:
- test/testfiles in CSV, 28 Mb
- docs/assets, assets, RPRMakie/examples in Makie, 81 Mb
- examples in GR, 25 Mb
- docs in ColorSchemes, 19 Mb
- test/results in Sobol, 18 Mb
And even more packages with files of ~10 Mb or less.
These sizes add up for each installed version of each package. Still smaller than `compiled/`, but would be nice to have an easy way to avoid storing these files everywhere.
Also, to avoid downloading them in the first place.
Especially nice, if possible for packages from both pkgserver or git repos.
---
### Author:  [@mike.ingold](https://discourse.julialang.org/u/mike.ingold)
#### Post date: [February 15, 2024, 1:41pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/6 "2024-02-15T13:41:13Z")
My immediate thought was to whether this could help reduce the size of compiled binaries (e.g. via PackageCompiler.jl) when a utilized package pulls in a lot of extraneous docs/testdata/etc, but I’m fairly certain those binaries wouldn’t include any of that anyway.
I could imagine this being useful for a package that for some reason want to include a large quantity of optional content and wants a flag to indicate that a user has opted-in to the full download… but you could currently accomplish this with git branches, e.g. `add MyPackage` vs `add MyPackage#thewholeenchilada`. That puts some extra work on the package maintainer to maintain parallel branches, but adding this to the package system likewise adds complexity to the whole package system.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [February 15, 2024, 1:47pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/7 "2024-02-15T13:47:37Z")
I would like this feature. I have a package that has ~300kb of functional stuff vs. 170Mb of data that is required for testing. Yes, I could store the data somewhere else, but that would increase quite a lot the complexity of the development workflow.
It could be a line in Project.toml `essential_dirs = ["src"]`.
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [February 15, 2024, 1:49pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/8 "2024-02-15T13:49:58Z")
I think for the most part the responsibility sits with package authors not committing those files. Perhaps one of the automatic checks during the package registration process could warn about large files.
I agree that maybe in the longer term this might be a good feature to think about. Lean vs complete package installations. I also had the thought that it could go in the Project.toml file (even before @lmiq edited his post to say that 🙂 )
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [February 15, 2024, 2:00pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/9 "2024-02-15T14:00:07Z")
> [@tbeason](#):
>
> responsibility sits with package authors not committing those files
By the way, if someone wants to explain how one can develop a package, its docs, and its tests, having a some files in a branch and other files in another branch, without having to clone the package repo multiple times, or having multiple repositories, I’m open to suggestions.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [February 15, 2024, 2:01pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/10 "2024-02-15T14:01:03Z")
> [@lmiq](#):
>
> 170Mb of data that is required for testing. Yes, I could store the data somewhere else
Yes, in artifacts. This is one of the suggested use cases.
[https://pkgdocs.julialang.org/v1/artifacts/](https://pkgdocs.julialang.org/v1/artifacts/)
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [February 15, 2024, 2:07pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/11 "2024-02-15T14:07:23Z")
Well, yes, that is indeed complicating the developers life. One alternative could be having a default `artifacts` directory which was not downloaded upon installation, then we could just put the heavy data there.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [February 15, 2024, 2:13pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/12 "2024-02-15T14:13:25Z")
Note that putting binary data in a git repo meant for code is itself a bad idea.
You can get away with it if it is small (and “small” can be hundreds of megabytes or more), but it is still a bad idea.
The right solution is putting all that data in artifacts. See all the nice tooling, eg
> **[GitHub - JuliaPackaging/ArtifactUtils.jl: A tool for package developers for...](https://github.com/JuliaPackaging/ArtifactUtils.jl)**
>
> A tool for package developers for working with artifacts. - JuliaPackaging/ArtifactUtils.jl
---
### Author:  [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)
#### Post date: [February 15, 2024, 2:25pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/13 "2024-02-15T14:25:05Z")
> [@singularitti](#):
>
> **Enhanced Installation Efficiency:** Streamlining the number of files to download can lead to quicker installation times and a more user-friendly experience.
A package download from a package server is one single file, regardless of size. Moreover it is indexed by a content hash, which is the same as is stored in the registry, so packages are by design indivisible units. You could discuss filtering out directories when unpacking the package archive, but that doesn’t come without its own share of drawbacks.
There are primarily two options to store things in the repository without being included in the package:
- Place them in some separate branch which is not used by package releases.
- Place the package in a subdirectory of the repository and additional stuff outside of the subdirectory.
Regardless how you organize it, it’s still not a great idea to put large data in the repository, since this will hurt the developers with slow git clones and large disk usage. The only sound approach is to have large data in external sources and download it on demand.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [February 15, 2024, 2:26pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/14 "2024-02-15T14:26:06Z")
I find the artifact amazing, particularly for shared files among projects. But for my usecase (a bunch of ~10mb files) required for testing the package, I don’t really see a cost/benefit advantage in adding that complication. At least I would have to disentangle the data structure of my tests from the data files, and that can be quiet messy. And the test files can get updated when I update the package, so I would have to separately update all artifacts…
This is the first time that I find some compelling reason not to store this data in the package repo. But I need a more compelling reason to dive into the artifact complications.
---
### Author:  [@singularitti](https://discourse.julialang.org/u/singularitti)
#### Post date: [February 15, 2024, 11:17pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/15 "2024-02-15T23:17:35Z")
That’s a good suggestion!
However, it’s more complicated than that. There are not only images for logos but also sometimes notebooks or data. It’s not intuitive for developers to put them in other branches, since they may not be found by users, not to mention that these packages often have many branches. And, if that works, we have to persuade some developers in the Julia community to follow this convention, which seems harder than changing the `Pkg.jl` behaviors.
---
### Author:  [@nhz2](https://discourse.julialang.org/u/nhz2)
#### Post date: [February 17, 2024, 4:33pm UTC](https://discourse.julialang.org/t/should-julia-packages-allow-authors-to-specify-essential-directories-for-installation/110238/16 "2024-02-17T16:33:40Z")
Having essential and non-essential directories in a repo can lead to testing and installation issues. For example, it is easy to accidentally add an essential file to a non-essential directory. Then when running CI everything looks fine, but when users install the package it doesn’t work. This happened to me when I added a file to ensure zarr-python could read files from Zarr.jl but didn’t understand the details of how python packaging ignores some files ([Run Zarr CI tests out-of-tree · Issue #1347 · zarr-developers/zarr-python · GitHub](https://github.com/zarr-developers/zarr-python/issues/1347)).
I use the artifact system for storing large example zip files in [ZipArchives.jl/test/Fixture-README.md at main · JuliaIO/ZipArchives.jl · GitHub](https://github.com/JuliaIO/ZipArchives.jl/blob/main/test/Fixture-README.md). I store the tarball as an extra file in one of the releases on GitHub. Maybe this kind of workflow could be made easier somehow.