# Providing datasets in a package

**URL:** <https://discourse.julialang.org/t/providing-datasets-in-a-package/111622>\
**Category:** General Usage\
**Created:** [March 14, 2024, 3:30pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622 "2024-03-14T15:30:16Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [March 14, 2024, 3:30pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/1 "2024-03-14T15:30:16Z")

</div>

Hi,

I want to include some dataset, on the form of an object (say of type `Mytype` from my package, so that `import Mypackage` will provide a `Mypackage.myobject` binding for this dataset.

I might have to do this for several datasets, and these objects are a bit complicated to instantiate: Let’s consider for the sake of the argument that they are constructed from a .csv file through a potentially long-running function `build_my_object("this_particular_file.csv")`

The issue is that I cannot serialize the object itself, since the definition of `Mytype` and of `build_my_object` might change from one package version to another.

Likewise, the .csv files _might_ change from one version to the other… at least during early development.

**How can i set up such a system that would build the objects at precompilation time ?**

I’m a bit lost between `DataDeps.jl`, `Pkg.Artifacts`, `ArtifactsUtils.jl`… is there somewhere a guide on this particular problem that i could follow and/or an example of a package that does this already ?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [March 14, 2024, 3:53pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/2 "2024-03-14T15:53:38Z")

</div>

You don’t have to do anything special to get it built at precompilation. Just

```julia
module MyPackage
const data = long_running_data_preparation()
end

```

will serialize `MyPackage.data` as part of the precompilation and `import MyPackage` will deserialize it.

You may or may not want to use some of the mentioned packages to manage the source of your data, but that is independent of the precompilation question.

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [March 14, 2024, 3:56pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/3 "2024-03-14T15:56:58Z")

</div>

Thanks for pointing this out! Looks fantastic that it will auto serialize it at precompilation, but now that you say it it makes sense.

So then I _could_ use something fancy to store my `.csv`s and these files only. Do you have a recommendation for this ? I think it would make more sense to have them live in the package’s registry

---

<div class="post-metadata">

**Author:** ![non-Jedi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/non-jedi/32/3645_2.png) [@non-Jedi](https://discourse.julialang.org/u/non-Jedi)\
**Post date:** [March 14, 2024, 4:03pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/4 "2024-03-14T16:03:01Z")

</div>

I couldn’t remember exactly how precompiling worked, so I was playing around with it and didn’t get my reply in before @GunnarFarneback did, but maybe it’s useful to have a self-contained example in this thread:

```sh
$ julia -e "using Pkg; Pkg.generate(\"PrecompileExample\")"
  Generating project PrecompileExample:
    PrecompileExample/Project.toml
    PrecompileExample/src/PrecompileExample.jl
$ echo "module PrecompileExample
f() = sleep(5); return 3
const x = f()
end # module" > PrecompileExample/src/PrecompileExample.jl
$ julia --project=PrecompileExample/ -e "@time using PrecompileExample"
  5.526768 seconds (141.25 k allocations: 10.659 MiB, 2.78% compilation time)
$ julia --project=PrecompileExample/ -e "@time using PrecompileExample"
  0.007801 seconds (5.61 k allocations: 374.812 KiB)

```

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [March 14, 2024, 4:07pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/5 "2024-03-14T16:07:40Z")

</div>

> [@lrnv](#):
>
> So then I _could_ use something fancy to store my `.csv`s and these files only. Do you have a recommendation for this ?

It depends on how large the files are and what kind of storage options you have for them. However, if they are small enough that you can conveniently store them in the package repository, that’s likely to be the easiest option.

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [March 14, 2024, 4:09pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/6 "2024-03-14T16:09:46Z")

</div>

Okay so i’ll simply do that, these files are not huge. Do i need a bit of caution to access them ? Should I put them in the `src/` folder or somewhere else ?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [March 14, 2024, 4:15pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/7 "2024-03-14T16:15:15Z")

</div>

You can place them wherever you want inside your package. You want to start from `@ __DIR__ ` when you construct the path to them, e.g.

```julia
csv_path = joinpath(@ __DIR__ , "..", "data", "data.csv")

```

if the code is in a file in `src` and you store the csv in a sibling directory `data`.

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [March 14, 2024, 4:16pm UTC](https://discourse.julialang.org/t/providing-datasets-in-a-package/111622/8 "2024-03-14T16:16:40Z")

</div>

Perfect then I’ll use that. Thanks a lot for the guidance !