# Static data in a package

**URL:** https://discourse.julialang.org/t/static-data-in-a-package/25165
**Category:** New to Julia
**Created:** [June 11, 2019, 7:23pm UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165 "2019-06-11T19:23:43Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Pavel\_Kalouguine](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_kalouguine/32/7730_2.png) [@Pavel\_Kalouguine](https://discourse.julialang.org/u/Pavel_Kalouguine)
#### Post date: [June 11, 2019, 7:23pm UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/1 "2019-06-11T19:23:43Z")

</div>

What is the best place for static data in a package? The data in question is Waasmaier-Kirfel table of atomic X-ray formfactors. The current solution is to place the datafile in the subfolder “data” of the “src” folder, and load it at compile time in the following way:

```julia
function load_waaskirf()
    waaskirf_filepath = joinpath(@ __DIR__ , "data", "f0_WaasKirf.dat")
    t=readlines(waaskirf_filepath)
    # More code...
end

const wktab=load_waaskirf()

```

Is this the right way of doing things? In particular, is it OK to place the data in a subfolder of “src”?

---

<div class="post-metadata">

### Author: ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)
#### Post date: [June 12, 2019, 12:29am UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/2 "2019-06-12T00:29:31Z")

</div>

Depending on how big the dataset is, you might consider DataDeps.jl instead. If it’s really big, you might also want to use JLD or some other way of storing directly as a Julia data structure so you don’t have the overhead of re-parsing it each time (DataDeps can do stuff like that too so you don’t have to store the large file in your repo)

But if it’s a relatively small file I think that’s a fine solution.

---

<div class="post-metadata">

### Author: ![Pavel\_Kalouguine](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_kalouguine/32/7730_2.png) [@Pavel\_Kalouguine](https://discourse.julialang.org/u/Pavel_Kalouguine)
#### Post date: [June 12, 2019, 7:42am UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/3 "2019-06-12T07:42:32Z")

</div>

Thank you for the reference to DataDeps.jl, this seems to be the right way to distribute code examples using lots of data. However, in this case the data file is about 200 lines long, and its contents did not change since 1995. In this sense the data is as close to the fundamental physical constants as it gets, and distributing it with the package seems to be the right thing. The question is rather about the package layout - should we keep the data in a subfolder of “src” or elsewhere.

Incidentally, I wonder whether loading the data at compile time is OK. An alternative would be using the **init** () function, but it looks like the latter is considered as a fallback solution, for the cases when the compile time loading is not possible (e.g. when the package uses third part libraries needed to be initialized at runtime).

---

<div class="post-metadata">

### Author: ![tlienart](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlienart/32/7640_2.png) [@tlienart](https://discourse.julialang.org/u/tlienart)
#### Post date: [June 12, 2019, 8:01am UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/4 "2019-06-12T08:01:30Z")

</div>

Given that it’s a small file, an alternative would be to just declare this as a constant in a julia file say `waaskirf.jl` with

```julia
const WAASKIRF = """
#F f0_WaasKirf.dat                                                           
#UT Elastic Photon-Atom Scattering, relativistic form factors.
#UF0TYPE PARAMETRIZATION ; TABLE OR PARAMETRIZATION?
#UIDL xf0
(rest of the lines here)
""" # end

```

and then just include that in your package:

```julia
module Package
# ...
include("waaskirf.jl")
# ...
end

```

wherever before you had something like `const WAASKIRF = read("path/to/waaskirf.dat", String)`.

---

<div class="post-metadata">

### Author: ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [June 12, 2019, 8:32am UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/5 "2019-06-12T08:32:38Z")

</div>

> [@Pavel\_Kalouguine](#):
>
> The question is rather about the package layout - should we keep the data in a subfolder of “src” or elsewhere.

I would make a `data/` directory and put it there. Also, declare a

```julia
data_path() = abspath(joinpath(@ __DIR__ , "..", "data", "constants.dat"))

```

function in the _source_.

Alternatively, depending on the format, putting it in a `Matrix` or `Dict` in the source code could be fine, too.

---

<div class="post-metadata">

### Author: ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [June 12, 2019, 8:39am UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/6 "2019-06-12T08:39:43Z")

</div>

Along the lines of what @tlienart proposed, in [PeriodicTable.jl](https://github.com/rahulkp220/PeriodicTable.jl/blob/master/src/elements.jl) and [PhysicalConstants.jl](https://github.com/JuliaPhysics/PhysicalConstants.jl/blob/master/src/codata2018.jl) we hard coded the (small) datasets.

---

<div class="post-metadata">

### Author: ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [June 12, 2019, 9:01am UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/7 "2019-06-12T09:01:53Z")

</div>

I would suggest that the correct location is:

```julia
/deps/data/Constants/constants.dat

```

i.e. from `/src/`

```julia
joinpath(@ __DIR__ , "..", "deps", "data", "Constants", "constants.dat"

```

This is where DataDeps looks for package specific data `<CURRENTPKG>/deps/data/` is always on the DataDeps load path.  
One can reference data there with a `ManualDataDep`,  
but there is little need to.

Alt: and especially if it is plan text: tlienart’s suggestion is pretty solid, since it is so small.

> [@tlienart](#):
>
> Given that it’s a small file, an alternative would be to just declare this as a constant in a julia file say `waaskirf.jl` with

I suggest using a `raw` string macro:

```julia
const WAASKIRF = raw"""
 Why use raw string ? 
 $ are not interpolated in raw strings
 backslashed (\) are not treated as escapes -- no \n to newline 
(rest of the lines here)
""" # end

```

---

<div class="post-metadata">

### Author: ![Pavel\_Kalouguine](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_kalouguine/32/7730_2.png) [@Pavel\_Kalouguine](https://discourse.julialang.org/u/Pavel_Kalouguine)
#### Post date: [June 12, 2019, 5:33pm UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/8 "2019-06-12T17:33:06Z")

</div>

The main reason of not putting the data directly in a source file is to keep the data file “pristine”, as it should be for the third party component.

It seems that the common practice is either to put the static data either in “data” or in “deps/data” directory in the main package folder. I thought at first that “deps/data” is a better choice (“deps” is already a de-facto standard place for the third party stuff), but it seems that “deps” is mostly used as the placeholder for the build.jl script and the downloaded binary third party libraries. Thus “data” dubdirectory of the main folder seems to be the right place.

---

<div class="post-metadata">

### Author: ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [June 12, 2019, 5:40pm UTC](https://discourse.julialang.org/t/static-data-in-a-package/25165/9 "2019-06-12T17:40:44Z")

</div>

> [@Pavel\_Kalouguine](#):
>
> Thus “data” dubdirectory of the main folder seems to be the right place.

AFAIK there is no strong convention, so anything goes at this point. `data/` is a sensible choice.

If these things get formalized later you can always change it, I would not worry too much about it.
