I would like to create a package for a model that ships with its own internal data. Basically, I’m looking for the Julia equivalent to R’s R/sysdata.rda (link).
I currently have something that looks like this:
using Serialization: deserialized
mydata = deserialize("data/mydata")
function myfunction(a, b)
a .* mydata[:,1] .+ b .* mydata[:,2]
My questions are:
(1) How do I generalize the data path so it works anywhere? Right now this only works when I’m doing analysis inside the package directory. I would like to be able to install this package from GitHub (via Pkg.add) and use it in other projects, but when I try to do that, the package install fails because it can’t find the data directory (because the path is relative to the current directory).
(2) Is the fact that myfunction uses a variable from outside its scope inefficient (similar to how functions using global variables are inefficient)? Or is it fine because it’s in a module?
An alternative to Artifacts is DataDeps.jl.
Either a normal datadep, with the data stored externally, e.g. on FigShare or Zenodo.
Or a ManualDataDep which will work for data stored in <project>/deps/data
(or you can put instructions on how to manually load the data.)
DataDeps lets you avoid worrying about where the data is stored,
because instead of writing things like ./../data/GoodData
you write datadep"GoodData" and it resolves to a string tht is the file path
(can also do datadepp"GoodData/myfile.csv etc)
IIRC you can do similar with artifacts using artifact"GoodData" but I am not 100% sure
There are number of pros and cons between Artifacts and DataDeps.
One pro of DataDeps is it works with julia 1.0.x (the LTS) not just with 1.3+
the others are around being more flexible for transport (can use a secure download e.g. AWSS3.jl, GoogleDrive.jl), and have post-fetch methods for unpacking random archieves (not just tarballs)
Downside of DataDeps is Artifacts use content addressing which is really clever and means it is basiclly impossibly to run into a name collision.
Artifacts also know how to clean themselves up when not needed anymore
Thanks! This is very useful. Technically, this is probably the (more) correct answer, but I marked the other one as the solution because it works with Julia 1.3 (which I happen to currently be using right now…though I’ll be sure to update to 1.4 soon!).