# Creating Artifacts.toml for existing tarball
**URL:** https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365
**Category:** General Usage
**Created:** [January 14, 2020, 9:06pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365 "2020-01-14T21:06:37Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@ssfrr](https://discourse.julialang.org/u/ssfrr)
#### Post date: [January 14, 2020, 9:06pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/1 "2020-01-14T21:06:38Z")
I’m trying to figure out how to create an `Artifacts.toml` to download some data that my package depends on. I’m creating a `setup_artifacts.jl` script to create it.
Here’s what I have so far:
```julia
using Pkg.Artifacts
using Pkg.GitTools
using Tar
toml = joinpath(@ __DIR__ , "Artifacts.toml")
ir_url = "https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
ir_hash = create_artifact() do artifact_dir
tarball = download(ir_url)
try
global tarball_hash = bytes2hex(GitTools.blob_hash(tarball))
Tar.extract(tarball, artifact_dir)
finally
rm(tarball)
end
end
bind_artifact!(toml, "tidmarsh_irs", ir_hash;
download_info=[(ir_url, tarball_hash)],
lazy=true)
```
My goal here is to have an Artifacts.toml file that I can include with my package that will download the referenced file and unpack it, making the contents (a bunch of `.flac` files in my case) available at runtime.
Currently `Tar.extract` is choking on the tarball (“invalid octal digit: `G`”), perhaps because it’s compressed?
Also - is this the right way to be doing this? I’m trying to adapt the blog post [here](https://julialang.org/blog/2019/11/artifacts/) but it doesn’t seem like the examples set the URL that the artifacts can be downloaded from.
---
### Author:  [@ssfrr](https://discourse.julialang.org/u/ssfrr)
#### Post date: [January 15, 2020, 4:10am UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/2 "2020-01-15T04:10:08Z")
Ah, it looks like instead of using `Tar.jl` I can use `unpack` from `Pkg.PlatformEngines`.
So my script looks like:
```julia
using Pkg.Artifacts
using Pkg.GitTools
using Pkg.PlatformEngines
toml = joinpath(@ __DIR__ , "Artifacts.toml")
ir_url = "https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
ir_hash = create_artifact() do artifact_dir
tarball = download(ir_url)
@show artifact_dir
try
global tarball_hash = bytes2hex(GitTools.blob_hash(tarball))
unpack(tarball, artifact_dir)
finally
rm(tarball)
end
end
bind_artifact!(toml, "tidmarsh_irs", ir_hash;
download_info=[(ir_url, tarball_hash)],
lazy=true,
force=true)
```
The idea would be that I would run this script if my data ever changed, to update the Artifacts.toml.
@dmbates maybe this helps with the question you asked in the other thread?
---
### Author:  [@dmbates](https://discourse.julialang.org/u/dmbates)
#### Post date: [January 15, 2020, 9:58pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/3 "2020-01-15T21:58:51Z")
Thanks for the suggestion. I am still struggling to understand the section with `create_artifact`, even after reading the documentation and seeing examples. I tried to run this code and was unsuccessful.
```julia
julia> using Pkg.Artifacts
julia> using Pkg.GitTools
julia> using Pkg.PlatformEngines
julia> toml = joinpath(@ __DIR__ , "Artifacts.toml")
"/tmp/Artifacts.toml"
julia> ir_url = "https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
"https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
julia> ir_hash = create_artifact() do artifact_dir
tarball = download(ir_url)
@show artifact_dir
try
global tarball_hash = bytes2hex(GitTools.blob_hash(tarball))
unpack(tarball, artifact_dir)
finally
rm(tarball)
end
end
artifact_dir = "/home/bates/.julia/artifacts/jl_Oj8ETe"
ERROR: MethodError: no method matching (::Pkg.PlatformEngines.var"#3#5")(::String, ::String, ::Nothing)
Closest candidates are:
#3(::AbstractString, ::AbstractString; excludelist) at /home/bates/src/julia-1.3.1/share/julia/stdlib/v1.3/Pkg/src/PlatformEngines.jl:41
Stacktrace:
[1] #unpack#91(::Bool, ::typeof(unpack), ::String, ::String) at /home/bates/src/julia-1.3.1/lib/julia/sys.so:?
[2] unpack at /home/bates/src/julia-1.3.1/share/julia/stdlib/v1.3/Pkg/src/PlatformEngines.jl:723 [inlined]
[3] (::var"#3#4")(::String) at ./REPL[6]:6
[4] create_artifact(::var"#3#4") at /home/bates/src/julia-1.3.1/share/julia/stdlib/v1.3/Pkg/src/Artifacts.jl:213
[5] top-level scope at REPL[6]:1
```
---
### Author:  [@ssfrr](https://discourse.julialang.org/u/ssfrr)
#### Post date: [January 16, 2020, 1:21am UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/4 "2020-01-16T01:21:17Z")
I think you’re seeing a bug. I filed [an issue](https://github.com/JuliaLang/Pkg.jl/issues/1616).
The main point of `create_artifact()` is to compute the hash of the artifact contents. It creates a directory and passes it to the function (`artifact_dir` here). You then fill that directory with whatever you want the contents to be, and after your function returns it will compute the hash. In my script here get the files by downloading a tarball and unpacking into the given directory. I _also_ need to compute the hash of the tarball itself, which I do manually and save it into `tarball_hash`. Then when I call `bind_artifact!` I’m actually writing to the `Artifacts.toml` file. The `download_info` list needs the tarball hash, which I assume it validates after downloading.
Arguably this script uses some things that might be considered internal to Pkg, but I’m not sure if there’s a better way. @staticfloat would you mind weighing in on whether this is the right way to add a tarball as an artifact?
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [January 16, 2020, 2:31am UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/5 "2020-01-16T02:31:31Z")
Tar will eventually replace Pkg.PlatformEnginee, but not yet. Probably in Julia 1.5.
---
### Author:  [@staticfloat](https://discourse.julialang.org/u/staticfloat)
#### Post date: [January 16, 2020, 7:40am UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/6 "2020-01-16T07:40:07Z")
While Spencer did find a small bug, that’s not the root cause here; you need to call `Pkg.PlatformEngines.probe_platform_engines!()` first in order to be able to use things like `download_verify()` or `unpack()`. (`Pkg` does this automatically, which may be why Spencer didn’t hit this error).
You look like you’re on the right track Douglas; just call that `probe_platform_engines!()` first so that `Pkg` can look around a bit and figure out what executables to use to do things like extract tar files, and you should be fine.
---
### Author:  [@ssfrr](https://discourse.julialang.org/u/ssfrr)
#### Post date: [January 16, 2020, 1:03pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/7 "2020-01-16T13:03:07Z")
Great, thanks! I’ll add that to my script.
---
### Author:  [@dmbates](https://discourse.julialang.org/u/dmbates)
#### Post date: [January 16, 2020, 4:14pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/8 "2020-01-16T16:14:33Z")
Thanks for checking this @staticfloat and @ssfrr. This code snippet works for me if I call `probe_platform_engines!` or if I use julia-nightly.
And thanks for your explanation @ssfrr. I am beginning to see the light.
---
### Author:  [@dmbates](https://discourse.julialang.org/u/dmbates)
#### Post date: [January 22, 2020, 6:16pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/9 "2020-01-22T18:16:46Z")
I have created an `Artifacts.toml` for the [`MixedModels`](https://github.com/JuliaStats/MixedModels.jl) package but have been unable to get it to pass travis tests. Apparently it is failing to download a file from `https://figshare.com/`. See the discussion in [Switch to Feather/Artifacts for test data by dmbates · Pull Request #238 · JuliaStats/MixedModels.jl · GitHub](https://github.com/JuliaStats/MixedModels.jl/pull/238)
I put the archive on figshare as suggested in the `DataDeps` package documentation by @oxinabox.
Any suggestions on a cleaner way of accomplishing this?
---
### Author:  [@CiaranOMara](https://discourse.julialang.org/u/CiaranOMara)
#### Post date: [February 2, 2020, 1:33pm UTC](https://discourse.julialang.org/t/creating-artifacts-toml-for-existing-tarball/33365/10 "2020-02-02T13:33:59Z")
@dmbates, the Artifact framework doesn’t know how to process the download because the figshare URL doesn’t include a file extension. You’ll either need a more explicit URL or manually process the artifact entry on **init**. You may find something in [GitHub - CiaranOMara/ArtifactHelpers.jl: Bind and initialise reproducible Artifacts](https://github.com/CiaranOMara/ArtifactHelpers.jl) that is helpful/useful.