I’m trying to figure out how to create an Artifacts.toml to download some data that my package depends on. I’m creating a setup_artifacts.jl script to create it.
Here’s what I have so far:
using Pkg.Artifacts
using Pkg.GitTools
using Tar
toml = joinpath(@__DIR__, "Artifacts.toml")
ir_url = "https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
ir_hash = create_artifact() do artifact_dir
tarball = download(ir_url)
try
global tarball_hash = bytes2hex(GitTools.blob_hash(tarball))
Tar.extract(tarball, artifact_dir)
finally
rm(tarball)
end
end
bind_artifact!(toml, "tidmarsh_irs", ir_hash;
download_info=[(ir_url, tarball_hash)],
lazy=true)
My goal here is to have an Artifacts.toml file that I can include with my package that will download the referenced file and unpack it, making the contents (a bunch of .flac files in my case) available at runtime.
Currently Tar.extract is choking on the tarball (“invalid octal digit: G”), perhaps because it’s compressed?
Also - is this the right way to be doing this? I’m trying to adapt the blog post here but it doesn’t seem like the examples set the URL that the artifacts can be downloaded from.
Thanks for the suggestion. I am still struggling to understand the section with create_artifact, even after reading the documentation and seeing examples. I tried to run this code and was unsuccessful.
julia> using Pkg.Artifacts
julia> using Pkg.GitTools
julia> using Pkg.PlatformEngines
julia> toml = joinpath(@__DIR__, "Artifacts.toml")
"/tmp/Artifacts.toml"
julia> ir_url = "https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
"https://tidmarsh.media.mit.edu/~sfr/tidmarsh_irs.tar.gz"
julia> ir_hash = create_artifact() do artifact_dir
tarball = download(ir_url)
@show artifact_dir
try
global tarball_hash = bytes2hex(GitTools.blob_hash(tarball))
unpack(tarball, artifact_dir)
finally
rm(tarball)
end
end
artifact_dir = "/home/bates/.julia/artifacts/jl_Oj8ETe"
ERROR: MethodError: no method matching (::Pkg.PlatformEngines.var"#3#5")(::String, ::String, ::Nothing)
Closest candidates are:
#3(::AbstractString, ::AbstractString; excludelist) at /home/bates/src/julia-1.3.1/share/julia/stdlib/v1.3/Pkg/src/PlatformEngines.jl:41
Stacktrace:
[1] #unpack#91(::Bool, ::typeof(unpack), ::String, ::String) at /home/bates/src/julia-1.3.1/lib/julia/sys.so:?
[2] unpack at /home/bates/src/julia-1.3.1/share/julia/stdlib/v1.3/Pkg/src/PlatformEngines.jl:723 [inlined]
[3] (::var"#3#4")(::String) at ./REPL[6]:6
[4] create_artifact(::var"#3#4") at /home/bates/src/julia-1.3.1/share/julia/stdlib/v1.3/Pkg/src/Artifacts.jl:213
[5] top-level scope at REPL[6]:1
The main point of create_artifact() is to compute the hash of the artifact contents. It creates a directory and passes it to the function (artifact_dir here). You then fill that directory with whatever you want the contents to be, and after your function returns it will compute the hash. In my script here get the files by downloading a tarball and unpacking into the given directory. I also need to compute the hash of the tarball itself, which I do manually and save it into tarball_hash. Then when I call bind_artifact! I’m actually writing to the Artifacts.toml file. The download_info list needs the tarball hash, which I assume it validates after downloading.
Arguably this script uses some things that might be considered internal to Pkg, but I’m not sure if there’s a better way. @staticfloat would you mind weighing in on whether this is the right way to add a tarball as an artifact?
While Spencer did find a small bug, that’s not the root cause here; you need to call Pkg.PlatformEngines.probe_platform_engines!() first in order to be able to use things like download_verify() or unpack(). (Pkg does this automatically, which may be why Spencer didn’t hit this error).
You look like you’re on the right track Douglas; just call that probe_platform_engines!() first so that Pkg can look around a bit and figure out what executables to use to do things like extract tar files, and you should be fine.