`add_artifact!` fails because of a checksum test, while the tarball was made with `Tar.jl`

General Usage
,
1

Hello, I’m trying to add some binary artifacts to one of my packages.
As I am on windows, I do not use BinaryBuilder as it does not seem supported at the moment for that OS. However, I tried using ArtifactUtils to ease up the process.

Here are the steps I followed :

  1. I created a tarball of the folder that I want to use as an artifact with
Tar.create("<path to folder>","<myartifactname>.tar.gz")
  1. I sent it in a shared folder on a NAS server in my private network
cp("<myartifactname>.tar.gz","<myartifact path>.tar.gz")
  1. I attempted to add the artifact to my package, here is where it fails for now
using ArtifactUtils, Pkg.Artifacts, Base.BinaryPlatforms

add_artifact!(
    "Artifacts.toml",
    "<myartifactname>",
    "file::////<myartifact path>.tar.gz", # becasuse `add_artifact` wants URLs
    platform = Platform("x86_64","windows")
)

At that point I receive several errors

ERRORS:
Unexpected end of archive
ERROR: The communication channel is about to be closed (in the system regional language)
ERROR: This does not appear to be a TAR file/stream — malformed chksum field: “\xb9\x01\xc1\xf8\xa3j\xc4\xf9”. Note: Tar.jl does not handle decompression; if the tarball is compressed you must use an external command like gzcat or package like CodecZlib.jl to decompress it. See the README file for examples.

However, the tarball involved here was created using Tar.jl, and I can extract it without errors with Tar or with an external tool.

What I can’t figure is why does add_artifact complain about the tarball, while Tar itself does not. It may be straightforward for experienced Artifact users, but it’s my first time attempting that or working with tarballs.

The stacktrace is the following

Stacktrace:
[1] error(s::String)
@ Base .\error.jl:35
[2] |>(x::String, f::typeof(error))
@ Base .\operators.jl:911
[3] header_error(buf::Vector{UInt8}, msg::String)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:555
[4] header_error(buf::Vector{UInt8}, fld::Symbol)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:567
[5] read_header_int(buf::Vector{UInt8}, fld::Symbol)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:658
[6] check_checksum_field(buf::Vector{UInt8})
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:618
[7] read_standard_header(io::Base.Process; buf::Vector{UInt8}, tee::Base.DevNull)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:587
[8] #read_header#50
@ Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:415 [inlined]
[9] read_tarball(callback::Tar.var"#26#28"{Vector{UInt8}, Bool, Bool, Base.Process, String}, predicate::Tar.var"#1#2", tar::Base.Process; buf::Vector{UInt8}, skeleton::Base.DevNull)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:345
[10] extract_tarball(predicate::Function, tar::Base.Process, root::String; buf::Vector{UInt8}, skeleton::Base.DevNull, copy_symlinks::Bool, set_permissions::Bool)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\extract.jl:58
[11] (::Tar.var"#85#88"{String, Base.Process, Bool, Tar.var"#1#2"})(skeleton::Base.DevNull)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\Tar.jl:237
[12] arg_write(f::Tar.var"#85#88"{String, Base.Process, Bool, Tar.var"#1#2"}, arg::Base.DevNull)
@ ArgTools Julia-1.8.0\share\julia\stdlib\v1.8\ArgTools\src\ArgTools.jl:134
[13] #84
@ Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\Tar.jl:236 [inlined]
[14] arg_mkdir(f::Tar.var"#84#87"{Base.Process, Base.DevNull, Bool, Tar.var"#1#2"}, arg::String)
@ ArgTools Julia-1.8.0\share\julia\stdlib\v1.8\ArgTools\src\ArgTools.jl:185
[15] #83
@ Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\Tar.jl:232 [inlined]
[16] open(::Tar.var"#83#86"{Base.DevNull, Bool, Tar.var"#1#2", String}, ::Cmd; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Base .\process.jl:427
[17] open(::Function, ::Cmd)
@ Base .\process.jl:414
[18] arg_read
@ Julia-1.8.0\share\julia\stdlib\v1.8\ArgTools\src\ArgTools.jl:75 [inlined]
[19] extract(predicate::Function, tarball::Cmd, dir::String; skeleton::Nothing, copy_symlinks::Nothing, set_permissions::Bool)
@ Tar Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\Tar.jl:231
[20] #extract#89
@ Julia-1.8.0\share\julia\stdlib\v1.8\Tar\src\Tar.jl:255 [inlined]
[21] unpack(tarball_path::String, dest::String; verbose::Bool)
@ Pkg.PlatformEngines Julia-1.8.0\share\julia\stdlib\v1.8\Pkg\src\PlatformEngines.jl:389
[22] unpack
@ Julia-1.8.0\share\julia\stdlib\v1.8\Pkg\src\PlatformEngines.jl:384 [inlined]
[23] #25
@ .julia\packages\ArtifactUtils\vpjlQ\src\ArtifactUtils.jl:66 [inlined]
[24] create_artifact(f::ArtifactUtils.var"#25#26"{String})
@ Pkg.Artifacts Julia-1.8.0\share\julia\stdlib\v1.8\Pkg\src\Artifacts.jl:45
[25] add_artifact!(artifacts_toml::String, name::String, tarball_url::String; clear::Bool, options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ ArtifactUtils .julia\packages\ArtifactUtils\vpjlQ\src\ArtifactUtils.jl:65
[26] add_artifact!(artifacts_toml::String, name::String, tarball_url::String)
@ ArtifactUtils .julia\packages\ArtifactUtils\vpjlQ\src\ArtifactUtils.jl:52

2

If you look at the Compression section of the Tar.jl Readme:

It is typical to compress tarballs when saving or transferring them. In the UNIX tradition of doing one thing and doing it well, the Tar package does not do any kind of compression and instead makes it easy to compose its API with external compression tools.

Translation: when you call Tar.create above, you’re actually creating just a .tar file, even though you’ve given it a .tar.gz extension. The Readme then goes on to give suggestions about how to create a compressed tarball.

You can adapt the “Creating a tarball with the gzip command” code, to use whichever compression tool you have available on Windows.
Or you can use the TranscodingStreams based idea suggested after that.

The difference is that Tar.extract expects a .tar file i.e. a basic tarball, whereas add_artifact! expects a compressed tarball (even though its documentation doesn’t clearly say this and should be updated).

1 Like
4

The really important part is indeed that

1 Like