# ANN: Tar.jl
**URL:** https://discourse.julialang.org/t/ann-tar-jl/32527
**Category:** Package Announcements
**Created:** [December 20, 2019, 7:52pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527 "2019-12-20T19:52:09Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 20, 2019, 7:52pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/1 "2019-12-20T19:52:09Z")
I’d like to announce [Tar.jl](https://github.com/JuliaLang/Tar.jl), a pure Julia package for reading and writing POSIX TAR files for the purpose of transmitting trees of files and directories from one system to another. The README should hopefully explain the design and usage clearly. If you need to record and send file trees between systems, please give it a try!
---
### Author:  [@anon92994695](https://discourse.julialang.org/u/anon92994695)
#### Post date: [December 20, 2019, 8:45pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/2 "2019-12-20T20:45:25Z")
Ooo this is something to celebrate!
---
### Author:  [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [December 29, 2019, 8:11am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/3 "2019-12-29T08:11:10Z")
Could you add examples how to use it in readme?
How could one do for example:
```julia
# transfer /home/julia to dest_machine
cd /home; tar -czpf - julia | ssh dest_machine 'cd /home; tar -xzpvf -'
```
I also miss motivation part. Is it about doing standard things at least on primitive level (for example without restoring timestamps) in Windows?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 29, 2019, 8:33am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/4 "2019-12-29T08:33:39Z")
If I understand correctly, permissions (`-p`) would not be supported.
Did you read the motivation part in the readme? I think it describes the use case extensively (intersystem file transfer).
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 29, 2019, 11:09am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/5 "2019-12-29T11:09:32Z")
You would do something like this:
```julia
using Tar
tarball = Tar.create("/home/julia")
# compress by your favorite means
# send to destination system somehow
```
On the destination system:
```julia
using Tar
# decompress tarball first
Tar.extract(tarball, "/home/julia")
```
The readme is mostly motivation, so I’m a bit unclear on what’s missing there.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [December 31, 2019, 2:01am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/6 "2019-12-31T02:01:29Z")
It’s very nice indeed! Just curious - what’s the motivation for creating this package? Is GNU tar not nice enough? 😄
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [December 31, 2019, 6:48am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/7 "2019-12-31T06:48:01Z")
> **[GitHub - JuliaIO/Tar.jl: TAR files: create, list, extract them in pure Julia](https://github.com/JuliaIO/Tar.jl#design--features)**
>
> TAR files: create, list, extract them in pure Julia - GitHub - JuliaIO/Tar.jl: TAR files: create, list, extract them in pure Julia
it’s stated in the README
---
### Author:  [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [December 31, 2019, 4:02pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/8 "2019-12-31T16:02:01Z")
I showed one-liner which don’t need additional space on hardisks for tarball and could tar + pack and unpack + untar parallely. It also preserve timestamps and permissions. Which I think is standard clever way how to do file tree transfer and implement steps (you have “somehow” in comments) in elegant way.
I read the readme and **didn’t find any good reason not to prefer proved old solution**.
I was also curious if there is some clever simple (oneliner or twoliner) way how to use `tarball::IO` to write tarball through network and read it (and extract it) on destination machine. Something similarly elegant to posix solution I showed.
One possible reason I saw was for people who need to use inferior OSes without ssh and tar utilities.
Second possible reason could be performance gain. Did you do performance comparison? Is there some benefit?
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 31, 2019, 4:09pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/9 "2019-12-31T16:09:09Z")
Feel free to continue using command-line `tar`. It’s quite possible to use Tar.jl to send data directly over a network connection (open a socket/http/whatever and pass the I/O handle to `create`), but based on your stance here, is suspect this package isn’t for you.
---
### Author:  [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [December 31, 2019, 4:11pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/10 "2019-12-31T16:11:50Z")
If you mean this:
> Unlike the `tar` command line tool, which was originally designed to archive data in order to restore it back to the same system or to a replica thereof, the `Tar` package is designed for using the TAR format to transfer trees of files and directories from one system to another.
it seems like missunderstanding of unix philosophy where (simplified -\>) everything is file.
As I show that philosophy bring elegant solution how to interconnect processes and through pipes send data/tarball to destination machine where another processes would unpack, extract it without using additional space for tarball. I don’t think it is something which was not anticipated in time when tar utility was created! Pipe “files” between processes is very old (and very powerful) basic paradigm behind unix/posix ecosystem.
---
### Author:  [@tk3369](https://discourse.julialang.org/u/tk3369)
#### Post date: [December 31, 2019, 4:49pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/11 "2019-12-31T16:49:17Z")
Is there a practical use case that you can share? Maybe you can describe how this package helps?
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 31, 2019, 5:09pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/12 "2019-12-31T17:09:25Z")
GNU tar isn’t available everywhere, eg Windows, macOS, or FreeBSD. On Windows it’s common for no `tar` command to be present at all. BinaryBuilder + artifacts makes it possible to install a consistent `tar` binary on any system, but there’s a bootstrapping problem: artifacts are transmitted as tarballs, what do you use to unpack the artifact tarball containing the `tar` binary? This bootstrapping problem will be exacerbated by the [Pkg protocol](https://github.com/JuliaLang/Pkg.jl/issues/1377), which will be opt-in in Julia 1.4 (by doing `export JULIA_PKG_SERVER=pkg.julialang.org` in your shell) and opt-out in Julia 1.5: with this protocol, all resources are transferred as tarballs, including registries, packages and artifacts, so having a reliable, portable way to create and extract tarballs becomes rather important.
Even when you have a reasonable `tar` program, it’s remarkably hard to coax it to behave in a way that is “right” for the data transfer use case. For example, this is the line to unpack data correctly in Pkg’s PlatformEngines:
[https://github.com/JuliaLang/Pkg.jl/blob/44f0a2ddde225c/src/PlatformEngines.jl#L333](https://github.com/JuliaLang/Pkg.jl/blob/44f0a2ddde225c/src/PlatformEngines.jl#L333)
All of the surrounding code is just to try to figure out how to invoke some kind of tar-equivalent program in a cross-platform way. And on any new or slightly different system, it’s likely to break. We’re very eager to get rid of all of that code and replace it with portable, pure-Julia implementations that we know will work anywhere that Julia works. There are three pieces we need to get there:
1. Ability to download over HTTP/S — provided by [HTTP.jl](https://github.com/JuliaWeb/HTTP.jl).
2. Ability to decompress files — potentially provided for .gz files by [Inflate.jl](https://github.com/GunnarFarneback/Inflate.jl), although we may want to use bzip2 instead since it has better compression for the kinds of files we’re sending.
3. Ability to unpack tarballs — provided by Tar.jl
So Tar.jl is part of the plan to replace the nasty and brittle PlatformEngines code with pure Julia code for downloading and unpacking tarballs.
There’s one last, fairly non-obvious reason for Tar.jl to exist: to be able to compute and apply diffs to file trees. Part of the Pkg protocol plan linked above is to be able to send diffs from the previous state of a registry, package or artifact instead of having to send the whole thing. The [bsdiff](https://github.com/mendsley/bsdiff) program is pretty good for computing and sending very small binary diffs. However, bsdiff only computes a diff for a single file. How do you compute diffs of trees of files? One simple approach is to generate uncompressed tarballs of the before and after trees and then compute and send a diff of those two tarballs. However, the client then needs to be able to reproduce the exact same before tarball from an on-disk file tree in order to apply the diff. If the server and client have `tar` programs that behave differently, these tarballs might be quite different. Even different versions of GNU tar might produce very different tarballs. Enter Tar.jl: it guarantees that if two file trees with the same content are turned into tarballs, those tarballs will be bit-for-bit identical. So if the server and the client are both using Tar.jl to generate tarballs, we can compute and apply tarball diffs of trees, which will allow us to efficiently update registries, packages and artifacts.
As a side benefit, this means that one more step of the Pkg resource chain is perfectly reproducible: given a particular resource version (registry, package, artifact), identified by its content hash, it will always produce the same uncompressed tarball. If we standardize the compression as well, then we will have perfect reproducibility of the final tarball. If, in addition, we start doing reproducible builds in BinaryBuilder, then the whole build chain will be reproducible from end to end: a build will always produce the same final tarball.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 31, 2019, 5:23pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/13 "2019-12-31T17:23:01Z")
Again, if you are not convinced to use this, feel free not to.
If you did want to send a tarball over any kind of IO object, here’s how you would do it:
```julia
io = # connect to other end any way you want
Tar.create(source, io)
```
This streams the tarball to `io` directly without writing it to disk. On the receiving side:
```julia
io = # the other side of that connection
Tar.extract(io, destination)
```
This extracts the tarball from `io` without writing it to disk. The `io` object could, for example, be a [TranscodingStream](https://bicycle1885.github.io/TranscodingStreams.jl/latest/) wrapping an HTTPS connection.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 31, 2019, 5:31pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/14 "2019-12-31T17:31:50Z")
Regarding the invocation of `tar` in PlatformEngines, in just the past month, we have realized that we needed to add the `-m` flag because without it, systems with clock skew were complaining that tarballs were being extracted with time stamps in the future. Then we realized that we needed the `--no-same-owner` flag because when Julia was running as root, `tar` was trying to change the owners/groups of the contents to whatever it was on the origin system, which was generally total nonsense on the new system, and causing all kinds of problems. Are we done adding flags to `tar` to get it to behave the way we want? Hopefully 😬 Do these flags work with non-GNU `tar` commands? Maybe. The defaults of command-line `tar` programs are just totally wrong for just sending a tree of files to another unrelated system.
And PlatformEngines is the worst—just read that file I linked, it’s such a nasty rat’s nest. I’m looking forward to getting rid of it so much.
---
### Author:  [@mbauman](https://discourse.julialang.org/u/mbauman)
#### Post date: [December 31, 2019, 5:36pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/15 "2019-12-31T17:36:10Z")
> [@Liso](#):
>
> I read the readme and **didn’t find any good reason not to prefer proved old solution**.
It’s perhaps worth explicitly noting that Tar.jl **is** using [a proven standard for tar files](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html)… and files created by Tar.jl can be opened by other POSIX compliant `tar` implementations. It’s just a minimal implementation (in less than 1k loc) that solves some specific pain points and provides some nice advantages as Stefan describes above.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [December 31, 2019, 5:48pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/16 "2019-12-31T17:48:15Z")
The API is also quite a bit nicer for programmatic use than calling `tar` from Julia.
---
### Author:  [@anon92994695](https://discourse.julialang.org/u/anon92994695)
#### Post date: [January 17, 2020, 11:27pm UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/17 "2020-01-17T23:27:30Z")
All I know is I used this on a windows VM and it worked perfectly for my needs(made a zip). Was far easier to grok then ZipFiles (no offense ZipFiles!) and no blemishes. More packages of this quality/utility please!
---
### Author:  [@CiaranOMara](https://discourse.julialang.org/u/CiaranOMara)
#### Post date: [February 1, 2020, 1:44am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/18 "2020-02-01T01:44:42Z")
> [@StefanKarpinski](#):
>
> And PlatformEngines is the worst—just read that file I linked, it’s such a nasty rat’s nest. I’m looking forward to getting rid of it so much.
Having read that file is the very reason I came across this thread. I wanted to know whether the pieces needed to bootstrap the unpacking of variously compressed Artifacts with a pure-Julia implementation were available. It seems to me that once the BinaryWrappers can be acquired and unpacked, it will then be possible to use those wrapped binaries to unpack other artifacts compressed with different compression types.
---
### Author:  [@Laco\_Kovac](https://discourse.julialang.org/u/Laco_Kovac)
#### Post date: [April 23, 2021, 10:15am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/19 "2021-04-23T10:15:38Z")
I am trying to understand how to use this package.
I created locally a directory C:\x\a with big text file, size 573654 kB.
In Julia, I call:
```julia
julia> tarball = Tar.create("C:\\x\\a")
"C:\\Users\\userid\\AppData\\Local\\Temp\\jl_A31C.tmp"
```
I checked the temp directory, and jl\_A31C.tmp is there, but its size is exactly the size of the text file in input directory: 573654 kB.
And as soon as I exit Julia, that file disappears.
Am I doing something wrong?
Thank you.
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [April 23, 2021, 10:22am UTC](https://discourse.julialang.org/t/ann-tar-jl/32527/20 "2021-04-23T10:22:34Z")
> [@Laco\_Kovac](#):
>
> I checked the temp directory, and jl\_A31C.tmp is there, but its size is exactly the size of the text file in input directory: 573654 kB.
A tarball doesn’t compress by itself, if that’s what you’re asking, the main use of a tarball is to bundle multiple files together in a single file. If you’re after compression, see [https://github.com/JuliaIO/Tar.jl/blob/ada4f12589db336c8e65405c5bb1a5c3463d1f8d/README.md#compression](https://github.com/JuliaIO/Tar.jl/blob/ada4f12589db336c8e65405c5bb1a5c3463d1f8d/README.md#compression)
> [@Laco\_Kovac](#):
>
> And as soon as I exit Julia, that file disappears.
Maybe because the file was created in a temporary directory cleaned up at the end of the Julia session. Specify the path of the output tarball as second argument of `Tar.create`
[Next page](https://discourse.julialang.org/t/ann-tar-jl/32527.md?page=2)