# Best Julian way to copy many files?

**URL:** https://discourse.julialang.org/t/best-julian-way-to-copy-many-files/75142
**Category:** Performance
**Tags:** filesystem, copy
**Created:** [January 24, 2022, 6:28pm UTC](https://discourse.julialang.org/t/best-julian-way-to-copy-many-files/75142 "2022-01-24T18:28:37Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![chowerth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chowerth/32/30406_2.png) [@chowerth](https://discourse.julialang.org/u/chowerth)
#### Post date: [January 24, 2022, 6:28pm UTC](https://discourse.julialang.org/t/best-julian-way-to-copy-many-files/75142/1 "2022-01-24T18:28:37Z")

</div>

Hello.

I’m wondering what is the best Julian way to copy files. My particular use case is I have a large number of files (of varying size) and I’m automating their data cleaning and backup process. Without going into great detail, it’s the writing/copying in the backup process that is the bottleneck. Doing a normal broadcasting copy in Julia yields inferior results compared to a manual “copy & paste” in Windows so I’m looking for alternate solutions.

Manual select-all copy takes x time.

Broadcasting Julia’s `cp` takes ~5x as long as manual: `cp.(source_files, backup_files)`

Julia’s `asyncmap` takes ~4x as long as manual: `asyncmap(cp, source_files, backup_files)`

Julia’s `Threads.@spawn` macro takes ~1.5x as long as manual: `paired_files = zip(source_files, backup_files); tasks = [Threads.@spawn(cp(file[1], file[2])) for file in paired_files]`

I will say running `length(Sys.cpu_info())` gives 4 which is how many threads I run this script with so I’d assume it’s “even” with any multi-threading the copy & paste manual version does.

**Any suggestions for efficiently writing/copying multiple files quickly?** I’m happy with the `Threads.@spawn` approach performance but would like to see others’ ideas. Thanks.

---

<div class="post-metadata">

### Author: ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [January 24, 2022, 7:16pm UTC](https://discourse.julialang.org/t/best-julian-way-to-copy-many-files/75142/2 "2022-01-24T19:16:19Z")

</div>

You can try [GitHub - shashi/FileTrees.jl: Parallel file processing made easy](https://github.com/shashi/FileTrees.jl)

---

<div class="post-metadata">

### Author: ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)
#### Post date: [January 24, 2022, 7:24pm UTC](https://discourse.julialang.org/t/best-julian-way-to-copy-many-files/75142/3 "2022-01-24T19:24:00Z")

</div>

You may also want to consider Windows’ built-in [`robocopy`](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy) tool, which allows for over-network compression, restartable transfers, multithreading, incremental file transfer for backups, and other goodies.
