# \[ANN\] FileTrees.jl -- easy everyday parallelism on trees of files

**URL:** <https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891>\
**Category:** Package Announcements\
**Tags:** parallel, data, multithreading, distributed, filesystem\
**Created:** [August 13, 2020, 5:10pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891 "2020-08-13T17:10:43Z")\
**Posts on this page:** 1\
**Showing post:** 17

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 20, 2020, 6:14am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/17 "2020-08-20T06:14:43Z")

</div>

Hi, thanks for your quick intro to Dagger.jl! I’ve been wanting to look into it. I think now I can see that how it is out-of-core friendly.

BTW, I think it was a bit of exaggeration when I said “scheduler”. It was just a mechanism to hook different implementation of `reduce` (sequential, threaded, distributed, unordered variant of them, etc.) into the `for` loop syntax. So it’s not a scheduler in the sense of e.g., partr.

> [@shashi](#):
>
> For FileTrees, do you recommend implementing a `halve` method which would give a tree with approximately half of the files?

Just as a fair warning(?), I don’t think `halve` is tested outside of my packages yet. But if you don’t mind giving a shot at this, it would be fantastic!

If you have `halve` + `iterate` (or `halve` + ` __foldl__ `) then it should work well with Transducers.jl and all of its [related packages](https://github.com/JuliaFolds/Transducers.jl#related-packages) (e.g., [ThreadsX.jl](https://github.com/tkf/ThreadsX.jl), [FLoops.jl](https://github.com/JuliaFolds/FLoops.jl), [LazyGroupBy.jl](https://github.com/JuliaFolds/LazyGroupBy.jl), …). I’m not sure if it provides nice out-of-core facility ATM though. I just don’t have enough experience with mixing it with a bunch of I/O (and I know there are several possible improvements for this). But if you have a bit smaller scale problem, threading based `reduce` could be nice to have?

---

_[View the full topic](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891)._