# \[ANN\] FileTrees.jl -- easy everyday parallelism on trees of files

**URL:** <https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891>\
**Category:** Package Announcements\
**Tags:** parallel, data, multithreading, distributed, filesystem\
**Created:** [August 13, 2020, 5:10pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891 "2020-08-13T17:10:43Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [August 15, 2020, 12:06pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/6 "2020-08-15T12:06:38Z")

</div>

Very cool! Reading the (excellent) documentation and [https://github.com/shashi/FileTrees.jl/issues/9](https://github.com/shashi/FileTrees.jl/issues/9) (@c42f), I’m thinking a bit of the roles of the various packages to form a nice distributed array ecosystem, and relating it to for instance [https://dask.org/](https://dask.org/).

I’m interested in a related use case, namely large chunked array datasets, commonly used in fields like climate science. In dask there is [`dask.array`](https://docs.dask.org/en/latest/array.html). There are different packages that offer IO for these chunked datasets, e.g. [Zarr.jl](https://github.com/meggart/Zarr.jl), [HDF5.jl](https://github.com/JuliaIO/HDF5.jl/issues/615), [NCDatasets.jl](https://github.com/Alexander-Barth/NCDatasets.jl), [ArchGDAL.jl](https://github.com/yeesian/ArchGDAL.jl). In the case of Zarr.jl these chunks can each be separate files, but often the chunks are internal to the files. To help reason about these chunks within `AbstractArrays` @fabiangans created DiskArrays.jl, see [[ANN] DiskArrays.jl](https://discourse.julialang.org/t/ann-diskarrays-jl/34119), and we’ve been busy integrating this into above mentioned packages.

It would be cool to be able to easily put these data sources into a form where we can use Dagger.jl to apply scheduled operations over chunks, and also keep track of the positions of chunks in the dataset, for operations that go over chunk boundaries. I’m aware there is [`Dagger.DArray`](https://github.com/JuliaParallel/Dagger.jl/blob/69983f85c6c4f4559b6b53ba2463ddb04b29420d/src/array/darray.jl#L94-L117) but not sure of it’s place or future, it is a bit underdocumented. Is Dagger.jl going towards primarily being a scheduler?

---

_[View the full topic](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891)._