# \[ANN\] FileTrees.jl -- easy everyday parallelism on trees of files

**URL:** <https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891>\
**Category:** Package Announcements\
**Tags:** parallel, data, multithreading, distributed, filesystem\
**Created:** [August 13, 2020, 5:10pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891 "2020-08-13T17:10:43Z")\
**Posts on this page:** 15\
**Page:** 2

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [August 20, 2020, 10:42pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/21 "2020-08-20T22:42:55Z")

</div>

That’s really weird, I have no idea what’s going on off the top of my head.

Is there a FileTrees entry in your Manifest.toml file? What does it look like?

---

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [August 20, 2020, 10:45pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/22 "2020-08-20T22:45:16Z")

</div>

> Just as a fair warning(?), I don’t think `halve` is tested outside of my packages yet. But if you don’t mind giving a shot at this, it would be fantastic!

I think first we should make Dagger a consumer of this API. It’d be amazing to try distributed LazyGroupBy on a bunch of files. I think that’s a solid example to try and implement.

cc @piever

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 20, 2020, 11:56pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/23 "2020-08-20T23:56:21Z")

</div>

Yes, out-of-core group-by sounds like a nice application. I think it shouldn’t be too hard to implement `reduce` with Dagger API. I might try this myself at some point.

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [August 21, 2020, 6:47am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/24 "2020-08-21T06:47:11Z")

</div>

Hehe, “forceful” is really the right word to decribe it 🙂

Skimming through the files first might be an option as well although it might end up being to slow. I was also a bit worried that reading the same file from multiple workers would make them block each other, but maybe it doesn’t work that way?

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [August 21, 2020, 6:52am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/25 "2020-08-21T06:52:51Z")

</div>

I had some issues to get distributed working as well. Is there a chance that CSV works because it is in your shared/default environment?

I could get it to work using the method posted here: [Packages and workers - #10 by ksmcreynolds](https://discourse.julialang.org/t/packages-and-workers/14072/10)

I think that the flags you used should be equivalent but I don’t know enough about it to say for sure.

---

<div class="post-metadata">

**Author:** ![jonalm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonalm/32/16038_2.png) [@jonalm](https://discourse.julialang.org/u/jonalm)\
**Post date:** [August 21, 2020, 7:43am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/26 "2020-08-21T07:43:01Z")

</div>

Thanks @DrChainsaw! Your’e right.

---

<div class="post-metadata">

**Author:** ![jonalm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonalm/32/16038_2.png) [@jonalm](https://discourse.julialang.org/u/jonalm)\
**Post date:** [August 21, 2020, 7:57am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/27 "2020-08-21T07:57:51Z")

</div>

Thanks for the answer @shashi. The problem was that I needed to explicitly activate the environment for all workers. Now my script runs, but I still don’t see a parallelization.

Consider the following script, test.jl

```julia
@everywhere using Pkg
@everywhere Pkg.activate(".")
@everywhere using Distributed, FileTrees, .Threads

@show nthreads()
@show nprocs()

@everywhere function create_tree()
    t = maketree("test_file_tree" => [])
    for c in 'A':'Z'
        node_file = joinpath(string(c), "nodefile")
        t = touch(
            t, 
            node_file,
            value=1
        )
    end
    t
end

t = create_tree()
FileTrees.save(t) do file
    println("pid : $(myid()), threadid : $(threadid()), $(path(file))")
end |> exec

```

Running julia 1.5, FileTrees 0.1.2, with JULIA\_NUM\_THREADS=4, I get

```julia
> julia -p 2 test.jl
 Activating environment at `~testtree/Project.toml`
      From worker 3: Activating environment at `~/testtree/Project.toml`
      From worker 2: Activating environment at `~/testtree/Project.toml`
nthreads() = 4
nprocs() = 3
pid : 1, threadid : 1, test_file_tree/A/nodefile
pid : 1, threadid : 1, test_file_tree/B/nodefile
pid : 1, threadid : 1, test_file_tree/C/nodefile
pid : 1, threadid : 1, test_file_tree/D/nodefile
pid : 1, threadid : 1, test_file_tree/E/nodefile
pid : 1, threadid : 1, test_file_tree/F/nodefile
pid : 1, threadid : 1, test_file_tree/G/nodefile
pid : 1, threadid : 1, test_file_tree/H/nodefile
pid : 1, threadid : 1, test_file_tree/I/nodefile
pid : 1, threadid : 1, test_file_tree/J/nodefile
pid : 1, threadid : 1, test_file_tree/K/nodefile
pid : 1, threadid : 1, test_file_tree/L/nodefile
pid : 1, threadid : 1, test_file_tree/M/nodefile
pid : 1, threadid : 1, test_file_tree/N/nodefile
pid : 1, threadid : 1, test_file_tree/O/nodefile
pid : 1, threadid : 1, test_file_tree/P/nodefile
pid : 1, threadid : 1, test_file_tree/Q/nodefile
pid : 1, threadid : 1, test_file_tree/R/nodefile
pid : 1, threadid : 1, test_file_tree/S/nodefile
pid : 1, threadid : 1, test_file_tree/T/nodefile
pid : 1, threadid : 1, test_file_tree/U/nodefile
pid : 1, threadid : 1, test_file_tree/V/nodefile
pid : 1, threadid : 1, test_file_tree/W/nodefile
pid : 1, threadid : 1, test_file_tree/X/nodefile
pid : 1, threadid : 1, test_file_tree/Y/nodefile
pid : 1, threadid : 1, test_file_tree/Z/nodefile

```

Moreover, if I comment out the `value=1` line in the tree constructor, nothing is printed out. It the save loop seems to skip nodes without values. Is that intentional?

If these are issues with FileTrees I can create issues there.

---

<div class="post-metadata">

**Author:** ![jonalm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonalm/32/16038_2.png) [@jonalm](https://discourse.julialang.org/u/jonalm)\
**Post date:** [August 21, 2020, 8:32am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/28 "2020-08-21T08:32:43Z")

</div>

Ah. I used `save` and not `load` with `lazy=true`. Works now. Sorry for the noise.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 21, 2020, 10:36am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/29 "2020-08-21T10:36:24Z")

</div>

Forgive me for saying this. FileTrees looks great. It assumes a POSIX compatible filesystem (I think). Systems admins like me always howl when people read and write many small files - I suppose we should really suck it up and start to engineer for it.

Is there any though it Julia for using S3 object stores?  
I guess that would be a completely separate module from FileTrees.jl

---

<div class="post-metadata">

**Author:** ![dmolina](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmolina/32/5246_2.png) [@dmolina](https://discourse.julialang.org/u/dmolina)\
**Post date:** [August 21, 2020, 11:46am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/30 "2020-08-21T11:46:03Z")

</div>

Thank you, @shashi. This is a great and useful tool.

I write a not very small example, because I have a doubt.  
I have been working using it with some ML preprocess in which files has a certain structure “test//”

```julia
    # All files
    all = FileTree(dataset_dir)

    # Apply the function apply to each file
    data = FileTrees.load(all; lazy=true) do file
        # Get name
        fname_str = convert(String, path(file))
        # Apply the model to the image
        apply(fname_str)
    end

    # Recover the categories
    categories = FileTrees.load(all; lazy=true) do file
        # Get name
        category = path(file).segments[end-1]
        category
    end

    for type in ["test", "train"]
        # I love that part, that you can so easily filter train and test files
        sel = GlobMatch("$type/*/*.$ext")
        values = reducevalues(hcat, data[sel]) |> exec
        cats = reducevalues(hcat, categories[sel]) |> exec
        writedlm(..., values)
        writedlm(..., cats)
    end

```

It is right? For me it is working well, but I do not know if the sort  
of categories and data are right even in parallel way. It could be run in parallel without problem.

---

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [August 21, 2020, 3:54pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/31 "2020-08-21T15:54:52Z")

</div>

@tkf nice! I’d be down to pair program on that, it might be much quicker to do it together than our own.

@johnh  
I’m looking for users on all platforms! If you give it a go, let me know.

Right now we actually don’t tie into POSIX, by means of FilePathsBase.jl … I think the idea is depending on which platform you’re on, the path will be of a different type. But since I don’t have a good use case to test it on, [prefixes · Issue #8 · shashi/FileTrees.jl · GitHub](https://github.com/shashi/FileTrees.jl/issues/8)

I think at least we would need to change how `path` works. [https://github.com/shashi/FileTrees.jl/blob/master/src/datastructure.jl#L288](https://github.com/shashi/FileTrees.jl/blob/master/src/datastructure.jl#L288)

possibly `path(parent(d)) / Path(d.name)` should become `path(parent(d)) / d.name` and we store the right path type in the root node as stated in issue #8.

I’m actually not sure if there’s an AbstractPath implementation for S3 that is open source and based on FilePaths.jl cc @RoyiAvital (sorry for pinging 🙂 )

@jonalm Thanks! That makes sense! We should definitely document that.

> skip nodes without values. Is that intentional?

That’s right! But should I’m open to doing the “right thing”. Please do open an issue where we can discuss.

@dmolina that looks good!

You can also try `name(parent(file))` to get the category instead of reaching into the path segments…

I think `lazy` in the second case of getting the categories is overkill, I’d just skip that. (because you don’t need to run that in parallel).

---

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [August 21, 2020, 3:56pm UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/32 "2020-08-21T15:56:24Z")

</div>

I don’t think reading the same file will block workers. That would require a lockfile 😉

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 22, 2020, 2:43am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/33 "2020-08-22T02:43:28Z")

</div>

@shashi Here is a super quick hack at it to implement Transducers.jl API (including FLoops.jl) on top of Dagger.jl: [https://github.com/JuliaFolds/DaggerFolds.jl](https://github.com/JuliaFolds/DaggerFolds.jl). I have no idea if it is practically usable though.

Also, I realized implementing this and avoiding boilerplates requires [importing a bunch of internals from Transducers.jl](https://github.com/JuliaFolds/DaggerFolds.jl/blob/42002ad4ae914898221880f61fdbcb7731c056f8/src/DaggerFolds.jl#L11-L13). I need to think about a better interface to do this.

---

<div class="post-metadata">

**Author:** ![shashi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/shashi/32/1824_2.png) [@shashi](https://discourse.julialang.org/u/shashi)\
**Post date:** [August 22, 2020, 5:55am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/34 "2020-08-22T05:55:16Z")

</div>

DaggerFolds looks great!! I see some comments there `# Not sure how to cancel delayed computation.`

I will take a closer look over the weekend.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 22, 2020, 6:03am UTC](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891/35 "2020-08-22T06:03:45Z")

</div>

Thanks! BTW, if you don’t mind going down the big rabbit hole of cancellation, I recommend checking out [https://github.com/JuliaLang/julia/issues/33248](https://github.com/JuliaLang/julia/issues/33248) 🙂

[Previous page](https://discourse.julialang.org/t/ann-filetrees-jl-easy-everyday-parallelism-on-trees-of-files/44891.md?page=1)