# Using DataDeps for multiple files in multiple subdirectories

**URL:** <https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307>\
**Category:** General Usage\
**Created:** [December 6, 2022, 3:14pm UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307 "2022-12-06T15:14:18Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![wsshin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsshin/32/360_2.png) [@wsshin](https://discourse.julialang.org/u/wsshin)\
**Post date:** [December 6, 2022, 3:14pm UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/1 "2022-12-06T15:14:18Z")

</div>

For my package, I want to download some data upon the user’s request and store them locally for future usage. A bit of research seems to indicate that DataDeps.jl by @oxinabox is the package to use, but I’m not sure if it can be used to solve my specific problem described below.

There is a website at `https://website.com`, and multiple data files are stored at

- `https://website.com/data/subA/A1.csv`
- `https://website.com/data/subA/A2.csv`
- …

and

- `https://website.com/data/subB/B1.csv`
- `https://website.com/data/subB/B2.csv`
- …

etc. The shared portion of all these paths is `https://website.com/data`. I would like to register this part as a `DataDep` and let the users to download and store the individual files in the individual subdirectories.

I hoped that some thing like

```julia
register(DataDep("Website.COM Data", "Data published in website.com", "https://website.com/data"))

```

and subsequent calls of

```julia
data_A1 = read(datadep"Website.COM Data" * "/subA/A1.csv")
data_A2 = read(datadep"Website.COM Data" * "/subA/A2.csv")
data_B1 = read(datadep"Website.COM Data" * "/subB/B1.csv")
data_B2 = read(datadep"Website.COM Data" * "/subB/B2.csv")

```

would download the files into `~/.julia/datadeps/Website.COM Data/`, but it didn’t work.

Is there a way to achieve the goal described above using DataDeps or any other packages?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [December 7, 2022, 7:46pm UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/2 "2022-12-07T19:46:24Z")

</div>

Correct DataDeps needs a full list of files to fetch.  
You need to list them all in the `register` block.  
(DataDepsGenerators.jl can help with this, some of the time)

[RemoteFiles.jl](https://github.com/helgee/RemoteFiles.jl)  
might be a suitable alternative that works better for this usecase.  
I am not sure, I haven’t tried it

---

<div class="post-metadata">

**Author:** ![wsshin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsshin/32/360_2.png) [@wsshin](https://discourse.julialang.org/u/wsshin)\
**Post date:** [December 13, 2022, 3:21am UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/3 "2022-12-13T03:21:37Z")

</div>

@oxinabox, thanks for your answer!

Could you explain how to register `DataDep` such that subfolders are created as mentioned in [this documentation](https://www.oxinabox.net/DataDeps.jl/v0.5.3/z20-for-pkg-devs.html#Using-a-datadep-string-or-resolve-to-get-hold-of-the-data.-1)? In the documentation, I was not able to find an example describing the method to create subfolders. If I know how to create subfolders, I might be able to devise a method to achieve what I want.

I tried `RemoteFiles`, but it doesn’t seem to support creation of subfolders in the default location (the root directory of the package using `RemoteFiles`).

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [December 14, 2022, 3:21pm UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/4 "2022-12-14T15:21:29Z")

</div>

Here is an example

```julia
register(DataDep(
    "Pi3",
    "Some message",
    [
        "https://www.angio.net/pi/digits/10.txt",
        "https://www.angio.net/pi/digits/100.txt",
        [
           "https://www.angio.net/pi/digits/1000.txt",
           "https://www.angio.net/pi/digits/10000.txt",
           "https://www.angio.net/pi/digits/100000.txt"
        ]
    ],
    sha2_256,
    post_fetch_method = [
        # 1st applies to 1st file, i.e 10.txt
        filename -> mv(filename, joinpath(mkpath("ten"), basename(filename))),
        # 2nd applies to 2nd listed file, i.e 100.txt
        filename -> mv(filename, joinpath(mkpath("hundred"), basename(filename))),
        # Applies to all things in 3rd (the inner vector) ie. 1000.txt, 10000.txt, and 100000.txt)
        # alt could have written a vector of 3 function here to treat those differently
        filename -> mv(filename, joinpath(mkpath("lots"), basename(filename))),
    ]
))

readdir(datadep"Pi3")
readdir(datadep"Pi3/ten")
readdir(datadep"Pi3/lots")

```

Output at end is

```julia
julia> readdir(datadep"Pi3")
3-element Vector{String}:
 "hundred"
 "lots"
 "ten"

julia> readdir(datadep"Pi3/ten")
1-element Vector{String}:
 "10.txt"

julia> readdir(datadep"Pi3/lots")
3-element Vector{String}:
 "1000.txt"
 "10000.txt"
 "100000.txt"

```

In `post_fetch_method` you can run whatever code you like to derive the subfolder name from the filename. But the filename won’t have the subfolder embedded in it – blame [RFC 6266](https://www.rfc-editor.org/rfc/rfc6266) I guess.

---

<div class="post-metadata">

**Author:** ![wsshin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsshin/32/360_2.png) [@wsshin](https://discourse.julialang.org/u/wsshin)\
**Post date:** [December 14, 2022, 6:56pm UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/5 "2022-12-14T18:56:03Z")

</div>

@oxinabox, thanks! This is closer to what I am trying to do.

I have one more question. When you have many files listed in one `DataDep`, it seems that reading one file from the `DataDep` downloads all the listed files. Is there a way to make it download only one file if that is the only file the user requests?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [December 15, 2022, 1:10pm UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/6 "2022-12-15T13:10:06Z")

</div>

There is not

---

<div class="post-metadata">

**Author:** ![wsshin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsshin/32/360_2.png) [@wsshin](https://discourse.julialang.org/u/wsshin)\
**Post date:** [December 16, 2022, 12:56am UTC](https://discourse.julialang.org/t/using-datadeps-for-multiple-files-in-multiple-subdirectories/91307/7 "2022-12-16T00:56:11Z")

</div>

Thank you for the confirmation! I think I can live with the situation.