# DataDeps post\_fetch\_method

**URL:** <https://discourse.julialang.org/t/datadeps-post-fetch-method/24649>\
**Category:** Data\
**Tags:** question\
**Created:** [May 27, 2019, 12:49pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649 "2019-05-27T12:49:01Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [May 27, 2019, 12:49pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/1 "2019-05-27T12:49:01Z")

</div>

I have a bunch of zipped `.csv` files on an S3 repo. I’m using the excellent [DataDeps](https://github.com/oxinabox/DataDeps.jl) to retrieve them. Next, I use the also excellent [JuliaDB](https://juliadb.org/) to `loadtable` the `.csv`s into tables, and then from there it’s processing etc etc etc.

It occurred to me that I could define a new `post_fetch_method` method for `DataDeps`, that

1. retrieves the zipped file from the S3 repo, unpacks it
2. `loadtable`s the `.csv`s to tables
3. saves those tables in binary format with `JuliaDB.save`
4. and deletes the downloaded zipped file and the inflated `.csv` files.

But all the inflated files and everything else, including the folder in `~/.julia/datadeps/` gets deleted…

Anyone with some bright ideas?

BTW, the reason I don’t keep the information in binary from the get-go and insist on having it online as a `.csv` text file is just for compatibility.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [May 27, 2019, 1:01pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/2 "2019-05-27T13:01:22Z")

</div>

Can you post your code?  
That all sounds sane and doable.

(In the long term I have a plan for a new version of DataDeps that makes it easier to do this kind of thing, see DataDepsPaths.jl for the sketch)

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [May 27, 2019, 1:03pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/3 "2019-05-27T13:03:42Z")

</div>

So quick!!! Thanks, gonna check out `DataDepsPaths`. Here’s the code (redacted some of the sensitive stuff):

```julia
using DataDeps, JuliaDB, UUIDs, Dates
function csv2db(f)
    # unpack
    DataDeps.unpack(f)
    repo = datadep"database"
    # load the csv file into a JuliaDB.table
    data = loadtable(joinpath(repo, "data.csv"), indexcols = [1])
    # convert some column types
    data = setcol(data, :col1 => :col1 => UUID, :col2 => :col2 => Nanosecond)
    # save the table to binary
    save(data, joinpath(repo, "data.jldb"))
    # remove the csv file
    rm(joinpath(repo, "data.csv"))
end
register(DataDep("database", "the database", "https://s3.address/database.zip", post_fetch_method = csv2db))

```

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [May 27, 2019, 3:56pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/4 "2019-05-27T15:56:03Z")

</div>

That looks fine to me.  
`unpack` takes an argument as to if to delete the archive (default true).  
But that is fine as you want to.

Also the `post_fetch_method` is executed from the DataDep’s directory so the `joinpath` stuff isn’t needed.

So the following should be the same.

```julia
function csv2db(f)
    # unpack
    DataDeps.unpack(f)
    # load the csv file into a JuliaDB.table
    data = loadtable("data.csv", indexcols = [1])
    # convert some column types
    data = setcol(data, :col1 => :col1 => UUID, :col2 => :col2 => Nanosecond)
    # save the table to binary
    save(data, "data.jldb")
    # remove the csv file
    rm("data.csv")
end

```

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [May 27, 2019, 4:06pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/5 "2019-05-27T16:06:06Z")

</div>

Not sure what I’m doing wrong, but:

```julia
ERROR: IOError: unlink: no such file or directory (ENOENT)
Stacktrace:
 [1] rm at ./file.jl:245 [inlined]
 [2] csv2db(::String) at ./REPL[60]:24
 [3] #18 at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution_automatic.jl:122 [inlined]
 [4] cd(::getfield(DataDeps, Symbol("##18#19")){typeof(csv2db),String}, ::String) at ./file.jl:96
 [5] run_post_fetch(::typeof(csv2db), ::String) at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution_automatic.jl:119
 [6] #download#13(::String, ::Nothing, ::Bool, ::Function, ::DataDep{String,String,typeof(DataDeps.fetch_http),typeof(csv2db)}, ::String) at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution_automatic.jl:84
 [7] download at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution_automatic.jl:70 [inlined]
 [8] handle_missing at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution_automatic.jl:10 [inlined]
 [9] _resolve(::DataDep{String,String,typeof(DataDeps.fetch_http),typeof(csv2db)}, ::String) at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution.jl:83
 [10] resolve(::DataDep{String,String,typeof(DataDeps.fetch_http),typeof(csv2db)}, ::String, ::String) at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution.jl:29
 [11] resolve(::String, ::String, ::String) at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution.jl:54
 [12] resolve(::String, ::String) at /home/yakir/.julia/packages/DataDeps/LiEdA/src/resolution.jl:73
 [13] top-level scope at none:0

```

I’ll try to set up a complete MWE…

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [May 27, 2019, 4:15pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/6 "2019-05-27T16:15:06Z")

</div>

OK… This MWE actually works:

```julia
using DataDeps, JuliaDB, UUIDs, Dates
function csv2db(f)
    # unpack
    DataDeps.unpack(f)
    # load the csv file into a JuliaDB.table
    data = loadtable("database.csv", indexcols = [1])
    # convert some column types
    data = setcol(data, :uuid => :uuid => UUID, :ns => :ns => Nanosecond)
    # save the table to binary
    save(data, "database.jldb")
    # remove the csv file
    rm("database.csv")
end
register(DataDep("database", "the database", "https://s3.eu-central-1.amazonaws.com/vision-group-file-sharing/Fun%20Stuff/database.zip", post_fetch_method = csv2db))
datadep"database"

```

So I am not sure what exactly isn’t working in the real one. The only two differences between the MWE and the real deal are 1) using `@eval` to save all the tables, 2) removing a whole folder that contains many `.csv`s:

```julia
    for x in (:videofile, :video, :interval, :poi, :board, :calibration, :run, :experiment, :pixel_coord)
        @eval save($x, $("$x.jldb"))
        rm("$x.csv")
    end
    rm("pixel", recursive = true)

```

hmmm…

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [May 27, 2019, 4:52pm UTC](https://discourse.julialang.org/t/datadeps-post-fetch-method/24649/7 "2019-05-27T16:52:26Z")

</div>

OK, I found what I was doing wrong. Totally irrelevant to DataDeps. Sorry for the noise and thanks for the help!!!

For the intrested, in:

> [@yakir12](#):
>
> ```julia
> for x in (:videofile, :video, :interval, :poi, :board, :calibration, :run, :experiment, :pixel_coord)
> @eval save($x, $("$x.jldb"))
> rm("$x.csv")
> end
> rm("pixel", recursive = true)
> 
> ```

I’m saving the `pixel_coord` table in the for-loop, but I’m also attempting to `rm` the `pixel_coord` `csv` file, which doesn’t exist (those are all in a `pixel` directory - which I later remove recursively).