# DataDeps and Google Drive

**URL:** <https://discourse.julialang.org/t/datadeps-and-google-drive/12271>\
**Category:** Data\
**Tags:** question\
**Created:** [July 9, 2018, 11:38am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271 "2018-07-09T11:38:08Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 9, 2018, 11:38am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/1 "2018-07-09T11:38:09Z")

</div>

Hi all (and hopefully oxinabox)!

I’m trying to access data on a Google Drive using the excellent [DataDeps.jl](https://github.com/oxinabox/DataDeps.jl).

The drive has been shared with me (so I don’t _own_ it) and the data in it isn’t zipped (there are multiple folders in the shared parent folder). The data in it changes but very sporadically. While I could download the whole thing and work from there, it would be better if I could “access” it with `DataDeps`. I then could delete the local repository once in a while to keep it fresh.

**Has anyone managed to get `DataDeps` to work with Google Drive?**

Thanks!

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 9, 2018, 3:05pm UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/2 "2018-07-09T15:05:44Z")

</div>

I take it you don’t have a public URL for it?  
It is private, authed to you?  
I am sure this can be made to work.  
By replacing the `fetch_method` with something that does auth  
I just am not sure exactly how.

Of course you can always use a `ManualDataDep` and external syncing  
Which while it misses out on most of the advantages it does mm eave you an easy path forward once we workout how to automate it, since only the registration block would change

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 9, 2018, 3:22pm UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/3 "2018-07-09T15:22:00Z")

</div>

If you are on Linux easiest might be to make a function that wraps `run(`grive2 -s $remote`)`  
Plus some combination of `cd` and `mv` to make it work roughly like `download(remote, local)`.

[http://yourcmc.ru/wiki/Grive2](http://yourcmc.ru/wiki/Grive2)

Or maybe this command line too.  
I’ve not used but seems better maintained than grive2 [https://github.com/odeke-em/drive/](https://github.com/odeke-em/drive/)

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 9, 2018, 5:42pm UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/4 "2018-07-09T17:42:10Z")

</div>

> [@oxinabox](#):
>
> use a `ManualDataDep` and external syncing

This seems like a totally acceptable solution for now. I’ll do that, to start with.

> [@oxinabox](#):
>
> If you are on Linux

This will need to work on all three platforms.

Thanks @oxinabox! Ping here again if and when a complete solution comes up!

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 10, 2018, 3:00am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/5 "2018-07-10T03:00:47Z")

</div>

So I did some poking,  
it is actually really easy to download a file from drive **IF** you can sort out auth.  
You can get the ID out of the webpage (inspect element shows it as a `data` field in the list of files. And you can get it a few other ways).  
Which basically gives you a URL  
See: [Files: get &nbsp;|&nbsp; Drive API &nbsp;|&nbsp; Google Developers](https://developers.google.com/drive/api/v3/reference/files/get)

Problem is setting up Auth  
OAuth 2.0 is just a really annoying process to set up.  
I mean it is as nice as it can be while still being secure, but that is not nice.  
If we had a good Outh 2.0 library, this would be doable.  
I am not aware of once.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 10, 2018, 7:34am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/6 "2018-07-10T07:34:39Z")

</div>

> [@oxinabox](#):
>
> I did some poking

Thanks @oxinabox!

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 10, 2018, 11:35am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/7 "2018-07-10T11:35:41Z")

</div>

I found this kind by accident  
It might be possible to RCall it.  
Then wrapping it’s `drive_download` into a `fetch_method`  
would be easy

> **[An Interface to Google Drive](https://googledrive.tidyverse.org/index.html)**
>
> Manage Google Drive files from R.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 10, 2018, 11:57am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/8 "2018-07-10T11:57:41Z")

</div>

Wow, that sounds like a quick solution. I’ll look into that. Thanks!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 10, 2018, 1:30pm UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/9 "2018-07-10T13:30:09Z")

</div>

Tried it. It most probably could be made to work, but:

1. `googledrive` has `libcurl4-openssl-dev` as a dependency, so build dependencies need to include that.
2. there is no native solution for recursively downloading a folder (except [this](https://github.com/MilesMcBain/mmmisc/blob/master/R/utils.R#L211) code snippet).
3. because of the hell that is native Google file types, a folder can contain two files with identical names ([google-drive-ocamlfuse](https://github.com/astrada/google-drive-ocamlfuse) solves this by slapping an extra file extension on the ambiguous ones, e.g. a `file.csv` that is a google spreadsheet becomes `file.csv.csv` when downloaded). If trying to download such identical files it throws an error.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 10, 2018, 1:35pm UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/10 "2018-07-10T13:35:31Z")

</div>

> [@oxinabox](#):
>
> [An Interface to Google Drive • googledrive](https://googledrive.tidyverse.org/index.html)

I guess that ideally we would have some Julian API to deal with google drive (and other such popular solutions) and once that API is solid DataDeps could use _that_ to provide the same functionality it has for other repos. The API R has for google drive is lacking (e.g. we can’t even download a folder). So I’m not sure you want to build and rely on that. But I guess it’s better than nothing at all.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 10, 2018, 5:34pm UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/11 "2018-07-10T17:34:26Z")

</div>

Right, concept proved.  
This works with DataDeps.jl  
It wraps PyDrive.

It throws a lot of warnings, because DataDeps.jl has a kind of assumption that remotes would be HTTP URLS represented as Strings, but its fallbacks kick in and so does deal with it.  
Though it seems like there over tight constraints are a bit of a minor bug.

Another thing is that while the download of the Files is lazy,  
the download of the file names in the registration block is eager,  
I think the simple solution to that is to use some kind of LazyVector type,

[https://github.com/oxinabox/PyDrive.jl/blob/master/source/proto.ipynb](https://github.com/oxinabox/PyDrive.jl/blob/master/source/proto.ipynb)

Right now it is almost all in that notebook rather than in a proper julia repo.  
If you or someone else want to take PyDrive wrapping stuff and make it into a proper julia repo, that would be cool.  
idk when (/if) I’ld have time to work on this again.

Having the concept proven is pleasing to me.

* * *

DataDeps itself doesn’t deal with the idea of downloading a folder very well.  
Because most of the time when you want to download a folder that means downloadng a tarball or a .zip which keeps structure. Otherwise you are downloading a collection of files without structure to where they end up (except as imposed by applying `mv` as a post0fetch method)

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 11, 2018, 7:49am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/12 "2018-07-11T07:49:01Z")

</div>

> [@oxinabox](#):
>
> [https://github.com/oxinabox/PyDrive.jl/blob/master/source/proto.ipynb](https://github.com/oxinabox/PyDrive.jl/blob/master/source/proto.ipynb)

Wow! Great work! This seems like the best version of how to make this work!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 11, 2018, 8:05am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/13 "2018-07-11T08:05:38Z")

</div>

> [@oxinabox](#):
>
> take PyDrive wrapping stuff and make it into a proper julia repo

I’m trying to understand how you mean for people to use this. Do you mean that people that want to use DataDeps with a Google Drive will need to include [PyDrive.jl](https://github.com/oxinabox/PyDrive.jl) (after we clean it up and all) for it to work?  
Kind of like this?

```julia
using PyDrive, DataDeps
register(DataDep("GoogleDriveDemo",
        "Demonstration of google drive",
         list_files_in_folder("Demo"),
         fetch_method = drive_download));

```

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 11, 2018, 8:31am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/14 "2018-07-11T08:31:27Z")

</div>

Yes, exactly.

Optionally put `Any` in the checksum argument position to bypass the checksum checking,  
and suppress that warning about checksum not given

They could also use PyDrive.jl for other things without DataDeps.jl being in it.  
If it supported that

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [July 11, 2018, 8:44am UTC](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/15 "2018-07-11T08:44:20Z")

</div>

> [@oxinabox](#):
>
> They could also use PyDrive.jl for other things without DataDeps.jl being in it.

Yea, so `PyDrive` will be that google drive API I mentioned [before](https://discourse.julialang.org/t/datadeps-and-google-drive/12271/10). And DataDeps will be able to work with that. Nice.