# New Artifact System, Data, and OneDrive

**URL:** <https://discourse.julialang.org/t/new-artifact-system-data-and-onedrive/33250>\
**Category:** General Usage\
**Created:** [January 11, 2020, 11:40pm UTC](https://discourse.julialang.org/t/new-artifact-system-data-and-onedrive/33250 "2020-01-11T23:40:41Z")\
**Posts on this page:** 1\
**Showing post:** 11

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [January 18, 2020, 12:52am UTC](https://discourse.julialang.org/t/new-artifact-system-data-and-onedrive/33250/11 "2020-01-18T00:52:21Z")

</div>

> [@Crown421](#):
>
> I think for my use-case, `DataDeps` is more appropriate right now. However, the question of secure storage/access is still a problem.
> 
> - I suppose one could take [onedrive-sdk-python](https://github.com/OneDrive/onedrive-sdk-python), write a PyCall.jl wrapper around it, and create a nice API and extend the relevant methods in `Pkg.artifact` and/or `DataDeps` . (Probably alot of effort)
> - Alternatively, use a different Cloud Storage provider. Does anyone know a service that is nicely accessible, perhaps via SSH keys?

Yes, one of the this DataDeps lets you do that Artifacts intentionally doesn’t is customize your transport mechanism via setting the fetch method. Or via having a path-like type that overloads `Base.download`.

Its relatively easy to add support for downloading via some other means.  
You just set the `fetch_method` in the registration block.

Something like shelling out to the linux secure copy tool looks like:

```julia
fetch_method = (local_dir, remote_path) -> run(`scp myserver:$remote_dir $filepath)

```

Or you use a type that overloads `Base.download`:  
like  
[AWSS3.jl](https://github.com/JuliaCloud/AWSS3.jl/) which works with DataDeps and with AWS’s auth system out of the box.  
If you’ve used AWS S3 before that would be my go to choice, that gets used all the time at Invenia.

There is a more proof of conceppt example in  
[PyDrive.jl](https://github.com/oxinabox/PyDrive.jl) for how to use PyCall to access GoogleDrive with their auth system and DataDeps.jl

* * *

Artifacts has some lessons learned from DataDeps. So there are many overlaps.  
DataDeps is more flexible it also allows custom post-processing so you can use any file format not just tarballs. (and you don’t have to have all data inside tarballs).

Artifacts uses tree-hashs opf the post unpacking file structure which means they can check artififacts decompressed right.  
Where as DataDeps only uses a hash of the file downloaded so can only check if the download worked.

Artifacts use content addressing so can’t ever run into a name clash.

* * *

I also would like to know if anyone has used Artifacts for data in the wild.

---

_[View the full topic](https://discourse.julialang.org/t/new-artifact-system-data-and-onedrive/33250)._