# Is there a package that abstracts the storage location so that it's cloud and local path agnostic?
**URL:** https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923
**Category:** Data
**Created:** [September 9, 2021, 9:59am UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923 "2021-09-09T09:59:45Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 9, 2021, 9:59am UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/1 "2021-09-09T09:59:45Z")
Say I want to save a file somewhere on the hard drive, it’s actually not that different to saving it to a S3 bucket.
E.g. `save(location, object)` should work regardless if the location is S3 or local path. Wonder if such a package exists already. E.g. R’s pins is like that I think.
---
### Author:  [@aplavin](https://discourse.julialang.org/u/aplavin)
#### Post date: [September 9, 2021, 10:37am UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/2 "2021-09-09T10:37:38Z")
Not as general as you are looking for, but my package [Alexander Plavin / OpenScienceFramework.jl · GitLab](https://gitlab.com/aplavin/OpenScienceFramework.jl) defines methods for Base filesystem functions to operate (r/w) on remote files stored at [https://osf.io](https://osf.io).
[Alexander Plavin / SquashFS.jl · GitLab](https://gitlab.com/aplavin/SquashFS.jl) does the same (read-only) for squashfs archives.
So, it’s clearly possible to provide an almost-uniform interface to access files on different backends.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [September 9, 2021, 12:03pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/3 "2021-09-09T12:03:36Z")
You can have S3 compatible storage locally.
For instance if you configure a CEPH filesystem. Personally, I would say configuring CEPH on your laptop would be overkill and send you prematurely grey, but hey.
Doing a quick search you would be better setting up Minio, which is relatively easy.
What are you trying to achieve here? Have you looked at HDF5 type storage?
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 9, 2021, 12:04pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/4 "2021-09-09T12:04:56Z")
> [@johnh](#):
>
> What are you trying to achieve here?
make a set of code that abstracts away the storage mechanism so I can develop code offline and move them to aws or the like without changing one line of code.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [September 9, 2021, 12:07pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/5 "2021-09-09T12:07:36Z")
By the way, exposing my ignorance here, saving to a local filesystem and saving to S3 ona web service are different. (Lets assume you are not running Minio locally).
Your local filesystem is a POSIX filesystem and you make POSIX standard calls to manipulate files.
S3 is more recent and uses HTTP semantics to PUT and GET objects. You cannot change a a S3 object - you can change it and write it again, but it is then a different object (I may not be entirely correct here)
There wa a great presnetation on storage in the Dell HPC series yesterday. When it is available I will post a link (assuming I remember)
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [September 9, 2021, 12:20pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/6 "2021-09-09T12:20:03Z")
[FilePathsBase.jl](https://github.com/rofinn/FilePathsBase.jl) tries to make this abstraction, and e.g. [AWSS3.jl](https://github.com/JuliaCloud/AWSS3.jl) exposes a FilesPathBase-based `S3Path` that supports `write`, `read`, `open`, etc. s3 and local filesystems are different enough in subtle ways to make this a bit annoying and occasionally error-prone but it’s largely workable.
Edit: for another approach, see [Datasets.jl](https://www.youtube.com/watch?v=PJkf0CO5APs) ([JuliaCon talk](https://www.youtube.com/watch?v=PJkf0CO5APs))
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 9, 2021, 12:25pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/7 "2021-09-09T12:25:19Z")
> [@johnh](#):
>
> saving to a local filesystem and saving to S3 ona web service are different.
eh… that goes without sayng right? also `save(path_or_s3, object)` `load(path_or_s3, object)` totally abstract-able right?
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [September 9, 2021, 1:20pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/8 "2021-09-09T13:20:32Z")
Datasets.jl looks amazing. I worked for several years running an HPC system used by CFD engineers where we used meaningful directory names in a tree structure. That concept is just so outdated. You are using directory names as metadata - I know this is quite common.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [September 9, 2021, 11:45pm UTC](https://discourse.julialang.org/t/is-there-a-package-that-abstracts-the-storage-location-so-that-its-cloud-and-local-path-agnostic/67923/9 "2021-09-09T23:45:15Z")
FilePathsBase might work better as I don’t want an extra toml file for the workflow I want.