# S3 object stores - current state?

**URL:** https://discourse.julialang.org/t/s3-object-stores-current-state/24437
**Category:** Data
**Created:** [May 21, 2019, 2:56pm UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437 "2019-05-21T14:56:32Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [May 21, 2019, 2:56pm UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/1 "2019-05-21T14:56:32Z")

</div>

What is the current state of packages in Julia for accessing S3 object stores. In particular CEPH.  
Are people just using boto with PyCall?

Forgive me if I have asked this one before. Goldfish bowl memory.

---

<div class="post-metadata">

### Author: ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 21, 2019, 3:05pm UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/2 "2019-05-21T15:05:09Z")

</div>

You can use [AWSCore.jl](https://github.com/JuliaCloud/AWSCore.jl). This package just wraps the AWS REST API which is the same thing that boto does. You therefore shouldn’t notice much of a difference between using the Julia AWS packages and boto.

Because AWS has a REST API for absolutely everything, it’s actually one of the _easiest_ things to use with Julia. For example, I’m probably going to start using some AWS Athena tables soon, which for me is very simple and introduces no dependencies, in contrast to using just about any other kind of database. This is nice because the database situation in Julia is still not very good, one would likely have to resort to using SQLAlchemy through PyCall (since we “lost” JDBC until at least Julia 2.0).

---

<div class="post-metadata">

### Author: ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)
#### Post date: [May 21, 2019, 11:52pm UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/3 "2019-05-21T23:52:09Z")

</div>

> since we “lost” JDBC until at least Julia 2.0

Are you sure? If you’re thinking of [https://github.com/JuliaLang/julia/issues/31104](https://github.com/JuliaLang/julia/issues/31104) that looks more like something that could be addressed in 1.3 if someone has the time.

---

<div class="post-metadata">

### Author: ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 21, 2019, 11:59pm UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/4 "2019-05-21T23:59:38Z")

</div>

I exaggerated a bit. My understanding is that there will be a compile flag for Julia that will make it possible to use JavaCall, but that already drastically reduces the user base. It’s also pretty inconvenient in my case as it complicates deployment, so I probably will avoid JavaCall until I can use it with the pre-compiled binaries.

Please correct me if I’m wrong.

---

<div class="post-metadata">

### Author: ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)
#### Post date: [May 22, 2019, 12:02am UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/5 "2019-05-22T00:02:30Z")

</div>

No, Jeff explained the plan for adding a new calling convention here: [JVM fails to load in 1.1 (JavaCall.jl) · Issue #31104 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/31104#issuecomment-484256248)

If implemented this looks like it will basically solve the problem with relatively minor caveats and without the need for a command line option.

---

<div class="post-metadata">

### Author: ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [May 22, 2019, 12:10am UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/6 "2019-05-22T00:10:56Z")

</div>

I’m confused. To me it looks like he’s saying that there will be an `ALWAYS_COPY_STACKS` build option in 1.2, so if you need to use JavaCall you can build Julia with that option, but that a permanent fix will require a new feature in the `ccall` interface. I don’t see anything here saying that wouldn’t happen until 2.0, but that was my impression, perhaps that’s wrong.

---

<div class="post-metadata">

### Author: ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [May 22, 2019, 12:25am UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/7 "2019-05-22T00:25:41Z")

</div>

It sounds to me like that new `ccall` calling convention is what you need, which presumably could be added by 1.3 since it wouldn’t be breaking. JavaCall would just need to be updated to use this calling convention, which should be as simple as adding a `javacall` argument to all `ccall`s into the JVM. Hopefully someone will correct me if this sounds wrong.

---

<div class="post-metadata">

### Author: ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)
#### Post date: [May 22, 2019, 1:34am UTC](https://discourse.julialang.org/t/s3-object-stores-current-state/24437/8 "2019-05-22T01:34:25Z")

</div>

Exactly, it’s just extending ccall with support for an alternative calling convention. I expect the only thing holding this up is the limited time of the few people who have the right expertise.
