Creating a custom Julia AWS Layer Runtime

Did you have any success trying to get this to work? From what I understand it seems like it works, but you aren’t able to import packages?

Nope, I was trying to complete this in a 1-day hackathon and was unsuccessful. The final blocker was that I couldn’t get the size of the entire bundle within the aws limits. From https://docs.aws.amazon.com/lambda/latest/dg/limits.html, the constraining limit was the size of the unzipped deployment package (250 MB).

Just wandered across this thread and I’m curious: has anyone has nailed down Julia on AWS Lambda? I’ve found a variety of older repos, but very little documentation on how to make it work. Can someone point me to a newer repo and documentation? I’d much appreciate the assist!

Hi @abe I just saw your comment. Also replying to @ianfiske
We discussed this at the Cambridge Julia Meetup - indeed the AWS limits on size seem to be a blocker.
This is a shame - however I imagine that there are sound reasons for the size limits, as lammdas are supposed to execute quickly and be set up/torn down easily.

I would like to explore this further though. We recently saw a Top500 supercomputer entry from https://www.descarteslabs.com/ on AWS.
One of the problems with setting up large distributed computations there is the set up time - I believe they used a pre-configured VM instance.
Speaking very loosely, lambdas are meant to perform functions. It would be great to see Julia being able to run distributed functions using lambdas rather than setting up VMs.
I guess we need to pay attention to

a) the ahead of time compilation of packages - yes I know there is PackageCompiler.jl
There was also a package described on here recently which approached this problem.
I forget what it was - can anyone remind me?

b) do Distributed Arrays work in this configuration

1 Like

There was also a package described on here recently which approached this problem.
I forget what it was - can anyone remind me?

Probably Fezzik.jl

1 Like

Over on Openstack, I had not heard of Qinling which is Function As A Service for Openstack.

I am not sure if this offers any advantages for Julia, but might be interesting.