Did you have any success trying to get this to work? From what I understand it seems like it works, but you aren’t able to import packages?
Nope, I was trying to complete this in a 1-day hackathon and was unsuccessful. The final blocker was that I couldn’t get the size of the entire bundle within the aws limits. From Lambda quotas - AWS Lambda, the constraining limit was the size of the unzipped deployment package (250 MB).
Just wandered across this thread and I’m curious: has anyone has nailed down Julia on AWS Lambda? I’ve found a variety of older repos, but very little documentation on how to make it work. Can someone point me to a newer repo and documentation? I’d much appreciate the assist!
Hi @abe I just saw your comment. Also replying to @ianfiske
We discussed this at the Cambridge Julia Meetup - indeed the AWS limits on size seem to be a blocker.
This is a shame - however I imagine that there are sound reasons for the size limits, as lammdas are supposed to execute quickly and be set up/torn down easily.
I would like to explore this further though. We recently saw a Top500 supercomputer entry from https://www.descarteslabs.com/ on AWS.
One of the problems with setting up large distributed computations there is the set up time - I believe they used a pre-configured VM instance.
Speaking very loosely, lambdas are meant to perform functions. It would be great to see Julia being able to run distributed functions using lambdas rather than setting up VMs.
I guess we need to pay attention to
a) the ahead of time compilation of packages - yes I know there is PackageCompiler.jl
There was also a package described on here recently which approached this problem.
I forget what it was - can anyone remind me?
b) do Distributed Arrays work in this configuration
1 Like
There was also a package described on here recently which approached this problem.
I forget what it was - can anyone remind me?
Probably Fezzik.jl
1 Like
Over on Openstack, I had not heard of Qinling which is Function As A Service for Openstack.
I am not sure if this offers any advantages for Julia, but might be interesting.
Sorry for necro-posting, but I dont see why I would want to start a new thread. Has anyone successful made anything in this area? This should be even easier now with PackageCompiler.create_app
1 Like
Just to update here, looks like this is now possible (see [ANN] AWS Lambda Julia runtime and Running AWS Lambda functions with Julia | Tom Kwong - YouTube)
This is something I have been trying to use at my work, and I’ve created a package to simplify the process of creating the Lambda-compatible container and then pushing it to Lambda: GitHub - harris-chris/Jot
Overall Julia seems to work well here - one remaining problem is that if the container is cold-started (ie, if the function has not been called for a while) then there is about a 2s delay while it gets going, vs a ~10ms delay if warm-started. This can theoretically be eliminated by PackageCompiler.jl and I’m trying to incorporate that at the moment.
3 Likes
Very cool!