[RFC] Cloud+Container Infrastructure Management

mrufsvold
Mon, 08 Apr 2024 15:51:46 +0000
Package Management

I asked a question previously about managing incremental builds of Julia Docker containers but didn’t hear much back. Since then, I’ve been sketching out some ideas that I’d appreciate feedback on.

The Problem

Deploying Julia to the cloud (AWS in my case) poses some challenges.

1. Base images with facilities to interact with the cloud provider don’t have Julia installed. Julia images don’t have the facilities installed. Creating a cached provider image + Julia image requires nontrivial infrastructure work.

2. In a similar vein, you don’t really want to precompile your project’s whole dependency chain on every deploy. It is very time consuming and usually redundant. But caching that result is, again, nontrivial.

Right now, redeploying my project takes a very long time because it downloads Julia (I think this download gets cached fortunately), downloads my project dependencies, and precompiles the whole thing. My team uses Serverless Framework, but it isn’t easy declare via CloudFormation “Please rebuild a base image if my dependencies are updated, and then use that to build my new project image.”

The (Potential) Solution

I am considering a package that has the following facilities:

1. Build a + <your current julia version> image.

2. Build a PackageCompiler.jl compiled image of your current project’s dependencies built off of #1 iff your Manifest has changed.

3. Build a current project state image which copies your project code into #2, runs any precompile commands you have, and adds an entry point.

4. Functions which facilitate storing and retrieving these images from the image registry of your choice.

5. Potentially, some CLI facilities so that this could be managed by CodeBuild/Github Actions/etc.

Questions

1. Does this already exist somewhere that I’m missing?

2. Do you foresee any pitfalls that I might be missing (I would call my DevOps skills intermediate, so I’m sure I have some blind spots).

3. I’ve only really worked in AWS, and for my purposes, would be focused on ECR and CodeBuild, but I’d love for this to be extensible to whatever environment people might use. Are there any quirks I should pay attention to make sure the API is cloud agnostic?

12 posts - 6 participants