Error deploying in AWS Lambda and Google Cloud Run

General Usage
, , ,
1

I am trying to deploy a simple microservice in a docker container with a HTTP server that uses JuMP in AWS Lambda or Google Cloud Run. I based my application’s code in Jacob Quinn’s example from JuliaCon 2020 (JuliaCon 2020 | Building Microservices and Applications in Julia - YouTube).
When deploying only the HTTP server without JuMP, there are no errors and the service runs smoothly. However, by adding JuMP as a dependency and running “using JuMP” in the source code, the following error occurs when handling requests:
InitError(mod=:Bzip2_jll, error=ErrorException("Artifact "Bzip2" was not installed correctly. Try `using Pkg; Pkg.instantiate()` to re-install all missing resources.")).

Here is the full log from AWS Lambda:

OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
fatal: error thrown and no exception handler available.
InitError(mod=:Bzip2_jll, error=ErrorException("Artifact "Bzip2" was not installed correctly. Try `using Pkg; Pkg.instantiate()` to re-install all missing resources."))
error at ./error.jl:33
_artifact_str at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:551
jfptr__artifact_str_95799 at /home/AlgorithmService.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:714
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
#invokelatest#2 at ./essentials.jl:708
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
invokelatest at ./essentials.jl:706
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_fptr_interpret_call at /buildworker/worker/package_linux64/build/src/interpreter.c:650
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:680 [inlined]
find_artifact_dir at /root/.julia/packages/JLLWrappers/bkwIo/src/wrapper_generators.jl:15
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
__init__ at /root/.julia/packages/Bzip2_jll/aAOqr/src/wrappers/x86_64-linux-gnu.jl:7
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:72
_julia_init at /buildworker/worker/package_linux64/build/src/init.c:794
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:696
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4007d8)

This same situation occurs in both Google Cloud Run and AWS Lambda. Also, the same container runs locally without errors.

Any help would be greatly appreciated!

3 Likes
2

What happens if you only add Bzip2_jll as a dependency as you run using Bzip2_jll?

1 Like
3

I’m having the same issue. This happens when I compile my app using PackageCompiler.jl and delete the artifacts folder. I thought this folder was automatically downloaded when it does not exist.

@odow I tried to add the Bzip2_jll explicitly, but it didn’t work. Do you have any suggestions?

4

I solved it by downgrading the base Julia image of the Docker container from 1.6 to 1.5.4. It seems to me that this problem is specific to Julia 1.6. As I didn’t need the new release for my project, this was enough to get it working.
If anyone finds a solution using Julia 1.6 I would be glad to know it!

2 Likes
5

I tried to add the Bzip2_jll explicitly, but it didn’t work. Do you have any suggestions?

No. If Bzip2_jll didn’t install, then this is a problem with Bzip2, not JuMP.

Is it the same error as above?

6

I think this could be something more general happening with serverless. I had a similar problem without using PackageCompiler.jl and with the Dash julia package instead. Julia 1.7.1.

The build went fine and it included the Pkg.instantiate call. The Run deployment then fails.

First the warning:

"OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k"

Then the error that causes the deployment to fail:

ERROR: LoadError: ArgumentError: Package Dash [1b08a953-4be3-4667-9a23-3db579824955] is required but does not seem to be installed:

In a successful second attempt, I did two things.

  • Instantiate in the first line of the app run script, (despite having done that in the build).
  • Used the “Second generation” Google Cloud Run server.
    Not sure what made the difference.