PackageCompiler + Sockets + Web app

I’m trying to build an image of a simple web project. The project is present as a package in the global list of packages and also is located my dev path. But getting stable error:

./build_native.jl
[ Info: Copying system image: /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/backup/native/sys.dylib to /Applications/Julia-1.2.app/Contents/Resources/julia/lib/julia/sys.dylib
Julia program file:
  "/Users/.julia/dev/JWebImageDemo/run.jl"
C program file:
  "/Users/.julia/packages/PackageCompiler/CJQcs/examples/program.c"
Build directory:
  "/Users/.julia/dev/JWebImageDemo/build"
  4.508057 seconds (11.03 M allocations: 607.603 MiB, 8.55% gc time)
Test Summary:    | Pass  Total
JWebImageDemo.jl |    1      1
[ Info: used 294 out of 295 precompile statements
ERROR: LoadError: LoadError: LoadError: LoadError: UndefVarError: uv_jl_connectioncb not defined
Stacktrace:
 [1] #trylisten#14 at /Applications/Julia-1.2.app/Contents/Resources/julia/share/julia/stdlib/v1.2/Sockets/src/Sockets.jl:543 [inlined]
 [2] trylisten at /Applications/Julia-1.2.app/Contents/Resources/julia/share/julia/stdlib/v1.2/Sockets/src/Sockets.jl:542 [inlined]
 [3] #listen#13 at /Applications/Julia-1.2.app/Contents/Resources/julia/share/julia/stdlib/v1.2/Sockets/src/Sockets.jl:521 [inlined]
 [4] #listen at ./none:0 [inlined]
 [5] #listen#8(::Int64, ::typeof(listen), ::Sockets.InetAddr{IPv4}) at /Applications/Julia-1.2.app/Contents/Resources/julia/share/julia/stdlib/v1.2/Sockets/src/Sockets.jl:491
 [6] listen at /Applications/Julia-1.2.app/Contents/Resources/julia/share/julia/stdlib/v1.2/Sockets/src/Sockets.jl:489 [inlined]
 [7] #start#1(::IPv4, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Bukdu.start), ::Int64) at /Users/.julia/packages/Bukdu/FNxUN/src/server.jl:24
 [8] (::getfield(Bukdu, Symbol("#kw##start")))(::NamedTuple{(:host,),Tuple{IPv4}}, ::typeof(Bukdu.start), ::Int64) at ./none:0
 [9] top-level scope at /Users/.julia/dev/JWebImageDemo/src/server.jl:72
 [10] include at ./boot.jl:328 [inlined]
 [11] include_relative(::Module, ::String) at ./loading.jl:1094
 [12] include(::Module, ::String) at ./Base.jl:31
 [13] include(::String) at ./client.jl:431
 [14] top-level scope at /Users/.julia/dev/JWebImageDemo/run.jl:3
 [15] include at ./boot.jl:328 [inlined]
 [16] include_relative(::Module, ::String) at ./loading.jl:1094
 [17] include(::Module, ::String) at ./Base.jl:31
 [18] top-level scope at /Users/.julia/dev/JWebImageDemo/build/julia_main.jl:1
 [19] include at ./boot.jl:328 [inlined]
 [20] include_relative(::Module, ::String) at ./loading.jl:1094
 [21] include at ./Base.jl:31 [inlined]
 [22] include(::String) at /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:8
 [23] top-level scope at /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:10
 [24] include at ./boot.jl:328 [inlined]
 [25] include_relative(::Module, ::String) at ./loading.jl:1094
 [26] include(::Module, ::String) at ./Base.jl:31
 [27] exec_options(::Base.JLOptions) at ./client.jl:295
 [28] _start() at ./client.jl:464
in expression starting at /Users/.julia/dev/JWebImageDemo/src/server.jl:72
in expression starting at /Users/.julia/dev/JWebImageDemo/run.jl:3
in expression starting at /Users/.julia/dev/JWebImageDemo/build/julia_main.jl:1
in expression starting at /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl:10
ERROR: LoadError: failed process: Process(`/Applications/Julia-1.2.app/Contents/Resources/julia/bin/julia --output-o=run.a --track-allocation=none --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --project=@. --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/Applications/Julia-1.2.app/Contents/Resources/julia/lib/julia/sys.dylib --compiled-modules=yes --optimize=2 /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]

The project which I’m running is - https://github.com/rssdev10/JWebImageDemo.jl - just a simple demo.

I have entry point run.jl. And for that file I have prepared the build script:

#!/usr/bin/env julia --project=@.

using PackageCompiler

# Or if you simply want to get a native system image e.g. when you have downloaded the generic Julia install:
force_native_image!()

# Build an executable
build_executable(
    "run.jl", # Julia script containing a `julia_main` function, e.g. like `examples/hello.jl`
    snoopfile = "test/runtests.jl", # Julia script which calls functions that you want to make sure to have precompiled [optional]
    builddir = "build" # that's where the compiled artifacts will end up [optional]
)

# Build a shared library
build_shared_lib("run.jl")

For now I’m not getting where to search the issue. The mentioned file https://github.com/JuliaLang/julia/blob/master/stdlib/Sockets/src/Sockets.jl contains uv_jl_connectioncb initialization. And the only reason I see, why it might be unavailable, __init__() doesn’t work when PackageCompiler runs the code. libuv I also have explicitly installed. So, what might be the reason of the failure? Is it my mistake? Or PackageCompiler? Or Julia? Or Sockets?

__init__() doesn’t run when outputting code to a file. You can invoke it explicitly.

So, am I understand right that following globals from Sockets.jl should be re-declared outside __init__()?

function __init__()
    global uv_jl_getaddrinfocb = @cfunction(uv_getaddrinfocb, Cvoid, (Ptr{Cvoid}, Cint, Ptr{Cvoid}))
    global uv_jl_getnameinfocb = @cfunction(uv_getnameinfocb, Cvoid, (Ptr{Cvoid}, Cint, Cstring, Cstring))
    global uv_jl_recvcb        = @cfunction(uv_recvcb, Cvoid, (Ptr{Cvoid}, Cssize_t, Ptr{Cvoid}, Ptr{Cvoid}, Cuint))
    global uv_jl_connectioncb  = @cfunction(uv_connectioncb, Cvoid, (Ptr{Cvoid}, Cint))
    global uv_jl_connectcb     = @cfunction(uv_connectcb, Cvoid, (Ptr{Cvoid}, Cint))
end

Not sure what you mean with that, they cannot be seralized so need to be reinitialized on every julia start. If you want to use Socket operations while outputting to a file you need to explicitly call Sockets.__init__()

Not sure that I’m understanding the process of the package compiling… If __init__() isn’t called at all, I have a big issue with activation of most of external packages… Actually I meant something like:

uv_jl_connectioncb = nothing # declare the variable to avoid UndefVarError
function __init__()
    global uv_jl_connectioncb  = @cfunction(uv_connectioncb, Cvoid, (Ptr{Cvoid}, Cint))
    ...
end

But ok, I have added into my run.jl:

import Sockets
Sockets.__init__()

Looks like it was started as an app and I have some other error:

./build_native.jl
[ Info: Copying system image: /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/backup/native/sys.dylib to /Applications/Julia-1.2.app/Contents/Resources/julia/lib/julia/sys.dylib
Julia program file:
  "/Users/.julia/dev/JWebImageDemo/run.jl"
C program file:
  "/Users/.julia/packages/PackageCompiler/CJQcs/examples/program.c"
Build directory:
  "/Users/.julia/dev/JWebImageDemo/build"
  4.732265 seconds (11.03 M allocations: 607.603 MiB, 8.52% gc time)
Test Summary:    | Pass  Total
JWebImageDemo.jl |    1      1
[ Info: used 294 out of 295 precompile statements
Bukdu Listening on 127.0.0.1:8080
ERROR: LoadError: LoadError: LoadError: LoadError: PCRE.exec error: NULL argument passed
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] exec at ./pcre.jl:140 [inlined]
 [3] exec at /Users/.julia/packages/HTTP/hJSsm/src/parseutils.jl:43 [inlined] (repeats 2 times)
 [4] #parse_uri_reference#6(::Bool, ::typeof(HTTP.URIs.parse_uri_reference), ::String) at /Users/.julia/packages/HTTP/hJSsm/src/URIs.jl:123
 [5] parse_uri_reference at /Users/.julia/packages/HTTP/hJSsm/src/URIs.jl:123 [inlined]
 [6] Type at /Users/.julia/packages/HTTP/hJSsm/src/URIs.jl:143 [inlined]
 [7] handle(::HTTP.Messages.Request) at /Users/.julia/packages/Bukdu/FNxUN/src/Routing.jl:17
 [8] call(::HTTP.Messages.Request) at /Users/.julia/packages/Bukdu/FNxUN/src/Router.jl:30
 [9] call(::Function, ::String) at /Users/.julia/packages/Bukdu/FNxUN/src/Router.jl:23
 [10] top-level scope at /Users/.julia/dev/JWebImageDemo/src/server.jl:74
 [11] include at ./boot.jl:328 [inlined]

NULL argument passed looks like something is not initialized here too… Base.PCRE, HTTP.URIs, HTTP or Bukdu…

Ok, thanks. Primary issue was solved by adding of explicit init calls into run.jl which is entry point:

import Sockets
Sockets.__init__()

import HTTP.URIs
import HTTP.Parsers
URIs.__init__()
Parsers.__init__()

But there is next issue. I’m running the service process which might be stopped by system signal only. If I’m pressing CTRL-C in a console with PackageCompiler, the process breaks completely.

If I’m commenting a line Base.JLOptions().isinteractive==0 && wait(), I’m any way getting some other error:

./build_native.jl
[ Info: Copying system image: /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/backup/native/sys.dylib to /Applications/Julia-1.2.app/Contents/Resources/julia/lib/julia/sys.dylib
Julia program file:
  "/Users/.julia/dev/JWebImageDemo/run.jl"
C program file:
  "/Users/.julia/packages/PackageCompiler/CJQcs/examples/program.c"
Build directory:
  "/Users/.julia/dev/JWebImageDemo/build"
  4.158659 seconds (11.03 M allocations: 607.603 MiB, 8.95% gc time)
Test Summary:    | Pass  Total
JWebImageDemo.jl |    1      1
[ Info: used 294 out of 295 precompile statements
Bukdu Listening on 127.0.0.1:8080
INFO: GET     StaticController    readindexfile   200 /
fatal: error thrown and no exception handler available.
ErrorException("Task cannot be serialized")
rec_backtrace at /Users/sabae/buildbot/worker/package_macos64/build/src/stackwalk.c:94
record_backtrace at /Users/sabae/buildbot/worker/package_macos64/build/src/task.c:219
jl_throw at /Users/sabae/buildbot/worker/package_macos64/build/src/task.c:429
jl_error at /Users/sabae/buildbot/worker/package_macos64/build/src/rtutils.c:41
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:294
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_module at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:230 [inlined]
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:291
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:277
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_module at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:230 [inlined]
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:291
jl_serialize_module at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:232 [inlined]
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:291
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:286
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:303
jl_serialize_module at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:230 [inlined]
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:291
jl_serialize_module at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:222 [inlined]
jl_serialize_value_ at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:291
jl_save_system_image_to_stream at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:1274
jl_create_system_image at /Users/sabae/buildbot/worker/package_macos64/build/src/staticdata.c:1379
jl_write_compiler_output at /Users/sabae/buildbot/worker/package_macos64/build/src/precompile.c:76
jl_atexit_hook at /Users/sabae/buildbot/worker/package_macos64/build/src/init.c:231
main at /Applications/Julia-1.2.app/Contents/Resources/julia/bin/julia (unknown line)
ERROR: LoadError: failed process: Process(`/Applications/Julia-1.2.app/Contents/Resources/julia/bin/julia --output-o=run.a --track-allocation=none --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --project=@. --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/Applications/Julia-1.2.app/Contents/Resources/julia/lib/julia/sys.dylib --compiled-modules=yes --optimize=2 /Users/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:813 [inlined]
 [2] #run#536(::Bool, ::typeof(run), ::Cmd) at ./process.jl:728
 [3] run at ./process.jl:726 [inlined]
 [4] #run_julia#1 at /Users/.julia/packages/PackageCompiler/CJQcs/src/compiler_flags.jl:225 [inlined]
 [5] #run_julia at ./none:0 [inlined]

Regarding ErrorException("Task cannot be serialized") I see that it is still actual at https://github.com/JuliaLang/PackageCompiler.jl/issues/184

But how to do properly compile service process which should be never stopped?

You need to make sure that no tasks are left running when outputting code to a file.

But what is to do with the web app? The code looks like:

Bukdu.start(server.port; host=server.host)
Router.call(get, "/") #
Base.JLOptions().isinteractive==0 && wait()

It works while not getting a break signal. Definitely I can add some new route/handler pair for the web app to stop the server, but in that case I need to think about security. Otherwise anyone can stop the server in production…

Any specific recommendations about it? May be there is a way to detect that the code is running under PackageCompiler for doing some work around of the waiting loop?

PackageCompiler works by setting up a state that is serialized to a file (a sysimage). When loading that sysimage the state is recreated by deserializing. You cannot serialize tasks so they cannot be running when trying to serialize. It is up to you to make sure that is the case.

see also example for building an executable
https://github.com/wookay/Bukdu.jl/tree/master/examples/app

Thanks you for the help! For now it is fixed. As a result:

  1. PackageCompiler requires explicit call of __init__ method in dependent modules and explicit import of hidden dependencies (in my case it is QuartzImageIO)

Final run.jl script:

#!/usr/bin/env julia --project=@.
module Starter

# this part is required for PackageCompiler
import Sockets
Sockets.__init__()

import HTTP.URIs
import HTTP.Parsers
URIs.__init__()
Parsers.__init__()

import QuartzImageIO
#import ImageMagick
# end of PackageCompiler

include("src/server.jl")

Base.@ccallable function julia_main(ARGS::Vector{String})::Cint
    start_server()
    return 0
end

include("src/server.jl")

# run when exactly this script is activated
endswith(PROGRAM_FILE,  basename(@__FILE__)) && start_server()
end

The only question here how to detect that the code is executed not by PackageCompiler to exclude the part of code with explicit activation. @kristoffer.carlsson Is any way to do it? My idea is to use same run.jl script both for local running and for native code building. In case of local running definitely I have warnings about duplicate entities.

  1. Native building script is looking like:
#!/usr/bin/env julia --project=@.

using PackageCompiler

# Or if you simply want to get a native system image e.g. when you have downloaded the generic Julia install:
force_native_image!()

# Build an executable
build_executable(
    "run.jl", # Julia script containing a `julia_main` function, e.g. like `examples/hello.jl`
    snoopfile = "test/runtests.jl", # Julia script which calls functions that you want to make sure to have precompiled [optional]
    builddir = "build" # that's where the compiled artifacts will end up [optional]
)

# Build a shared library
build_shared_lib("src/JWebImageDemo.jl")

And I think it is better that juliac.jl because of collecting code by unit tests.

  1. A script for calling juliac.jl might be looking like:
#!/bin/bash

PACKAGE_PATH=`julia -e 'using PackageCompiler; PackageCompiler |> pathof |> dirname |> println'`

julia --project=@. "$PACKAGE_PATH/../juliac.jl" -vae run.jl

@wookyoung In that way it doesn’t require to move PackageCompiler into dev stage. And also the way how to run a service process from a script which doesn’t require to do any manual activities:

#!/bin/bash

terminate_all() {
	kill -9 $PID
	exit 0
}
trap terminate_all SIGHUP SIGINT SIGTERM

./build/run &
PID=$!
wait $PID

PID also might be stored in a file and be used later outside the script.

i got hung up on PackageCompiler not calling __init__ too. and with Sockets, you now need to using LibUV_jll; LibUV_jll.__init__(), as Sockets itself no longer as an __init__.

where in PackageCompiler’s docs is a good place to mention that __init__ is not serialized?