Create executable file to be used in another julia project

General Usage Performance
, , ,
1

I am trying to create executable file(probably .so file) that can be used in my another julia project to access methods in it. I have tried to do it so using package compiler but getting below errors while using methods “create_sysimage” and “create_library”.

I would like to check if this is the right approach (or) better way of handling this?

My requirement is I have julia app which will have to run various package code based on the request so instead of adding all those modules as dependency I am trying to create them as executables so that I can directly run them from my app.

⡆ [00m:21s] PackageCompiler: compiling incremental system imageUndefRefError: access to undefined reference
Stacktrace:
  [1] getproperty
    @ .\Base.jl:37 [inlined]
  [2] getindex
    @ .\refvalue.jl:59 [inlined]
⠙ [00m:21s] PackageCompiler: compiling incremental system image_extract_common_kw_args(service::AWS.JSONService, args::Dict{String, Any})
    @ AWS C:\Users\xxxxx\.julia\packages\AWS\SchLh\src\utilities\utilities.jl:53
⠸ [00m:21s] PackageCompiler: compiling incremental system image(::AWS.JSONService)(operation::String, args::Dict{String, Any}; aws_config::AWS.AWSConfig, feature_set::AWS.FeatureSet)
    @ AWS C:\Users\xxxxx\.julia\packages\AWS\SchLh\src\AWS.jl:367
  [5] JSONService
    @ C:\Users\xxxxx\.julia\packages\AWS\SchLh\src\AWS.jl:356 [inlined]
  [6] get_secret_value(SecretId::String; aws_config::AWS.AWSConfig)
    @ myapp.Secrets_Manager C:\Users\xxxxx\.julia\packages\AWS\SchLh\src\services\secrets_manager.jl:465
  [7] get_secret_value(SecretId::String)
    @ myapp.Secrets_Manager C:\Users\xxxxx\.julia\packages\AWS\SchLh\src\services\secrets_manager.jl:464
  [8] top-level scope
    @ C:\Users\xxx\\src\dep.jl:87
  [9] include(mod::Module, _path::String)
    @ Base .\Base.jl:495
 [10] include(x::String)
    @ myapp C:\Users\xxxxx\src\myapp.jl:1      
 [11] top-level scope
    @ C:\Users\xxx\\src\be.jl:32
 [12] include(mod::Module, _path::String)
    @ Base .\Base.jl:495
 [13] _require(pkg::Base.PkgId, env::Nothing)
    @ Base .\loading.jl:2014
 [14] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base .\loading.jl:1812
 [15] #invokelatest#2
    @ .\essentials.jl:892 [inlined]
 [16] invokelatest
    @ .\essentials.jl:889 [inlined]
 [17] _require_prelocked
    @ .\loading.jl:1805 [inlined]
 [18] _require_prelocked
    @ .\loading.jl:1802 [inlined]
 [19] macro expansion
    @ .\lock.jl:267 [inlined]
 [20] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1797
 [21] top-level scope
⢰ [00m:22s] PackageCompiler: compiling incremental system image
1 Like
2

While the PackageCompiler approach might work, I think it might simpler to use Distributed.jl. The way I envision it:

  • you have a Julia project for each component of your app
  • you start the main Julia process
  • the main process spawns a subprocess for every component
  • the subprocess can simply activate its environment normally and load the code

I think this approach is simpler because you don’t need PackageCompiler.jl and using the methods from the other components is already implemented by Distributed.jl. This means you don’t have to think about communication and calling a component’s method is not much different from using a regular Julia function.

1 Like
3

Not sure if I understand it but when I say multiple projects it’s means that I have code for each package in different git repos. The App I am building will just get module name and parameters required to run as input and I should execute that module directly from my app.

Let’s say I have modules A, B,C and I want to execute the logic inside module “B” without adding as dependency into my app.

Can you please give me an example for distributed.jl for my scenario?

4

Ok so the way I envision this:

  • you have your main app running from somewhere in the filesystem
  • you then want to execute a function foo() from DifferentPackage

So you do, i.e. the main app does this:

  1. Clone/download the package DifferentPackage to some place e.g. /tmp/DifferentPackage. I am pretty sure there are ways to do this from within Julia but I have never done it myself.
  2. You spawn a worker using Distributed.addprocs(1) and store its worker id
  3. This worker activates the project using Pkg.activate(), instantiates it Pkg.instantiate() and then loads it with using DifferentPackage
  4. At this point you can access all methods from DifferentPackage by just using Distrobuted.@spawnat workerID foo()
1 Like
5

@abraemer Thank you for the explanation, The main disconnect for me is, with your approach the process will always try to compile every time it’s being loaded isn’t it? I don’t want this and that’s the reason I have thought to precompile it to create .so file and place it in AWS S3 bucket and have it available in cache.

7

I have never built a system image for such cases, so unfortunately, I do not have an informed opinion here.

8

Please don’t ping individuals — especially folks who aren’t already a part of the conversation here — just to ask them for their (free) help. Beyond just being bothersome to those you ping (and unlikely to work), it’s diminishes the help you’ve already gotten and is even likely to backfire: it’ll probably stifle others from jumping in on their own accord.

1 Like
9

Noted. Thanks!!

10

Thank you sir for your time, I apologize for any inconvenience that caused.