Handling of package inter-dependency

New to Julia
,
1

Introduction

Hello. I am currently developing a software involving

  1. A GUI in C# / vb.net
  2. A computation kernel in julia

To handle to communication business I am using Json.net for the C# part and the BSON package in julia. The communication itself is performed over named Pipes as everything runs on the same local machine.

In my julia program, I have developed

  • a scientific package (let’s call it SCI) that handles the whole computation–heavy stuff, that can also be run autonomously
  • a server package (COM) that handles all the communication stuff including (de-)serialization, argument checking and so on. Once started, this server waits for instruction over the pipe, and calls some kind of compute command, or assigns values to variables when required.

Each upper-level function of SCI takes one or multiple arguments and returns only one.
Each of these functions has a specific return argument type (which is hence only used once).

Currently the process is the following:

  1. A message is received
  2. This message is converted to a struct based on types defined in SCI or an Instruction type defined in COM.
  3. Two cases :
    • If the object retrieved is a SCI-defined struct :
      • It is checked against a list of authorized types defined in COM based on knowledge of SCI
      • it is stored in the field of a mutable struct (Container) that is currently defined in COM
    • If the object is an instruction :
      • It is checked against a list of authorized instructions known from SCI
      • It is passed as an argument to a compute function that call function from SCI and store the result in Container
      • Special cases :
        • The instruction is exit, the server stops
        • The instruction is return_X, the server returns the value of variable X

The authorized types and authorized instructions are obtained from SCI thanks to the following bit of code in COM

using SCI
const auth_funs = filter(n -> eval(getfield(SCI, n)) isa Function, names(SCI))
const auth_types = filter(n -> getfield(SCI, n) isa Type, names(SCI))

The container type itself is defined manually.

My real question starts here

Obviously, if I were to only use these two packages, this state of development would be totally enough.
However, the COM package feels too much tied to the SCI package.
What if I wanted to use the same server, with another SCI2 package ?

What COM really requires is

  • To be able to load some package
  • To detect some properties of this package
  • To ensure that this package defines a compute function taking the appropriate arguments, and potentially a Container type

A very naive idea would then be to call something like using COM(SCI), however this is not possible.

So my question (finally) is : How to handle such dependency of one package versus another, dependency that is known at compile time (I believe,correct me if I’m wrong), but that would depend on some parameter defined in a const for instance, by the user.

The goal would be to make COM as much agnostic as possible to SCI so that COM does not depend on it. This point is kind of another question: if COM loads SCI, shouldn’t SCI be added to the environment of COM ? (which it is in my current implementation).

2

Is this a real concern or are you trying to account for some possible future use case that you’re not yet sure will actually occur in practice? If so, I’d make COM and SCI totally unaware of one another and create a third package/module/piece of code that ties these two together.

1 Like
3

Well, this is a real concern ^^. I am not sure when it will happen, but I am sure it will happen at some point, so I’d like to be prepared when it happens.

The goal is indeed to make COM and SCI unaware of one another, but I’d rather not add another package, otherwise this defeats a bit the purpose of making SCI unaware of COM.

4

If you want to keep COM generic enough to not have to change it when adding a new SIC2 to the mix, I don’t think there’s a better alternative available. If you make SIC aware of COM, you’ll have to make SIC2 aware of COM as well. If you make COM aware of SIC, you’ll have to make it aware of SIC2 as well, and on top of that have to make sure that it doesn’t break compatibility with SIC when adding SIC2. That’s why I think the cleanest solution is to make sure COM is taking care of communication (and only communication) and have SIC take care of the calculation (and only the calculation). The way to connect the two would be through a third package, which you can extend and modify at will, when you have to add SIC2, without worrying about breaking the core functionality of either SIC or COM by accident.

I have to admit though, it feels kind of wrong to define structs on the fly based on messages from some remote process, perhaps there’s a better abstraction to be found here?

5

My first idea beyond the using COM(SCI) was to:

  • define a globale variable, say const SciPackage = :SCI
  • add an __init__ function to COM that would use SciPackage to load SCI and define the relevant authorization lists upon using COM

but I don’t know if this is a relevant strategy.

Supposing I manage to make COM and SCI unaware of each other. I still need to perform my computations between communication steps, so somewhere inside COM functions, I need a compute call that is very SCI-dependent… i don’t see how I can avoid this.

6

That sounds a lot like Requires.jl.

I think this is the point where more information about your true code structure is necessary for more help from my side. I’m still not sure I understood you correctly - are you really generating new structs (new types) at runtime based on messages from a remote process? Or are you only instantiating instances of existing structs?

7

At runtime I am only instanciating types already declared. Basically, if the server receives something that can be deserialized to an authorized (and declared of course) type, it mutates the appropriate field of a Container instance (the single one instantiated actually), that is of the same type.

Another way to see this is that I’d like to have an overall structure of code as

using SCI
const SciPackage = SCI
using COM
<some code>

where SCI defines a compute function and somewhere inside COM I have a line calling SciPackage.compute.

Then what I would need for SCI (as well as a potential SCI2) is to define some types like Container in addition to their standard types, as well as the compute function. In some way, that they define their interface to COM.

This is what I would like, but I am really not sure that works in julia

8

@Sukera

Sorry, I missed this part at first…

Just to be clear, the target types are already defined, I am just modifying instances here. If the data source were not trustworthy, that would be bad indeed, but this is not the case here, as everything happens locally. And anything that is not an authorized type/instruction is discarded.

Regarding the Container this choice is based on previous discussions, though I can’t recall where, probably on Slack (edit: it was there "Locally global" variable? - #19 by BambOoxX)… It was advertised as the most appropriate choice for this kind of stuff. What would you have in mind instead ?

9

After reading thoroughly the documentation of Requires, this seems to be the case indeed… that would still make COM somewhat aware of SCI, but in a very limited fashion.
Also as SCI would then be under COM.SCI this would allow to use things like const compute = SCI.compute so that I can keep using the same implementation in COM.

Some remark I made to myself earlier regarding your comment

I feel like it is better to make SCI aware of COM rather than the other way around:

  • If SCI defines some unnecessary stuff sometimes it’s not that important.
  • If SCI is not ready to handle stuff passed by COM, it seems fairly easy to detect (check the presence of a Container type and of compute function or else when firing up COM)
  • In terms of maintenance, if I modify the functions/types in SCI I will surely need to modify the container, but probably not what COM does so modifications are performed in the same package (and git repo for the matter).

Do you agree with this ?

10

Yes, that sounds like a good idea - I didn’t suggest it because it seemed like you wanted to keep SCI computation-only :slight_smile: In general, roughly following something like the OSI model when designing software is a good idea (i.e., the innermost layer is infrastructure/communications/libraries, wrapped by application logic and UI). In your case, you really have two applications - the frontend UI in C# and the backend in julia. Applying the above logic to the julia code is why I suggested having SCI and COM be oblivious of each other and instead have some glue package that takes care of plugging things together (e.g. by using both SCI and COM, passing compute to COM etc. We have higher order functions after all, so there’s no need for COM to know about SCI beforehand).

1 Like
11

Since there is no minimal working example (MWE), I could not figure out what you really wanted to do. (Please read: make it easier to help you)

However, I thought that possibly you would like to do something like this. Please take a look at the MWE below.

Input:

module SCI

export Problem, Algorithm_A, Algorithm_B, solve

abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end

abstract type AbstractAlgorithm end
Base.@kwdef struct Algorithm_A{T} <: AbstractAlgorithm a::T = 2.0 end
Base.@kwdef struct Algorithm_B{T} <: AbstractAlgorithm a::T = 3.0 end
default_algorithm(prob::Problem) = Algorithm_A()

struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::AbstractAlgorithm) = Solution(alg.a * prob.x, prob, alg)

"""
Here, `Base` module plays the role of `COM` package.
SCI module can define the method of 
`Base.show` (≈ `COM.show`) function for `SCI.Solution` type.
"""
function Base.show(io::IO, sol::Solution)
    result = """
    Problem:   $(sol.prob)
    Algorithm: $(sol.alg)
    Result:    $(sol.result)
    """
    print(io, result)
end

end

using .SCI
@show prob = Problem(1.5)
println()
println(solve(prob))
println(solve(prob, Algorithm_B()))

Output:

prob = Problem(1.5) = Problem{Float64}(1.5)

Problem:   Problem{Float64}(1.5)
Algorithm: Algorithm_A{Float64}(2.0)
Result:    3.0

Problem:   Problem{Float64}(1.5)
Algorithm: Algorithm_B{Float64}(3.0)
Result:    4.5

Additional input:

module SCI2 # extension of SCI module

export Problem2, Algorithm_C

using ..SCI: SCI, AbstractProblem, AbstractAlgorithm, Solution

struct Problem2{T} <: AbstractProblem x::T; y::T end
Base.@kwdef struct Algorithm_C{T} <: AbstractAlgorithm a::T = 2.0; b = 3.0 end
SCI.default_algorithm(prob::Problem2) = Algorithm_C()
SCI.solve(prob::Problem2, alg::Algorithm_C) = Solution(alg.a * prob.x + alg.b * prob.y, prob, alg)

end

using .SCI2
@show prob2 = Problem2(10.0, 1.0)
println()
println(solve(prob2))

Output:

prob2 = Problem2(10.0, 1.0) = Problem2{Float64}(10.0, 1.0)

Problem:   Problem2{Float64}(10.0, 1.0)
Algorithm: Algorithm_C{Float64}(2.0, 3.0)
Result:    23.0

In the above example I don’t define AbstractSolution type, nor did I make Solution a subtype of it. But you can define it, and then define another Solution type as a subtype of it in SCI2 module, and also define how to display objects of another Solution type. Many other extensions are also possible.

See also Function depending on the global variable inside module - #10 by genkuroki

12

I actually did not put an MWE, because I felt my question was somewhat abstract, and I did not want to introduce bias in the answers by forcing to be close to my current implementation.

Some comments about your MWE.

  • AFAICT you define the SCI package such that it patches the COM package is this right ? If so :
    What would happen if COM is not loaded prior to SC. There is probably a way to circumvent this issue by detecting the existence of Main.COM but still… (that looks like what Requires.jl handles as mentioned by Sukera)
  • In my my original post, I do not think I state that SCI2 is an extension of SCI, only that I would like to reuse the same communication functionalities of COM for both SCI (for which COM was initially designed) and SCI2

Anyway, I thank you for your effort.

13

Let’s say that I want it to remain computation-mostly ^^. SCI defines and exports some function for the user that themselves call the base unexported functions, so I figure it’s not that weird to define and interface for another package too…

As an excuse for my current developments, I am not a software engineer, I am an engineer that does some software (among other things). So keeping things as simple as possible is crucial :slight_smile:

I think I will

  • move anything that is currently SCI-specific inside COM to SCI
  • use Require (or something close) to make COM SCI-aware
  • setup some tests to ensure that SCI is communication ready

Thanks for the help !

14

Yes. It is a standard pattern.

Assume that, unlike the MWE I have shown above, both COM and SCI are packaged and can be loaded with using COM and using SCI. Then the result of using SCI; using COM and the result of using COM; using SCI will be the same.

In the following example, COM package has a function prettify(x) whose methods can be defined for user-defined types, and a function prettyprint(x) that uses prettify(x). SCI package makes use of them.

src/COM.jl of COM package:

"""
`COM.prettyprint(x)` prints nicely the object `x` for which `COM.prettify(x)` method is defined.
"""
module COM
prettify(x) = sprint(io -> show(io, "text/plain", x))
prettyprint(io::IO, x) = print(io, prettify(x))
prettyprint(x) = prettyprint(stdout, x)
end

Example of COM

using COM
COM.prettyprint([π, 2π])

Output:

2-element Vector{Float64}:
 3.141592653589793
 6.283185307179586

src/SCI.jl of SCI package (with deps COM):

"""A scirntific module (calculate exp(x))"""
module SCI

abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end

abstract type AbstractAlgorithm end
struct Builtin <: AbstractAlgorithm end
Base.@kwdef struct Taylor <: AbstractAlgorithm n::Int = 10 end
default_algorithm(prob::Problem) = Builtin()

struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::Builtin) = Solution(exp(prob.x), prob, alg)
solve(prob::AbstractProblem, alg::Taylor) = Solution(sum(prob.x^k/factorial(k) for k in 0:alg.n), prob, alg)

using COM

COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Builtin} = """
Problem:   x = $(sol.prob.x)
Algorithm: builtin exp(x)
Result:    $(sol.result)
"""

COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Taylor} = """
Problem:   x = $(sol.prob.x)
Algorithm: Taylor series of exp(x) upto degree $(sol.alg.n)
Result:    $(sol.result)
"""

end

Code to run:

# The order of the following two lines may be reversed.
using SCI
using COM

prob = SCI.Problem(1)
COM.prettyprint(SCI.solve(prob))
println()
COM.prettyprint(SCI.solve(prob, SCI.Taylor()))

Output:

Problem:   x = 1
Algorithm: builtin exp(x)
Result:    2.718281828459045

Problem:   x = 1
Algorithm: Taylor series of exp(x) upto degree 10
Result:    2.7182818011463845

In this way, you can use the pretty printing functions provided by COM package in any other package without changing any code in COM package.

15

A quick question about patching a module from another module.
I think that you can add some methods to an existing function, but you cannot add a function.

For instance when doing

module COM
    compute() = println("nothing to compute")
    showsomething() = println("something")
end

module SCI
    abstract type MyAbstractType end
    struct MyIntType <: MyAbstractType 
        val::Int64
    end
    using Main.COM
    COM.compute(mi::MyIntType) = 2*mi.val
end


using .COM
using .SCI

Calling methods(COM.compute) returns

# 2 methods for generic function “compute”:
[1] compute() in Main.COM at REPL[1]:2
[2] compute(mi::Main.SCI.MyIntType) in Main.SCI at REPL[2]

But if the compute in COM is absent, I get

ERROR: UndefVarError: compute not defined
Stacktrace:
[1] getproperty(x::Module, f::Symbol)
@ Base .\Base.jl:26
[2] top-level scope
@ REPL[9]:7

Can you confirm this ?

16

why would you want to add a function. If COM doesn’t have compute in itself, it wont call COM.compute anyway, so what’s the point of “adding” a compute into COM afterwards?

1 Like
17

Before this discussion I would have said “Sure, there is no point doing that”. However, given that you can add methods, I really don’t see why you could not add functions.

18

you’re not adding methods BACK to COM, you’re adding methods to the “method table” (in this session, let’s say) which does not belong to COM like a function would.

this is similar to asking “why can I modify a field of mutable struct but can’t add field?” because they are different kind of “modification”

1 Like
19

You can add functions, but that involves eval (which is almost always a very bad idea indeed) to force your code into the existing module (i.e. evaluate it in the context of the other module). If you’re thinking about doing this, you’re very likely on a not-good-not-happy path (especially since you’re basically parsing some input received from another process).

1 Like
20

So if I undestand corrently, the idea is to setup a blank compute function that would only serve as a default result provider.

Then whenever I have to integrate COM in some SCI package, I just add some methods to update what compute actually does based on that context.

Is that it ?

1 Like