Handling of package inter-dependency

That sounds a lot like Requires.jl.

I think this is the point where more information about your true code structure is necessary for more help from my side. I’m still not sure I understood you correctly - are you really generating new structs (new types) at runtime based on messages from a remote process? Or are you only instantiating instances of existing structs?

At runtime I am only instanciating types already declared. Basically, if the server receives something that can be deserialized to an authorized (and declared of course) type, it mutates the appropriate field of a Container instance (the single one instantiated actually), that is of the same type.

Another way to see this is that I’d like to have an overall structure of code as

using SCI
const SciPackage = SCI
using COM
<some code>

where SCI defines a compute function and somewhere inside COM I have a line calling SciPackage.compute.

Then what I would need for SCI (as well as a potential SCI2) is to define some types like Container in addition to their standard types, as well as the compute function. In some way, that they define their interface to COM.

This is what I would like, but I am really not sure that works in julia

@Sukera

Sorry, I missed this part at first…

Just to be clear, the target types are already defined, I am just modifying instances here. If the data source were not trustworthy, that would be bad indeed, but this is not the case here, as everything happens locally. And anything that is not an authorized type/instruction is discarded.

Regarding the Container this choice is based on previous discussions, though I can’t recall where, probably on Slack (edit: it was there "Locally global" variable? - #19 by BambOoxX)… It was advertised as the most appropriate choice for this kind of stuff. What would you have in mind instead ?

After reading thoroughly the documentation of Requires, this seems to be the case indeed… that would still make COM somewhat aware of SCI, but in a very limited fashion.
Also as SCI would then be under COM.SCI this would allow to use things like const compute = SCI.compute so that I can keep using the same implementation in COM.

Some remark I made to myself earlier regarding your comment

I feel like it is better to make SCI aware of COM rather than the other way around:

  • If SCI defines some unnecessary stuff sometimes it’s not that important.
  • If SCI is not ready to handle stuff passed by COM, it seems fairly easy to detect (check the presence of a Container type and of compute function or else when firing up COM)
  • In terms of maintenance, if I modify the functions/types in SCI I will surely need to modify the container, but probably not what COM does so modifications are performed in the same package (and git repo for the matter).

Do you agree with this ?

Yes, that sounds like a good idea - I didn’t suggest it because it seemed like you wanted to keep SCI computation-only :slight_smile: In general, roughly following something like the OSI model when designing software is a good idea (i.e., the innermost layer is infrastructure/communications/libraries, wrapped by application logic and UI). In your case, you really have two applications - the frontend UI in C# and the backend in julia. Applying the above logic to the julia code is why I suggested having SCI and COM be oblivious of each other and instead have some glue package that takes care of plugging things together (e.g. by using both SCI and COM, passing compute to COM etc. We have higher order functions after all, so there’s no need for COM to know about SCI beforehand).

1 Like

Since there is no minimal working example (MWE), I could not figure out what you really wanted to do. (Please read: make it easier to help you)

However, I thought that possibly you would like to do something like this. Please take a look at the MWE below.

Input:

module SCI

export Problem, Algorithm_A, Algorithm_B, solve

abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end

abstract type AbstractAlgorithm end
Base.@kwdef struct Algorithm_A{T} <: AbstractAlgorithm a::T = 2.0 end
Base.@kwdef struct Algorithm_B{T} <: AbstractAlgorithm a::T = 3.0 end
default_algorithm(prob::Problem) = Algorithm_A()

struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::AbstractAlgorithm) = Solution(alg.a * prob.x, prob, alg)

"""
Here, `Base` module plays the role of `COM` package.
SCI module can define the method of 
`Base.show` (≈ `COM.show`) function for `SCI.Solution` type.
"""
function Base.show(io::IO, sol::Solution)
    result = """
    Problem:   $(sol.prob)
    Algorithm: $(sol.alg)
    Result:    $(sol.result)
    """
    print(io, result)
end

end

using .SCI
@show prob = Problem(1.5)
println()
println(solve(prob))
println(solve(prob, Algorithm_B()))

Output:

prob = Problem(1.5) = Problem{Float64}(1.5)

Problem:   Problem{Float64}(1.5)
Algorithm: Algorithm_A{Float64}(2.0)
Result:    3.0

Problem:   Problem{Float64}(1.5)
Algorithm: Algorithm_B{Float64}(3.0)
Result:    4.5

Additional input:

module SCI2 # extension of SCI module

export Problem2, Algorithm_C

using ..SCI: SCI, AbstractProblem, AbstractAlgorithm, Solution

struct Problem2{T} <: AbstractProblem x::T; y::T end
Base.@kwdef struct Algorithm_C{T} <: AbstractAlgorithm a::T = 2.0; b = 3.0 end
SCI.default_algorithm(prob::Problem2) = Algorithm_C()
SCI.solve(prob::Problem2, alg::Algorithm_C) = Solution(alg.a * prob.x + alg.b * prob.y, prob, alg)

end

using .SCI2
@show prob2 = Problem2(10.0, 1.0)
println()
println(solve(prob2))

Output:

prob2 = Problem2(10.0, 1.0) = Problem2{Float64}(10.0, 1.0)

Problem:   Problem2{Float64}(10.0, 1.0)
Algorithm: Algorithm_C{Float64}(2.0, 3.0)
Result:    23.0

In the above example I don’t define AbstractSolution type, nor did I make Solution a subtype of it. But you can define it, and then define another Solution type as a subtype of it in SCI2 module, and also define how to display objects of another Solution type. Many other extensions are also possible.

See also Function depending on the global variable inside module - #10 by genkuroki

I actually did not put an MWE, because I felt my question was somewhat abstract, and I did not want to introduce bias in the answers by forcing to be close to my current implementation.

Some comments about your MWE.

  • AFAICT you define the SCI package such that it patches the COM package is this right ? If so :
    What would happen if COM is not loaded prior to SC. There is probably a way to circumvent this issue by detecting the existence of Main.COM but still… (that looks like what Requires.jl handles as mentioned by Sukera)
  • In my my original post, I do not think I state that SCI2 is an extension of SCI, only that I would like to reuse the same communication functionalities of COM for both SCI (for which COM was initially designed) and SCI2

Anyway, I thank you for your effort.

Let’s say that I want it to remain computation-mostly ^^. SCI defines and exports some function for the user that themselves call the base unexported functions, so I figure it’s not that weird to define and interface for another package too…

As an excuse for my current developments, I am not a software engineer, I am an engineer that does some software (among other things). So keeping things as simple as possible is crucial :slight_smile:

I think I will

  • move anything that is currently SCI-specific inside COM to SCI
  • use Require (or something close) to make COM SCI-aware
  • setup some tests to ensure that SCI is communication ready

Thanks for the help !

Yes. It is a standard pattern.

Assume that, unlike the MWE I have shown above, both COM and SCI are packaged and can be loaded with using COM and using SCI. Then the result of using SCI; using COM and the result of using COM; using SCI will be the same.

In the following example, COM package has a function prettify(x) whose methods can be defined for user-defined types, and a function prettyprint(x) that uses prettify(x). SCI package makes use of them.

src/COM.jl of COM package:

"""
`COM.prettyprint(x)` prints nicely the object `x` for which `COM.prettify(x)` method is defined.
"""
module COM
prettify(x) = sprint(io -> show(io, "text/plain", x))
prettyprint(io::IO, x) = print(io, prettify(x))
prettyprint(x) = prettyprint(stdout, x)
end

Example of COM

using COM
COM.prettyprint([π, 2π])

Output:

2-element Vector{Float64}:
 3.141592653589793
 6.283185307179586

src/SCI.jl of SCI package (with deps COM):

"""A scirntific module (calculate exp(x))"""
module SCI

abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end

abstract type AbstractAlgorithm end
struct Builtin <: AbstractAlgorithm end
Base.@kwdef struct Taylor <: AbstractAlgorithm n::Int = 10 end
default_algorithm(prob::Problem) = Builtin()

struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::Builtin) = Solution(exp(prob.x), prob, alg)
solve(prob::AbstractProblem, alg::Taylor) = Solution(sum(prob.x^k/factorial(k) for k in 0:alg.n), prob, alg)

using COM

COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Builtin} = """
Problem:   x = $(sol.prob.x)
Algorithm: builtin exp(x)
Result:    $(sol.result)
"""

COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Taylor} = """
Problem:   x = $(sol.prob.x)
Algorithm: Taylor series of exp(x) upto degree $(sol.alg.n)
Result:    $(sol.result)
"""

end

Code to run:

# The order of the following two lines may be reversed.
using SCI
using COM

prob = SCI.Problem(1)
COM.prettyprint(SCI.solve(prob))
println()
COM.prettyprint(SCI.solve(prob, SCI.Taylor()))

Output:

Problem:   x = 1
Algorithm: builtin exp(x)
Result:    2.718281828459045

Problem:   x = 1
Algorithm: Taylor series of exp(x) upto degree 10
Result:    2.7182818011463845

In this way, you can use the pretty printing functions provided by COM package in any other package without changing any code in COM package.

A quick question about patching a module from another module.
I think that you can add some methods to an existing function, but you cannot add a function.

For instance when doing

module COM
    compute() = println("nothing to compute")
    showsomething() = println("something")
end

module SCI
    abstract type MyAbstractType end
    struct MyIntType <: MyAbstractType 
        val::Int64
    end
    using Main.COM
    COM.compute(mi::MyIntType) = 2*mi.val
end


using .COM
using .SCI

Calling methods(COM.compute) returns

# 2 methods for generic function “compute”:
[1] compute() in Main.COM at REPL[1]:2
[2] compute(mi::Main.SCI.MyIntType) in Main.SCI at REPL[2]

But if the compute in COM is absent, I get

ERROR: UndefVarError: compute not defined
Stacktrace:
[1] getproperty(x::Module, f::Symbol)
@ Base .\Base.jl:26
[2] top-level scope
@ REPL[9]:7

Can you confirm this ?

why would you want to add a function. If COM doesn’t have compute in itself, it wont call COM.compute anyway, so what’s the point of “adding” a compute into COM afterwards?

1 Like

Before this discussion I would have said “Sure, there is no point doing that”. However, given that you can add methods, I really don’t see why you could not add functions.

you’re not adding methods BACK to COM, you’re adding methods to the “method table” (in this session, let’s say) which does not belong to COM like a function would.

this is similar to asking “why can I modify a field of mutable struct but can’t add field?” because they are different kind of “modification”

1 Like

You can add functions, but that involves eval (which is almost always a very bad idea indeed) to force your code into the existing module (i.e. evaluate it in the context of the other module). If you’re thinking about doing this, you’re very likely on a not-good-not-happy path (especially since you’re basically parsing some input received from another process).

1 Like

So if I undestand corrently, the idea is to setup a blank compute function that would only serve as a default result provider.

Then whenever I have to integrate COM in some SCI package, I just add some methods to update what compute actually does based on that context.

Is that it ?

1 Like

that sounds pretty okay, spiritually similar to how abstract type works right?

1 Like

Yes, kind of, I am just not used to abstract julia stuff yet, I’m very much trying to remain concrete … :grinning_face_with_smiling_eyes:

Let me emphasize again. The design I presented with simple-as-toys MWEs is quite standard.

In particular, it is very common to tell the Base module how to display objects of the user-defined type MyType by defining the Base.show(io::IO, x::MyType) method. See the document of Base.show at I/O and Network · The Julia Language .

In general, it is the most common design pattern to teach package A, which does some work for you, how to do that work for a type T defined in package B by adding a new method for the type T in package B to a function defined in package A.

Since there are many possible ways to design a code with multiple modules, it would be easier for the questioners to get useful comments on the design if they could give concrete minimal working examples.

Well, I can’t be very specific, but I will try to provide a more precise idea of what I’d like to achieve.

Let’s assume that the SCI package looks like

module SCI
export fun1,fun2
fun1(x) = x
fun2(x) = x^2
end

this package should work standalone, and should perform only specific computations.
Now I want to send computations requests, specifically the ones that SCI performs, from outside of julia.
I found my way to do this using sockets and other stuff. That’s where COM pops up (though I can’t make a MWE out of this), to translate a request from the outside world to something SCI can do

module COM
using SCI
# function to initialize a container
init_container() = []
 # function to read a message from a pipe, assume it always return `:fun1` here
read_message() = :fun1
# function to check if instruction is authorized

# function doing something based on the message received
function compute(container,msg)
    if msg == :fun1
          push!(container,fun1(rand))
    else if msg ==:fun2
          push!(container,fun2(rand))
    end
    return container
end
end

# Load COM+SCI utilities
using COM

container = init_container()
msg = read_message() 
container = compute(container,msg)

Obviously, compute needs to be somehow aware of SCI otherwise what’s the point.

I don’t know if it makes much of a difference, but a big aspect of the COM/SCI interaction, maybe something that I forgot to clearly state before, is that when I run COM, I only run it with a single SCI package at once.
So for instance if I have SCI and SCI2 I will start a task using COM+SCI and another task for COM+SCI2.
I NEVER have COM+SCI+SCI2 running at the same time.

Because of the current inter-dependency due to compute and other functionalities of the same kind, my COM should be called COM_SCI.
To get rid of SCI, I tried to use a global constant pointing towards the right SCI package e.g. CURRSCI = SCI and do

using SCI
global const CURRSCI = SCI
using COM
# where all SCI defined functions are now called with CURRSCI.funX ... 
...

but that doesn’t work, CURRSCI is not known inside COM. I also tried with import ..CURRSCI but it seems to fail too.

Long story short, I can’t figure how to share just enough information so that I do not have to maintain two (or more) versions of my COM package.

I tried the Requires approach and the patch strategy you provided. This results in a modification of SCI as

module SCI
using Requires
...
function __init__()
    @require COM="<some_id>" include("COMsetup.jl")
end      

where COMsetup contains patches for e.g. init_container,compute and others, but that fails too…

Frankly, I’m lost there, I do not see how to organize this in a versatile enough manner.
Sorry for the inconvenience if I’m too vague, but it’s quite unclear in my mind how to handle this.

Why not put fun1 and fun2 in e.g. a Dict with :fun1 etc. as the keys, pass that Dict to compute and retrieve the functions at run time? That way, COM is oblivous to both SCI and SCI2 (and vice versa). In my opinion, that would lead to the cleanest design, since you can check e.g. haskey before retrieving and dispatching. Or even use get to retrieve a default erroring function. You’re not getting rid of dynamic dispatch anyway, since which functions is going to be dispatched to depends on runtime values.

As I’ve mentioned above, julia has higher order functions and you can pass them around like all regular objects.