Handling of package inter-dependency

Well, this is a real concern ^^. I am not sure when it will happen, but I am sure it will happen at some point, so I’d like to be prepared when it happens.

The goal is indeed to make COM and SCI unaware of one another, but I’d rather not add another package, otherwise this defeats a bit the purpose of making SCI unaware of COM.

If you want to keep COM generic enough to not have to change it when adding a new SIC2 to the mix, I don’t think there’s a better alternative available. If you make SIC aware of COM, you’ll have to make SIC2 aware of COM as well. If you make COM aware of SIC, you’ll have to make it aware of SIC2 as well, and on top of that have to make sure that it doesn’t break compatibility with SIC when adding SIC2. That’s why I think the cleanest solution is to make sure COM is taking care of communication (and only communication) and have SIC take care of the calculation (and only the calculation). The way to connect the two would be through a third package, which you can extend and modify at will, when you have to add SIC2, without worrying about breaking the core functionality of either SIC or COM by accident.

I have to admit though, it feels kind of wrong to define structs on the fly based on messages from some remote process, perhaps there’s a better abstraction to be found here?

My first idea beyond the using COM(SCI) was to:

  • define a globale variable, say const SciPackage = :SCI
  • add an __init__ function to COM that would use SciPackage to load SCI and define the relevant authorization lists upon using COM

but I don’t know if this is a relevant strategy.

Supposing I manage to make COM and SCI unaware of each other. I still need to perform my computations between communication steps, so somewhere inside COM functions, I need a compute call that is very SCI-dependent… i don’t see how I can avoid this.

That sounds a lot like Requires.jl.

I think this is the point where more information about your true code structure is necessary for more help from my side. I’m still not sure I understood you correctly - are you really generating new structs (new types) at runtime based on messages from a remote process? Or are you only instantiating instances of existing structs?

At runtime I am only instanciating types already declared. Basically, if the server receives something that can be deserialized to an authorized (and declared of course) type, it mutates the appropriate field of a Container instance (the single one instantiated actually), that is of the same type.

Another way to see this is that I’d like to have an overall structure of code as

using SCI
const SciPackage = SCI
using COM
<some code>

where SCI defines a compute function and somewhere inside COM I have a line calling SciPackage.compute.

Then what I would need for SCI (as well as a potential SCI2) is to define some types like Container in addition to their standard types, as well as the compute function. In some way, that they define their interface to COM.

This is what I would like, but I am really not sure that works in julia

@Sukera

Sorry, I missed this part at first…

Just to be clear, the target types are already defined, I am just modifying instances here. If the data source were not trustworthy, that would be bad indeed, but this is not the case here, as everything happens locally. And anything that is not an authorized type/instruction is discarded.

Regarding the Container this choice is based on previous discussions, though I can’t recall where, probably on Slack (edit: it was there "Locally global" variable? - #19 by BambOoxX)… It was advertised as the most appropriate choice for this kind of stuff. What would you have in mind instead ?

After reading thoroughly the documentation of Requires, this seems to be the case indeed… that would still make COM somewhat aware of SCI, but in a very limited fashion.
Also as SCI would then be under COM.SCI this would allow to use things like const compute = SCI.compute so that I can keep using the same implementation in COM.

Some remark I made to myself earlier regarding your comment

I feel like it is better to make SCI aware of COM rather than the other way around:

  • If SCI defines some unnecessary stuff sometimes it’s not that important.
  • If SCI is not ready to handle stuff passed by COM, it seems fairly easy to detect (check the presence of a Container type and of compute function or else when firing up COM)
  • In terms of maintenance, if I modify the functions/types in SCI I will surely need to modify the container, but probably not what COM does so modifications are performed in the same package (and git repo for the matter).

Do you agree with this ?

Yes, that sounds like a good idea - I didn’t suggest it because it seemed like you wanted to keep SCI computation-only :slight_smile: In general, roughly following something like the OSI model when designing software is a good idea (i.e., the innermost layer is infrastructure/communications/libraries, wrapped by application logic and UI). In your case, you really have two applications - the frontend UI in C# and the backend in julia. Applying the above logic to the julia code is why I suggested having SCI and COM be oblivious of each other and instead have some glue package that takes care of plugging things together (e.g. by using both SCI and COM, passing compute to COM etc. We have higher order functions after all, so there’s no need for COM to know about SCI beforehand).

1 Like

Since there is no minimal working example (MWE), I could not figure out what you really wanted to do. (Please read: make it easier to help you)

However, I thought that possibly you would like to do something like this. Please take a look at the MWE below.

Input:

module SCI

export Problem, Algorithm_A, Algorithm_B, solve

abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end

abstract type AbstractAlgorithm end
Base.@kwdef struct Algorithm_A{T} <: AbstractAlgorithm a::T = 2.0 end
Base.@kwdef struct Algorithm_B{T} <: AbstractAlgorithm a::T = 3.0 end
default_algorithm(prob::Problem) = Algorithm_A()

struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::AbstractAlgorithm) = Solution(alg.a * prob.x, prob, alg)

"""
Here, `Base` module plays the role of `COM` package.
SCI module can define the method of 
`Base.show` (≈ `COM.show`) function for `SCI.Solution` type.
"""
function Base.show(io::IO, sol::Solution)
    result = """
    Problem:   $(sol.prob)
    Algorithm: $(sol.alg)
    Result:    $(sol.result)
    """
    print(io, result)
end

end

using .SCI
@show prob = Problem(1.5)
println()
println(solve(prob))
println(solve(prob, Algorithm_B()))

Output:

prob = Problem(1.5) = Problem{Float64}(1.5)

Problem:   Problem{Float64}(1.5)
Algorithm: Algorithm_A{Float64}(2.0)
Result:    3.0

Problem:   Problem{Float64}(1.5)
Algorithm: Algorithm_B{Float64}(3.0)
Result:    4.5

Additional input:

module SCI2 # extension of SCI module

export Problem2, Algorithm_C

using ..SCI: SCI, AbstractProblem, AbstractAlgorithm, Solution

struct Problem2{T} <: AbstractProblem x::T; y::T end
Base.@kwdef struct Algorithm_C{T} <: AbstractAlgorithm a::T = 2.0; b = 3.0 end
SCI.default_algorithm(prob::Problem2) = Algorithm_C()
SCI.solve(prob::Problem2, alg::Algorithm_C) = Solution(alg.a * prob.x + alg.b * prob.y, prob, alg)

end

using .SCI2
@show prob2 = Problem2(10.0, 1.0)
println()
println(solve(prob2))

Output:

prob2 = Problem2(10.0, 1.0) = Problem2{Float64}(10.0, 1.0)

Problem:   Problem2{Float64}(10.0, 1.0)
Algorithm: Algorithm_C{Float64}(2.0, 3.0)
Result:    23.0

In the above example I don’t define AbstractSolution type, nor did I make Solution a subtype of it. But you can define it, and then define another Solution type as a subtype of it in SCI2 module, and also define how to display objects of another Solution type. Many other extensions are also possible.

See also Function depending on the global variable inside module - #10 by genkuroki

I actually did not put an MWE, because I felt my question was somewhat abstract, and I did not want to introduce bias in the answers by forcing to be close to my current implementation.

Some comments about your MWE.

  • AFAICT you define the SCI package such that it patches the COM package is this right ? If so :
    What would happen if COM is not loaded prior to SC. There is probably a way to circumvent this issue by detecting the existence of Main.COM but still… (that looks like what Requires.jl handles as mentioned by Sukera)
  • In my my original post, I do not think I state that SCI2 is an extension of SCI, only that I would like to reuse the same communication functionalities of COM for both SCI (for which COM was initially designed) and SCI2

Anyway, I thank you for your effort.

Let’s say that I want it to remain computation-mostly ^^. SCI defines and exports some function for the user that themselves call the base unexported functions, so I figure it’s not that weird to define and interface for another package too…

As an excuse for my current developments, I am not a software engineer, I am an engineer that does some software (among other things). So keeping things as simple as possible is crucial :slight_smile:

I think I will

  • move anything that is currently SCI-specific inside COM to SCI
  • use Require (or something close) to make COM SCI-aware
  • setup some tests to ensure that SCI is communication ready

Thanks for the help !

Yes. It is a standard pattern.

Assume that, unlike the MWE I have shown above, both COM and SCI are packaged and can be loaded with using COM and using SCI. Then the result of using SCI; using COM and the result of using COM; using SCI will be the same.

In the following example, COM package has a function prettify(x) whose methods can be defined for user-defined types, and a function prettyprint(x) that uses prettify(x). SCI package makes use of them.

src/COM.jl of COM package:

"""
`COM.prettyprint(x)` prints nicely the object `x` for which `COM.prettify(x)` method is defined.
"""
module COM
prettify(x) = sprint(io -> show(io, "text/plain", x))
prettyprint(io::IO, x) = print(io, prettify(x))
prettyprint(x) = prettyprint(stdout, x)
end

Example of COM

using COM
COM.prettyprint([π, 2π])

Output:

2-element Vector{Float64}:
 3.141592653589793
 6.283185307179586

src/SCI.jl of SCI package (with deps COM):

"""A scirntific module (calculate exp(x))"""
module SCI

abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end

abstract type AbstractAlgorithm end
struct Builtin <: AbstractAlgorithm end
Base.@kwdef struct Taylor <: AbstractAlgorithm n::Int = 10 end
default_algorithm(prob::Problem) = Builtin()

struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::Builtin) = Solution(exp(prob.x), prob, alg)
solve(prob::AbstractProblem, alg::Taylor) = Solution(sum(prob.x^k/factorial(k) for k in 0:alg.n), prob, alg)

using COM

COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Builtin} = """
Problem:   x = $(sol.prob.x)
Algorithm: builtin exp(x)
Result:    $(sol.result)
"""

COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Taylor} = """
Problem:   x = $(sol.prob.x)
Algorithm: Taylor series of exp(x) upto degree $(sol.alg.n)
Result:    $(sol.result)
"""

end

Code to run:

# The order of the following two lines may be reversed.
using SCI
using COM

prob = SCI.Problem(1)
COM.prettyprint(SCI.solve(prob))
println()
COM.prettyprint(SCI.solve(prob, SCI.Taylor()))

Output:

Problem:   x = 1
Algorithm: builtin exp(x)
Result:    2.718281828459045

Problem:   x = 1
Algorithm: Taylor series of exp(x) upto degree 10
Result:    2.7182818011463845

In this way, you can use the pretty printing functions provided by COM package in any other package without changing any code in COM package.

A quick question about patching a module from another module.
I think that you can add some methods to an existing function, but you cannot add a function.

For instance when doing

module COM
    compute() = println("nothing to compute")
    showsomething() = println("something")
end

module SCI
    abstract type MyAbstractType end
    struct MyIntType <: MyAbstractType 
        val::Int64
    end
    using Main.COM
    COM.compute(mi::MyIntType) = 2*mi.val
end


using .COM
using .SCI

Calling methods(COM.compute) returns

# 2 methods for generic function “compute”:
[1] compute() in Main.COM at REPL[1]:2
[2] compute(mi::Main.SCI.MyIntType) in Main.SCI at REPL[2]

But if the compute in COM is absent, I get

ERROR: UndefVarError: compute not defined
Stacktrace:
[1] getproperty(x::Module, f::Symbol)
@ Base .\Base.jl:26
[2] top-level scope
@ REPL[9]:7

Can you confirm this ?

why would you want to add a function. If COM doesn’t have compute in itself, it wont call COM.compute anyway, so what’s the point of “adding” a compute into COM afterwards?

1 Like

Before this discussion I would have said “Sure, there is no point doing that”. However, given that you can add methods, I really don’t see why you could not add functions.

you’re not adding methods BACK to COM, you’re adding methods to the “method table” (in this session, let’s say) which does not belong to COM like a function would.

this is similar to asking “why can I modify a field of mutable struct but can’t add field?” because they are different kind of “modification”

1 Like

You can add functions, but that involves eval (which is almost always a very bad idea indeed) to force your code into the existing module (i.e. evaluate it in the context of the other module). If you’re thinking about doing this, you’re very likely on a not-good-not-happy path (especially since you’re basically parsing some input received from another process).

1 Like

So if I undestand corrently, the idea is to setup a blank compute function that would only serve as a default result provider.

Then whenever I have to integrate COM in some SCI package, I just add some methods to update what compute actually does based on that context.

Is that it ?

1 Like

that sounds pretty okay, spiritually similar to how abstract type works right?

1 Like

Yes, kind of, I am just not used to abstract julia stuff yet, I’m very much trying to remain concrete … :grinning_face_with_smiling_eyes: