That sounds a lot like Requires.jl.
I think this is the point where more information about your true code structure is necessary for more help from my side. I’m still not sure I understood you correctly - are you really generating new structs (new types) at runtime based on messages from a remote process? Or are you only instantiating instances of existing structs?
At runtime I am only instanciating types already declared. Basically, if the server receives something that can be deserialized to an authorized (and declared of course) type, it mutates the appropriate field of a Container instance (the single one instantiated actually), that is of the same type.
Another way to see this is that I’d like to have an overall structure of code as
using SCI
const SciPackage = SCI
using COM
<some code>
where SCI defines a compute function and somewhere inside COM I have a line calling SciPackage.compute.
Then what I would need for SCI (as well as a potential SCI2) is to define some types like Container in addition to their standard types, as well as the compute function. In some way, that they define their interface to COM.
This is what I would like, but I am really not sure that works in julia
Sorry, I missed this part at first…
Just to be clear, the target types are already defined, I am just modifying instances here. If the data source were not trustworthy, that would be bad indeed, but this is not the case here, as everything happens locally. And anything that is not an authorized type/instruction is discarded.
Regarding the Container this choice is based on previous discussions, though I can’t recall where, probably on Slack (edit: it was there "Locally global" variable? - #19 by BambOoxX)… It was advertised as the most appropriate choice for this kind of stuff. What would you have in mind instead ?
After reading thoroughly the documentation of Requires, this seems to be the case indeed… that would still make COM somewhat aware of SCI, but in a very limited fashion.
Also as SCI would then be under COM.SCI this would allow to use things like const compute = SCI.compute so that I can keep using the same implementation in COM.
Some remark I made to myself earlier regarding your comment
I feel like it is better to make SCI aware of COM rather than the other way around:
SCI defines some unnecessary stuff sometimes it’s not that important.SCI is not ready to handle stuff passed by COM, it seems fairly easy to detect (check the presence of a Container type and of compute function or else when firing up COM)SCI I will surely need to modify the container, but probably not what COM does so modifications are performed in the same package (and git repo for the matter).Do you agree with this ?
Yes, that sounds like a good idea - I didn’t suggest it because it seemed like you wanted to keep SCI computation-only 
In general, roughly following something like the OSI model when designing software is a good idea (i.e., the innermost layer is infrastructure/communications/libraries, wrapped by application logic and UI). In your case, you really have two applications - the frontend UI in C# and the backend in julia. Applying the above logic to the julia code is why I suggested having SCI and COM be oblivious of each other and instead have some glue package that takes care of plugging things together (e.g. by using both SCI and COM, passing compute to COM etc. We have higher order functions after all, so there’s no need for COM to know about SCI beforehand).
Since there is no minimal working example (MWE), I could not figure out what you really wanted to do. (Please read: make it easier to help you)
However, I thought that possibly you would like to do something like this. Please take a look at the MWE below.
Input:
module SCI
export Problem, Algorithm_A, Algorithm_B, solve
abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end
abstract type AbstractAlgorithm end
Base.@kwdef struct Algorithm_A{T} <: AbstractAlgorithm a::T = 2.0 end
Base.@kwdef struct Algorithm_B{T} <: AbstractAlgorithm a::T = 3.0 end
default_algorithm(prob::Problem) = Algorithm_A()
struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::AbstractAlgorithm) = Solution(alg.a * prob.x, prob, alg)
"""
Here, `Base` module plays the role of `COM` package.
SCI module can define the method of
`Base.show` (≈ `COM.show`) function for `SCI.Solution` type.
"""
function Base.show(io::IO, sol::Solution)
result = """
Problem: $(sol.prob)
Algorithm: $(sol.alg)
Result: $(sol.result)
"""
print(io, result)
end
end
using .SCI
@show prob = Problem(1.5)
println()
println(solve(prob))
println(solve(prob, Algorithm_B()))
Output:
prob = Problem(1.5) = Problem{Float64}(1.5)
Problem: Problem{Float64}(1.5)
Algorithm: Algorithm_A{Float64}(2.0)
Result: 3.0
Problem: Problem{Float64}(1.5)
Algorithm: Algorithm_B{Float64}(3.0)
Result: 4.5
Additional input:
module SCI2 # extension of SCI module
export Problem2, Algorithm_C
using ..SCI: SCI, AbstractProblem, AbstractAlgorithm, Solution
struct Problem2{T} <: AbstractProblem x::T; y::T end
Base.@kwdef struct Algorithm_C{T} <: AbstractAlgorithm a::T = 2.0; b = 3.0 end
SCI.default_algorithm(prob::Problem2) = Algorithm_C()
SCI.solve(prob::Problem2, alg::Algorithm_C) = Solution(alg.a * prob.x + alg.b * prob.y, prob, alg)
end
using .SCI2
@show prob2 = Problem2(10.0, 1.0)
println()
println(solve(prob2))
Output:
prob2 = Problem2(10.0, 1.0) = Problem2{Float64}(10.0, 1.0)
Problem: Problem2{Float64}(10.0, 1.0)
Algorithm: Algorithm_C{Float64}(2.0, 3.0)
Result: 23.0
In the above example I don’t define AbstractSolution type, nor did I make Solution a subtype of it. But you can define it, and then define another Solution type as a subtype of it in SCI2 module, and also define how to display objects of another Solution type. Many other extensions are also possible.
See also Function depending on the global variable inside module - #10 by genkuroki
I actually did not put an MWE, because I felt my question was somewhat abstract, and I did not want to introduce bias in the answers by forcing to be close to my current implementation.
Some comments about your MWE.
SCI package such that it patches the COM package is this right ? If so :COM is not loaded prior to SC. There is probably a way to circumvent this issue by detecting the existence of Main.COM but still… (that looks like what Requires.jl handles as mentioned by Sukera)SCI2 is an extension of SCI, only that I would like to reuse the same communication functionalities of COM for both SCI (for which COM was initially designed) and SCI2
Anyway, I thank you for your effort.
Let’s say that I want it to remain computation-mostly ^^. SCI defines and exports some function for the user that themselves call the base unexported functions, so I figure it’s not that weird to define and interface for another package too…
As an excuse for my current developments, I am not a software engineer, I am an engineer that does some software (among other things). So keeping things as simple as possible is crucial
I think I will
SCI-specific inside COM to SCI
COM SCI-awareSCI is communication ready
Thanks for the help !
Yes. It is a standard pattern.
Assume that, unlike the MWE I have shown above, both COM and SCI are packaged and can be loaded with using COM and using SCI. Then the result of using SCI; using COM and the result of using COM; using SCI will be the same.
In the following example, COM package has a function prettify(x) whose methods can be defined for user-defined types, and a function prettyprint(x) that uses prettify(x). SCI package makes use of them.
src/COM.jl of COM package:
"""
`COM.prettyprint(x)` prints nicely the object `x` for which `COM.prettify(x)` method is defined.
"""
module COM
prettify(x) = sprint(io -> show(io, "text/plain", x))
prettyprint(io::IO, x) = print(io, prettify(x))
prettyprint(x) = prettyprint(stdout, x)
end
Example of COM
using COM
COM.prettyprint([π, 2π])
Output:
2-element Vector{Float64}:
3.141592653589793
6.283185307179586
src/SCI.jl of SCI package (with deps COM):
"""A scirntific module (calculate exp(x))"""
module SCI
abstract type AbstractProblem end
struct Problem{T} <: AbstractProblem x::T end
abstract type AbstractAlgorithm end
struct Builtin <: AbstractAlgorithm end
Base.@kwdef struct Taylor <: AbstractAlgorithm n::Int = 10 end
default_algorithm(prob::Problem) = Builtin()
struct Solution{R, P<:AbstractProblem, A<:AbstractAlgorithm} result::R; prob::P; alg::A end
solve(prob::AbstractProblem) = solve(prob, default_algorithm(prob))
solve(prob::AbstractProblem, alg::Builtin) = Solution(exp(prob.x), prob, alg)
solve(prob::AbstractProblem, alg::Taylor) = Solution(sum(prob.x^k/factorial(k) for k in 0:alg.n), prob, alg)
using COM
COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Builtin} = """
Problem: x = $(sol.prob.x)
Algorithm: builtin exp(x)
Result: $(sol.result)
"""
COM.prettify(sol::Solution{R, P, A}) where {R, P<:AbstractProblem, A<:Taylor} = """
Problem: x = $(sol.prob.x)
Algorithm: Taylor series of exp(x) upto degree $(sol.alg.n)
Result: $(sol.result)
"""
end
Code to run:
# The order of the following two lines may be reversed.
using SCI
using COM
prob = SCI.Problem(1)
COM.prettyprint(SCI.solve(prob))
println()
COM.prettyprint(SCI.solve(prob, SCI.Taylor()))
Output:
Problem: x = 1
Algorithm: builtin exp(x)
Result: 2.718281828459045
Problem: x = 1
Algorithm: Taylor series of exp(x) upto degree 10
Result: 2.7182818011463845
In this way, you can use the pretty printing functions provided by COM package in any other package without changing any code in COM package.
A quick question about patching a module from another module.
I think that you can add some methods to an existing function, but you cannot add a function.
For instance when doing
module COM
compute() = println("nothing to compute")
showsomething() = println("something")
end
module SCI
abstract type MyAbstractType end
struct MyIntType <: MyAbstractType
val::Int64
end
using Main.COM
COM.compute(mi::MyIntType) = 2*mi.val
end
using .COM
using .SCI
Calling methods(COM.compute) returns
# 2 methods for generic function “compute”:
[1] compute() in Main.COM at REPL[1]:2
[2] compute(mi::Main.SCI.MyIntType) in Main.SCI at REPL[2]
But if the compute in COM is absent, I get
ERROR: UndefVarError: compute not defined
Stacktrace:
[1] getproperty(x::Module, f::Symbol)
@ Base .\Base.jl:26
[2] top-level scope
@ REPL[9]:7
Can you confirm this ?
why would you want to add a function. If COM doesn’t have compute in itself, it wont call COM.compute anyway, so what’s the point of “adding” a compute into COM afterwards?
Before this discussion I would have said “Sure, there is no point doing that”. However, given that you can add methods, I really don’t see why you could not add functions.
you’re not adding methods BACK to COM, you’re adding methods to the “method table” (in this session, let’s say) which does not belong to COM like a function would.
this is similar to asking “why can I modify a field of mutable struct but can’t add field?” because they are different kind of “modification”
You can add functions, but that involves eval (which is almost always a very bad idea indeed) to force your code into the existing module (i.e. evaluate it in the context of the other module). If you’re thinking about doing this, you’re very likely on a not-good-not-happy path (especially since you’re basically parsing some input received from another process).
So if I undestand corrently, the idea is to setup a blank compute function that would only serve as a default result provider.
Then whenever I have to integrate COM in some SCI package, I just add some methods to update what compute actually does based on that context.
Is that it ?
that sounds pretty okay, spiritually similar to how abstract type works right?
Yes, kind of, I am just not used to abstract julia stuff yet, I’m very much trying to remain concrete … 
Let me emphasize again. The design I presented with simple-as-toys MWEs is quite standard.
In particular, it is very common to tell the Base module how to display objects of the user-defined type MyType by defining the Base.show(io::IO, x::MyType) method. See the document of Base.show at I/O and Network · The Julia Language .
In general, it is the most common design pattern to teach package A, which does some work for you, how to do that work for a type T defined in package B by adding a new method for the type T in package B to a function defined in package A.
Since there are many possible ways to design a code with multiple modules, it would be easier for the questioners to get useful comments on the design if they could give concrete minimal working examples.
Well, I can’t be very specific, but I will try to provide a more precise idea of what I’d like to achieve.
Let’s assume that the SCI package looks like
module SCI
export fun1,fun2
fun1(x) = x
fun2(x) = x^2
end
this package should work standalone, and should perform only specific computations.
Now I want to send computations requests, specifically the ones that SCI performs, from outside of julia.
I found my way to do this using sockets and other stuff. That’s where COM pops up (though I can’t make a MWE out of this), to translate a request from the outside world to something SCI can do
module COM
using SCI
# function to initialize a container
init_container() = []
# function to read a message from a pipe, assume it always return `:fun1` here
read_message() = :fun1
# function to check if instruction is authorized
# function doing something based on the message received
function compute(container,msg)
if msg == :fun1
push!(container,fun1(rand))
else if msg ==:fun2
push!(container,fun2(rand))
end
return container
end
end
# Load COM+SCI utilities
using COM
container = init_container()
msg = read_message()
container = compute(container,msg)
Obviously, compute needs to be somehow aware of SCI otherwise what’s the point.
I don’t know if it makes much of a difference, but a big aspect of the COM/SCI interaction, maybe something that I forgot to clearly state before, is that when I run COM, I only run it with a single SCI package at once.
So for instance if I have SCI and SCI2 I will start a task using COM+SCI and another task for COM+SCI2.
I NEVER have COM+SCI+SCI2 running at the same time.
Because of the current inter-dependency due to compute and other functionalities of the same kind, my COM should be called COM_SCI.
To get rid of SCI, I tried to use a global constant pointing towards the right SCI package e.g. CURRSCI = SCI and do
using SCI
global const CURRSCI = SCI
using COM
# where all SCI defined functions are now called with CURRSCI.funX ...
...
but that doesn’t work, CURRSCI is not known inside COM. I also tried with import ..CURRSCI but it seems to fail too.
Long story short, I can’t figure how to share just enough information so that I do not have to maintain two (or more) versions of my COM package.
I tried the Requires approach and the patch strategy you provided. This results in a modification of SCI as
module SCI
using Requires
...
function __init__()
@require COM="<some_id>" include("COMsetup.jl")
end
where COMsetup contains patches for e.g. init_container,compute and others, but that fails too…
Frankly, I’m lost there, I do not see how to organize this in a versatile enough manner.
Sorry for the inconvenience if I’m too vague, but it’s quite unclear in my mind how to handle this.
Why not put fun1 and fun2 in e.g. a Dict with :fun1 etc. as the keys, pass that Dict to compute and retrieve the functions at run time? That way, COM is oblivous to both SCI and SCI2 (and vice versa). In my opinion, that would lead to the cleanest design, since you can check e.g. haskey before retrieving and dispatching. Or even use get to retrieve a default erroring function. You’re not getting rid of dynamic dispatch anyway, since which functions is going to be dispatched to depends on runtime values.
As I’ve mentioned above, julia has higher order functions and you can pass them around like all regular objects.