that sounds pretty okay, spiritually similar to how abstract type works right?
1 Like
Yes, kind of, I am just not used to abstract julia stuff yet, Iām very much trying to remain concrete ā¦ 
Let me emphasize again. The design I presented with simple-as-toys MWEs is quite standard.
In particular, it is very common to tell the Base module how to display objects of the user-defined type MyType by defining the Base.show(io::IO, x::MyType) method. See the document of Base.show at I/O and Network Ā· The Julia Language .
In general, it is the most common design pattern to teach package A, which does some work for you, how to do that work for a type T defined in package B by adding a new method for the type T in package B to a function defined in package A.
Since there are many possible ways to design a code with multiple modules, it would be easier for the questioners to get useful comments on the design if they could give concrete minimal working examples.
Well, I canāt be very specific, but I will try to provide a more precise idea of what Iād like to achieve.
Letās assume that the SCI package looks like
module SCI
export fun1,fun2
fun1(x) = x
fun2(x) = x^2
end
this package should work standalone, and should perform only specific computations.
Now I want to send computations requests, specifically the ones that SCI performs, from outside of julia.
I found my way to do this using sockets and other stuff. Thatās where COM pops up (though I canāt make a MWE out of this), to translate a request from the outside world to something SCI can do
module COM
using SCI
# function to initialize a container
init_container() = []
# function to read a message from a pipe, assume it always return `:fun1` here
read_message() = :fun1
# function to check if instruction is authorized
# function doing something based on the message received
function compute(container,msg)
if msg == :fun1
push!(container,fun1(rand))
else if msg ==:fun2
push!(container,fun2(rand))
end
return container
end
end
# Load COM+SCI utilities
using COM
container = init_container()
msg = read_message()
container = compute(container,msg)
Obviously, compute needs to be somehow aware of SCI otherwise whatās the point.
I donāt know if it makes much of a difference, but a big aspect of the COM/SCI interaction, maybe something that I forgot to clearly state before, is that when I run COM, I only run it with a single SCI package at once.
So for instance if I have SCI and SCI2 I will start a task using COM+SCI and another task for COM+SCI2.
I NEVER have COM+SCI+SCI2 running at the same time.
Because of the current inter-dependency due to compute and other functionalities of the same kind, my COM should be called COM_SCI.
To get rid of SCI, I tried to use a global constant pointing towards the right SCI package e.g. CURRSCI = SCI and do
using SCI
global const CURRSCI = SCI
using COM
# where all SCI defined functions are now called with CURRSCI.funX ...
...
but that doesnāt work, CURRSCI is not known inside COM. I also tried with import ..CURRSCI but it seems to fail too.
Long story short, I canāt figure how to share just enough information so that I do not have to maintain two (or more) versions of my COM package.
I tried the Requires approach and the patch strategy you provided. This results in a modification of SCI as
module SCI
using Requires
...
function __init__()
@require COM="<some_id>" include("COMsetup.jl")
end
where COMsetup contains patches for e.g. init_container,compute and others, but that fails tooā¦
Frankly, Iām lost there, I do not see how to organize this in a versatile enough manner.
Sorry for the inconvenience if Iām too vague, but itās quite unclear in my mind how to handle this.
Why not put fun1 and fun2 in e.g. a Dict with :fun1 etc. as the keys, pass that Dict to compute and retrieve the functions at run time? That way, COM is oblivous to both SCI and SCI2 (and vice versa). In my opinion, that would lead to the cleanest design, since you can check e.g. haskey before retrieving and dispatching. Or even use get to retrieve a default erroring function. Youāre not getting rid of dynamic dispatch anyway, since which functions is going to be dispatched to depends on runtime values.
As Iāve mentioned above, julia has higher order functions and you can pass them around like all regular objects.
As this list is constant, it felt better to define it in a const. Passing it as an argument to the COM functions hence means to have some sort of surrounding package to handle thisā¦
I am not sure what you mean by that.
Thatās what I keep telling you to do 
You can still avoid it if you really want, but itās going to be more of a pain in the future when youāre extending this. E.g. one approach would be to have COM only define the interface for compute and then have SCI and SCI2 have their own methods for compute, which is dispatched to via some trait defined in those packages. Of course this is redundant if they always do the same thing (get some message, call some function), which is why Iām trying to tell you that itās going to be more of a hassle to do it this way.
You can get all exported names of a module by calling (drumroll) names:
julia> module A
export fun
function fun()
"I'm a function"
end
end
Main.A
julia> names(A)
2-element Vector{Symbol}:
:A
:fun
Youāll have to filter non-functions, but none of this information is hidden by any means. On the contrary, there is no data hiding in julia. Not in structs, not in modules. You can get it all via reflection and introspection. This is by design.
Iām a fan of āshow, donāt tellā:
julia> example_fun(x) = "hello I'm a regular function with argument $x"
example_fun (generic function with 1 method)
julia> map(example_fun, [1, 2, 3])
3-element Vector{String}:
"hello I'm a regular function with argument 1"
"hello I'm a regular function with argument 2"
"hello I'm a regular function with argument 3"
julia> funcs = [ example_fun, sum, map ]
3-element Vector{Function}:
example_fun (generic function with 1 method)
sum (generic function with 14 methods)
map (generic function with 60 methods)
julia> typeof.(funcs)
3-element Vector{DataType}:
typeof(example_fun) (singleton type of function example_fun, subtype of Function)
typeof(sum) (singleton type of function sum, subtype of Function)
typeof(map) (singleton type of function map, subtype of Function)
It basically means that functions are first class objects. They have their own type, they can be passed into other functions - the whole nine yards. In that regard, theyāre no different from numbers or structs.
Yeah, I see that ^^ (thanks for that). I just do not see how this approach helps right now.
I understand what you mean by passing functions now: compute should remain abstract and just apply a function argument to a list of arguments (I guess).
But I just canāt figure the syntax.
Right now my compute looks like
function compute(container,instruction)
if any(instruction .== list_of_authorized_instructions
if instruction == fun1
<do something with fun1>
else if instruction == fun2
<do something with fun2>
end
else
@error "e.g. instruction $instruction is not available in SCI"
end
what I canāt figure is how to remove the use of the funs in compute, but the solution is probably to put that in the famous third package 
.
Maybe something like
function compute(container,field,fun::Function,ags::Vector{Any})
container.field = fun(args...)
end
But it feels like this is only postponig the problem, as compute is nested in other functions of COM
I donāt like giving away answers if it means people can have a big āA-ha!ā moment themselves, but here you go for one way how this could work:
error_fun(x...) = error("not applicable: '$x'")
function compute(container, instruction)
# this check could be much more thorough of course
f = get(container.functions, instruction, error_fun)
f(container.args...) # do something with whichever f we get
end
This would live in COM. SCI as well as SCI2 need only supply a suitable container (or have their own compute which is dispatched to based on the type of container and COM only defines the interface, whichever you prefer).
Thatās it! (modulo some other customization steps of course, depending on which state you want to keep in container or pass explicitly to compute)
There again, not sure what you meanā¦
I never used get but sometimes, the simplest things are also the most usefulā¦
Thank you very much for showing us your MWE.
However, your MWE does not work as it is, so I have corrected it as follows.
module SCI
export fun1,fun2
fun1(x) = x
fun2(x) = x^2
end
module COM
using ..SCI
"""function to initialize a container"""
init_container() = []
"""function to read a message from a pipe, assume it reads from the argument for simplicity"""
read_message(x) = x
# function to check if instruction is authorized (omitted for simplicity)
"""function doing something based on the message received"""
function compute!(container, msg)
if msg == :fun1
push!(container, fun1(rand()))
elseif msg ==:fun2
push!(container, fun2(rand()))
end
return container
end
end
# Load COM+SCI utilities
using .COM
@show container = COM.init_container()
@show msg = COM.read_message(:fun1)
@show container = COM.compute!(container, msg)
@show msg = COM.read_message(:fun2)
@show container = COM.compute!(container, msg);
Output:
container = COM.init_container() = Any[]
msg = COM.read_message(:fun1) = :fun1
container = COM.compute!(container, msg) = Any[0.8424998776259705]
msg = COM.read_message(:fun2) = :fun2
container = COM.compute!(container, msg) = Any[0.8424998776259705, 0.043802504213398666]
You may move the compute! function to SCI package (or somewhere outside of COM) by defining the process!(compute!, container, msg) function in COM package. Then you are free to change the compute! function as you wish.
The MWE is the following.
module COM
"""function to initialize a container"""
init_container() = []
"""function to read a message from a pipe, assume it reads from the argument for simplicity"""
read_message(x) = x
# function to check if instruction is authorized (omitted for simplicity)
"""
function doing something based on the message received
compute! function will be defined elsewhere
"""
process!(compute!, container, msg) = compute!(container, msg)
end
module SCI
fun1(x) = x
fun2(x) = x^2
function compute!(container, msg)
if msg == :fun1
push!(container, fun1(rand()))
elseif msg ==:fun2
push!(container, fun2(rand()))
end
return container
end
end
@show compute! = SCI.compute!
@show container = COM.init_container()
@show msg = COM.read_message(:fun1)
@show COM.process!(compute!, container, msg)
@show msg = COM.read_message(:fun2)
@show COM.process!(compute!, container, msg);
Output:
compute! = SCI.compute! = Main.SCI.compute!
container = COM.init_container() = Any[]
msg = COM.read_message(:fun1) = :fun1
COM.process!(compute!, container, msg) = Any[0.619876019321522]
msg = COM.read_message(:fun2) = :fun2
COM.process!(compute!, container, msg) = Any[0.619876019321522, 0.40342923543449455]
Edit: Remove redundant container = 's.
The thing here is that you can specify manually that compute! = SCI.compute!. But as I said before, my compute is nested inside other functions in COM it is really the interface between both packages.
So I do not see how I can pass the information to COM : hey, you should use that particular compute if I set that particular setting.
To be more specific, letās say that to make my server work I use the following functions, defined in COM:
abstract type AbstractContainer end
function start_server(servername::String)
container = init_container()
# Initialize named pipe server
server = listen("\\\\.\\pipe\\$(servername)")
while true
# Read information from pipe server
msg = read_message(server)
# Handle message and update container accordingly
container = handle_message!(container, msg)
end
return container
end
function handle_message!(container::AbstractContainer, msg::Union{<list of authorized types>})
instruction = Symbol(msg.instruction)
container = compute!(container, instruction)
end
return container
end
I have to tell start_server that I actually want a SCI-server so one way could be to pass SCI as an argument I guess.
abstract type AbstractContainer end
function start_server(servername::String,Package::Module)
container = Package.init_container()
# Initialize named pipe server
server = listen("\\\\.\\pipe\\$(servername)")
while true
# Read information from pipe server
msg = read_message(server)
# Handle message and update container accordingly
container = handle_message!(Package,container, msg)
end
return container
end
function handle_message!(Package::Module,container::AbstractContainer, msg::Union{<list of authorized types>})
instruction = Symbol(msg.instruction)
container = Package.compute!(container, instruction)
end
return container
end
Though I really do not know if it is something usual in juliaā¦
Your code is not a minimal working example again. It does not work as it is. We need not only code for explanation but also MWE.
How much additional processing is COM actually doing? How much code is it?
Passing in the package is one way, yes, but Iād argue that you should pass in the container itself to start_server and have handle_message! be dispatched differently depending on that container. Youād have SCI and SCI2 have two distinct container types, which will lead to multiple dispatch selecting the correct handle_message! when calling.
E.g.
SCIInterface
module SCIInterface
export handle_message!
"""
A function to handle a single message
"""
function handle_message! end
end
SCI:
module SCI
import .SCIInterface: handle_message!
export fun1, fun2, SCIContainer
struct SCIContainer
# fields
end
fun1() = "fun1"
fun2() = "fun2"
function handle_message!(container::SCIContainer, msg)
"message is handled by SCI"
end
end
and COM:
module COM
using .SCIInterface
export start_server
function start_server(name, container)
server = listen("\\\\.\\pipe\\$(servername)")
while true
msg = read_message(server)
handle_message!(container, msg)
end
return container # not sure why you'd want that here
end
and then have your main entry point do something like
using SCI
using COM
container = SCIContainer()
start_server("my_server", container)
In start_server, which method is called depends on all input types, i.e. in this case on the type of container (which is SCIContainer here).
Now if you add a second module called SCI2 and it has a different container struct like
module SCI2
import .SCIInterface: handle_message!
export SCI2Con
struct SCI2Con
my_field
# and other fields
end
handle_message!(container::SCI2Con, msg) = "SCI2 here, welcome!"
and just swap using SCI to using SCI2 and set container = SCI2Con("one arg"), it will select the method from SCI2. This is because this construction defines one function handle_message! in SCIInterface, to which both SCI and SCI2 add a single method. Which of these two is selected depends solely on the type of its container argument. This is the power of controlling dispatch.
I know it is less than ideal, but I just canāt provide the whole thing. Thatās on me, I am not making it easy for you, and I know you are just trying to help.
It handles communication (de-)serialization, type/instruction checking, logging and other stuff. Few hundreds of lines, with some bits I canāt share.
Once again, I learnt something here, I did not know one could declare a method-less functionā¦
Minimal working example (MWE) does not need to include the whole thing. If it were, it would not be minimal.
In this discourse, it is very strongly recommended to show MWE. (āDo your best to make your example self-contained (āminimal working exampleā, MWE ), so that it runs (or gets to the error that you want help with) as is.ā (Please read: make it easier to help you)) In fact, many people have followed it to help solve problems.
I find that I can learn a lot about julia by reading other peoples code In this case, I recommend checking out the Tables.jl package I linked to earlier again (and reading its code). That package really does nothing more than define an interface and other packages (like DataFrames.jl for example) just depend on Tables.jl and implement the interface. Code that takes something Table-like then also only depends on Tables.jl and itāll work with any package being compatible with/implementing the interface provided by the version of Tables.jl thatās the common denominator between all packages requiring it. Itās the exact same situation youāre in - you have a set of methods (e.g. compute) that is required by one package (COM) and a bunch of implementors (SCI, SCI2). The code joining the two is what Iāve been referring to as its own package, but really this can be a standalone file just as well.
2 Likes
Thanks, that is a good advice indeed. I think Iāve tried too many things, and now Iām lost in my own implementation. I have to let the information sink in and plan this further.
1 Like