Handling of package inter-dependency

You can add functions, but that involves eval (which is almost always a very bad idea indeed) to force your code into the existing module (i.e. evaluate it in the context of the other module). If you’re thinking about doing this, you’re very likely on a not-good-not-happy path (especially since you’re basically parsing some input received from another process).

1 Like

So if I undestand corrently, the idea is to setup a blank compute function that would only serve as a default result provider.

Then whenever I have to integrate COM in some SCI package, I just add some methods to update what compute actually does based on that context.

Is that it ?

1 Like

that sounds pretty okay, spiritually similar to how abstract type works right?

1 Like

Yes, kind of, I am just not used to abstract julia stuff yet, I’m very much trying to remain concrete … :grinning_face_with_smiling_eyes:

Let me emphasize again. The design I presented with simple-as-toys MWEs is quite standard.

In particular, it is very common to tell the Base module how to display objects of the user-defined type MyType by defining the Base.show(io::IO, x::MyType) method. See the document of Base.show at I/O and Network · The Julia Language .

In general, it is the most common design pattern to teach package A, which does some work for you, how to do that work for a type T defined in package B by adding a new method for the type T in package B to a function defined in package A.

Since there are many possible ways to design a code with multiple modules, it would be easier for the questioners to get useful comments on the design if they could give concrete minimal working examples.

Well, I can’t be very specific, but I will try to provide a more precise idea of what I’d like to achieve.

Let’s assume that the SCI package looks like

module SCI
export fun1,fun2
fun1(x) = x
fun2(x) = x^2
end

this package should work standalone, and should perform only specific computations.
Now I want to send computations requests, specifically the ones that SCI performs, from outside of julia.
I found my way to do this using sockets and other stuff. That’s where COM pops up (though I can’t make a MWE out of this), to translate a request from the outside world to something SCI can do

module COM
using SCI
# function to initialize a container
init_container() = []
 # function to read a message from a pipe, assume it always return `:fun1` here
read_message() = :fun1
# function to check if instruction is authorized

# function doing something based on the message received
function compute(container,msg)
    if msg == :fun1
          push!(container,fun1(rand))
    else if msg ==:fun2
          push!(container,fun2(rand))
    end
    return container
end
end

# Load COM+SCI utilities
using COM

container = init_container()
msg = read_message() 
container = compute(container,msg)

Obviously, compute needs to be somehow aware of SCI otherwise what’s the point.

I don’t know if it makes much of a difference, but a big aspect of the COM/SCI interaction, maybe something that I forgot to clearly state before, is that when I run COM, I only run it with a single SCI package at once.
So for instance if I have SCI and SCI2 I will start a task using COM+SCI and another task for COM+SCI2.
I NEVER have COM+SCI+SCI2 running at the same time.

Because of the current inter-dependency due to compute and other functionalities of the same kind, my COM should be called COM_SCI.
To get rid of SCI, I tried to use a global constant pointing towards the right SCI package e.g. CURRSCI = SCI and do

using SCI
global const CURRSCI = SCI
using COM
# where all SCI defined functions are now called with CURRSCI.funX ... 
...

but that doesn’t work, CURRSCI is not known inside COM. I also tried with import ..CURRSCI but it seems to fail too.

Long story short, I can’t figure how to share just enough information so that I do not have to maintain two (or more) versions of my COM package.

I tried the Requires approach and the patch strategy you provided. This results in a modification of SCI as

module SCI
using Requires
...
function __init__()
    @require COM="<some_id>" include("COMsetup.jl")
end      

where COMsetup contains patches for e.g. init_container,compute and others, but that fails too…

Frankly, I’m lost there, I do not see how to organize this in a versatile enough manner.
Sorry for the inconvenience if I’m too vague, but it’s quite unclear in my mind how to handle this.

Why not put fun1 and fun2 in e.g. a Dict with :fun1 etc. as the keys, pass that Dict to compute and retrieve the functions at run time? That way, COM is oblivous to both SCI and SCI2 (and vice versa). In my opinion, that would lead to the cleanest design, since you can check e.g. haskey before retrieving and dispatching. Or even use get to retrieve a default erroring function. You’re not getting rid of dynamic dispatch anyway, since which functions is going to be dispatched to depends on runtime values.

As I’ve mentioned above, julia has higher order functions and you can pass them around like all regular objects.

As this list is constant, it felt better to define it in a const. Passing it as an argument to the COM functions hence means to have some sort of surrounding package to handle this…

I am not sure what you mean by that.

That’s what I keep telling you to do :smiley: You can still avoid it if you really want, but it’s going to be more of a pain in the future when you’re extending this. E.g. one approach would be to have COM only define the interface for compute and then have SCI and SCI2 have their own methods for compute, which is dispatched to via some trait defined in those packages. Of course this is redundant if they always do the same thing (get some message, call some function), which is why I’m trying to tell you that it’s going to be more of a hassle to do it this way.

You can get all exported names of a module by calling (drumroll) names:

julia> module A           
                          
       export fun         
                          
       function fun()     
          "I'm a function"
       end                
       end                
Main.A                    
                          
julia> names(A)           
2-element Vector{Symbol}: 
 :A                       
 :fun                     

You’ll have to filter non-functions, but none of this information is hidden by any means. On the contrary, there is no data hiding in julia. Not in structs, not in modules. You can get it all via reflection and introspection. This is by design.

I’m a fan of “show, don’t tell”:

julia> example_fun(x) = "hello I'm a regular function with argument $x"
example_fun (generic function with 1 method)                           
                                                                       
julia> map(example_fun, [1, 2, 3])                                     
3-element Vector{String}:                                              
 "hello I'm a regular function with argument 1"                        
 "hello I'm a regular function with argument 2"                        
 "hello I'm a regular function with argument 3"

julia> funcs = [ example_fun, sum, map ]                                          
3-element Vector{Function}:                                                       
 example_fun (generic function with 1 method)                                     
 sum (generic function with 14 methods)                                           
 map (generic function with 60 methods)                                           
                                                                                  
julia> typeof.(funcs)                                                             
3-element Vector{DataType}:                                                       
 typeof(example_fun) (singleton type of function example_fun, subtype of Function)
 typeof(sum) (singleton type of function sum, subtype of Function)                
 typeof(map) (singleton type of function map, subtype of Function)                

It basically means that functions are first class objects. They have their own type, they can be passed into other functions - the whole nine yards. In that regard, they’re no different from numbers or structs.

Yeah, I see that ^^ (thanks for that). I just do not see how this approach helps right now.
I understand what you mean by passing functions now: compute should remain abstract and just apply a function argument to a list of arguments (I guess).
But I just can’t figure the syntax.
Right now my compute looks like

function compute(container,instruction)
if any(instruction .== list_of_authorized_instructions
     if instruction == fun1
          <do something with fun1>
     else if instruction == fun2
          <do something with fun2>
     end
else
    @error "e.g. instruction $instruction is not available in SCI"
end

what I can’t figure is how to remove the use of the funs in compute, but the solution is probably to put that in the famous third package :slight_smile: .

Maybe something like

function compute(container,field,fun::Function,ags::Vector{Any})
    container.field = fun(args...)
end

But it feels like this is only postponig the problem, as compute is nested in other functions of COM

I don’t like giving away answers if it means people can have a big “A-ha!” moment themselves, but here you go for one way how this could work:

error_fun(x...) = error("not applicable: '$x'")

function compute(container, instruction)
    # this check could be much more thorough of course
    f = get(container.functions, instruction, error_fun)
    f(container.args...) # do something with whichever f we get
end

This would live in COM. SCI as well as SCI2 need only supply a suitable container (or have their own compute which is dispatched to based on the type of container and COM only defines the interface, whichever you prefer).

That’s it! :slight_smile: (modulo some other customization steps of course, depending on which state you want to keep in container or pass explicitly to compute)

There again, not sure what you mean… :slight_smile:

I never used get but sometimes, the simplest things are also the most useful…

Thank you very much for showing us your MWE.

However, your MWE does not work as it is, so I have corrected it as follows.

module SCI
export fun1,fun2
fun1(x) = x
fun2(x) = x^2
end

module COM
using ..SCI
"""function to initialize a container"""
init_container() = []
"""function to read a message from a pipe, assume it reads from the argument for simplicity"""
read_message(x) = x
# function to check if instruction is authorized (omitted for simplicity)

"""function doing something based on the message received"""
function compute!(container, msg)
    if msg == :fun1
        push!(container, fun1(rand()))
    elseif msg ==:fun2
        push!(container, fun2(rand()))
    end
    return container
end
end

# Load COM+SCI utilities
using .COM

@show container = COM.init_container()
@show msg = COM.read_message(:fun1) 
@show container = COM.compute!(container, msg)
@show msg = COM.read_message(:fun2) 
@show container = COM.compute!(container, msg);

Output:

container = COM.init_container() = Any[]
msg = COM.read_message(:fun1) = :fun1
container = COM.compute!(container, msg) = Any[0.8424998776259705]
msg = COM.read_message(:fun2) = :fun2
container = COM.compute!(container, msg) = Any[0.8424998776259705, 0.043802504213398666]

You may move the compute! function to SCI package (or somewhere outside of COM) by defining the process!(compute!, container, msg) function in COM package. Then you are free to change the compute! function as you wish.

The MWE is the following.

module COM
"""function to initialize a container"""
init_container() = []
"""function to read a message from a pipe, assume it reads from the argument for simplicity"""
read_message(x) = x
# function to check if instruction is authorized (omitted for simplicity)

"""
function doing something based on the message received
compute! function will be defined elsewhere
"""
process!(compute!, container, msg) = compute!(container, msg)
end

module SCI
fun1(x) = x
fun2(x) = x^2

function compute!(container, msg)
    if msg == :fun1
        push!(container, fun1(rand()))
    elseif msg ==:fun2
        push!(container, fun2(rand()))
    end
    return container
end
end

@show compute! = SCI.compute!
@show container = COM.init_container()
@show msg = COM.read_message(:fun1) 
@show COM.process!(compute!, container, msg)
@show msg = COM.read_message(:fun2) 
@show COM.process!(compute!, container, msg);

Output:

compute! = SCI.compute! = Main.SCI.compute!
container = COM.init_container() = Any[]
msg = COM.read_message(:fun1) = :fun1
COM.process!(compute!, container, msg) = Any[0.619876019321522]
msg = COM.read_message(:fun2) = :fun2
COM.process!(compute!, container, msg) = Any[0.619876019321522, 0.40342923543449455]

Edit: Remove redundant container = 's.

The thing here is that you can specify manually that compute! = SCI.compute!. But as I said before, my compute is nested inside other functions in COM it is really the interface between both packages.
So I do not see how I can pass the information to COM : hey, you should use that particular compute if I set that particular setting.

To be more specific, let’s say that to make my server work I use the following functions, defined in COM:

abstract type AbstractContainer end
function start_server(servername::String)
    container = init_container()
    # Initialize named pipe server
    server = listen("\\\\.\\pipe\\$(servername)")
    while true
        # Read information from pipe server
        msg = read_message(server)
        # Handle message and update container accordingly
        container = handle_message!(container, msg)
    end
    return container

end
        
function handle_message!(container::AbstractContainer, msg::Union{<list of authorized types>})
        instruction = Symbol(msg.instruction)
        container = compute!(container, instruction)
    end
    return container
end

I have to tell start_server that I actually want a SCI-server so one way could be to pass SCI as an argument I guess.

abstract type AbstractContainer end
function start_server(servername::String,Package::Module)
    container = Package.init_container()
    # Initialize named pipe server
    server = listen("\\\\.\\pipe\\$(servername)")
    while true
        # Read information from pipe server
        msg = read_message(server)
        # Handle message and update container accordingly
        container = handle_message!(Package,container, msg)
    end
    return container

end
        
function handle_message!(Package::Module,container::AbstractContainer, msg::Union{<list of authorized types>})
        instruction = Symbol(msg.instruction)
        container = Package.compute!(container, instruction)
    end
    return container
end

Though I really do not know if it is something usual in julia…

Your code is not a minimal working example again. It does not work as it is. We need not only code for explanation but also MWE.

How much additional processing is COM actually doing? How much code is it?

Passing in the package is one way, yes, but I’d argue that you should pass in the container itself to start_server and have handle_message! be dispatched differently depending on that container. You’d have SCI and SCI2 have two distinct container types, which will lead to multiple dispatch selecting the correct handle_message! when calling.

E.g.

SCIInterface

module SCIInterface

export handle_message!

"""
   A function to handle a single message
"""
function handle_message! end

end

SCI:

module SCI

import .SCIInterface: handle_message!

export fun1, fun2, SCIContainer
struct SCIContainer
# fields
end

fun1() = "fun1"
fun2() = "fun2"

function handle_message!(container::SCIContainer, msg)
     "message is handled by SCI"
end
end

and COM:

module COM

using .SCIInterface

export start_server

function start_server(name, container)
    server = listen("\\\\.\\pipe\\$(servername)")
    while true
        msg = read_message(server)
        handle_message!(container, msg)
    end
    return container # not sure why you'd want that here
end

and then have your main entry point do something like

using SCI
using COM

container = SCIContainer()
start_server("my_server", container)

In start_server, which method is called depends on all input types, i.e. in this case on the type of container (which is SCIContainer here).

Now if you add a second module called SCI2 and it has a different container struct like

module SCI2

import .SCIInterface: handle_message!

export SCI2Con

struct SCI2Con
   my_field
   # and other fields
end

handle_message!(container::SCI2Con, msg) = "SCI2 here, welcome!"

and just swap using SCI to using SCI2 and set container = SCI2Con("one arg"), it will select the method from SCI2. This is because this construction defines one function handle_message! in SCIInterface, to which both SCI and SCI2 add a single method. Which of these two is selected depends solely on the type of its container argument. This is the power of controlling dispatch.

I know it is less than ideal, but I just can’t provide the whole thing. That’s on me, I am not making it easy for you, and I know you are just trying to help.

It handles communication (de-)serialization, type/instruction checking, logging and other stuff. Few hundreds of lines, with some bits I can’t share.

Once again, I learnt something here, I did not know one could declare a method-less function…

Minimal working example (MWE) does not need to include the whole thing. If it were, it would not be minimal.

In this discourse, it is very strongly recommended to show MWE. (“Do your best to make your example self-contained (“minimal working example”, MWE ), so that it runs (or gets to the error that you want help with) as is.” (Please read: make it easier to help you)) In fact, many people have followed it to help solve problems.

I find that I can learn a lot about julia by reading other peoples code :slight_smile: In this case, I recommend checking out the Tables.jl package I linked to earlier again (and reading its code). That package really does nothing more than define an interface and other packages (like DataFrames.jl for example) just depend on Tables.jl and implement the interface. Code that takes something Table-like then also only depends on Tables.jl and it’ll work with any package being compatible with/implementing the interface provided by the version of Tables.jl that’s the common denominator between all packages requiring it. It’s the exact same situation you’re in - you have a set of methods (e.g. compute) that is required by one package (COM) and a bunch of implementors (SCI, SCI2). The code joining the two is what I’ve been referring to as its own package, but really this can be a standalone file just as well.

2 Likes

Thanks, that is a good advice indeed. I think I’ve tried too many things, and now I’m lost in my own implementation. I have to let the information sink in and plan this further.

1 Like