Architecture: general module and specific module

New to Julia
1

I guess I am lacking knowledge of design patterns,
can someone suggest the right solution (common in Julia) to set of specific modules that reexport general module functionality with one argument inserted?

The general module defines functions that depend on a state (require the state as an input).
I what to have a set of specific modules that do not require typing the state explicitly.

module General

export state, st1, st2
struct state{T}
   par1::T
   par2::T
end

st1 = state(1.1, 2.2)
st2 = state(2.2, 3.3)

# methods
export calculation, someothercalculation
calculation(x, s::state) = 2x * s.par1 + s.par2
someothercalculation(x, s::state) = 3x * s.par1 + s.par2

end

The specific modules implement the same methods for specific states, these are now for single-argument.

module Specific1

using Reexport
@reexport using General

export st
st = st1

import General: calculation, someothercalculation
calculation(x) = calculation(x,st)
someothercalculation(x) = someothercalculation(x,st)

end

to create Specific2 I actually would duplicate the Specific1 code except for the st = st1 line,
that become st = st2.

There must be a better way.

2

I think I donā€™t fully understand your use cases for Specific1 and Specific2. Do they both operate on the same types and define the same functions? Are they just ā€œworkspacesā€ to perform calculations on a single instance of State{Float64}?

1 Like
3

ah, yes, I oversimplified, perhaps. The point is that a lot of functionality is shared between Specific1 and Specific2, but they might contain some different functions.

The use cases are as follows:

file: my_first_analysis.jl 

using Specific1

data = load_sample()
results  = calculation.(data)

@save "results1.jld2" results

file: my_second_analysis.jl 

using Specific2

data = load_sample()
results  = calculation.(data)

@save "results2.jld2" results
4

So to see if I understand correctly, Specific1 exports calculations, which is defined as

# where `Specific1.st1` is some global const.
Specific1.calculation(x) = General.calculation(x, st1)

This doesnā€™t seem like a case where you want two different modules. Rather you should use a function to do the work, and perhaps multiple dispatch or higher order functions as necessary. Consider, for example:

using General: calculation, State

# Returns a function that wraps st. Maybe this is 
# incorporated into the General api if very common
# and necessary?
make_specific(f, st) = x -> f(x, st)

# used as in
data = load_data()
st1 = State(1.1, 2.2)
calc_st1 = make_specific(calculation, st1)

results = calc_st1.(data)
@save "results1.jld2" results

Alternatively, you could wrap all the state-dependent computation you want to in a function that takes in the state. For example,

using General: calculation, State

function do_the_thing(st, data, filename)
    # using Ref to broadcast the entire `st` with 
    # each element of data. Maybe this was holding you back?
    results = calculation.(data, Ref(st))
    @save "$filename.jld2" results   
end

#used as in:
data = load_data()
do_the_thing(State(1.1, 2.2), data, "results1")

One of these two is probably the design paradigm you want.

1 Like
5

thanks for the thoughts, that is useful.

the problem is that I have ~100 functions that need to be redefined for a specific state.
I want to hide this redefinitions to a module that can be easily imported (it cannot be the general one).

Once I am done with the analysis of the st1. I want to move on to the analysis of st2 without duplicating redefinition code.

6

I donā€™t know the details of your problem, so maybe this is justified, but it looks like a code smell.

When designed idiomatically, even complex APIs in Julia have about 5ā€“15 core functions, and a couple more optional ones for performance optimizations etc.

1 Like
7

The second approach with do_the_thing(st, ...) is a good one.
While the thing gets too specific with many dependencies, so it is hard to make a good function out of this.

8

The first part suggests you may need a (slight) refactoring. The second is a sentiment I strongly agree with!

For example, if calculation is already defined in General as

calculation(x, st) = # something that uses st and x together

then, strictly speaking, a version calculations(x) may be 1. not necessary, 2. more trouble than itā€™s worth. You already have a perfectly general function that can handle arbitrary states!

The user of General (you, if youā€™re running the analyses yourself) can simply call a function (or several ā€“ but certainly not 100) like do_the_thing that pass the state as far into the callstack as it needs to go.

1 Like
9

The state in the previous example is something fundamental that you would like to fix once and use throughout the whole analysis. I would have it const defined once on the general module.

I am trying to come up with a good analogy for my real case.
Letā€™s say, I study the dynamics of falling objects on the Earth. My state contains the gravitational acceleration, speed of sound, air viscosity (I am making things up).

All calculations depend on these parameters or their derivatives, but I do not want to pull them as an argument. I am never interested in how results depend on these parameters because I cannot change them.

I have got a similar project on Venus, I need to do almost the same calculations, with different fundamental parameters.

Do you see what I mean?

10

Sure, I think so. Should I assume the ā€œstateā€ is (or could/should be) a global const? If so, you could make it e.g. a mutable struct (or a dict, etc.) so that its parameters can be redefined if needed. For example

# inside General

const STATE = Dict(:g => 9.8)

some_calc(x) = 0.5 * x * STATE[:g]^2

# if you already have a version with st, like you said, 
# you can have global state as the default, with optional
# second argument
some_calc(x, st = STATE) = 0.5 * x * st[:g]^2

Then General can also define a method for overwriting the state in case the user needs it to change.

function edit_global_state!(;kwargs...)
    for (k, v) in kwargs
        STATE[k] = v
    end
end

# allows:
#   edit_global_state!(g = 11.23)
# to set :g

By the way, the other paradigms I mentioned would still work just fine in the astrodynamics case. For example, a user could do

# let's say Earth is exported for user convenience:
some_calc(parameters, Earth) 
# can also set Earth to be default positional:
some_calc(parameters)  

some_calc(parameters, General.Jupiter) # not exported
some_calc(parameters, Planet(1,1,1,1)) # user-defined

As a final note, if nothing Iā€™ve said so far is the right fit, (first of all, sorry!) then I really need to know more about the use case to see how it is unique (these are all the most standard and common approaches in Julia). Iā€™m happy to look through source code if you care to post or link.

11

Fantastic. Yes, that is it, I think.

It seems that I can write my code

using General
set_planet!(:earth)

no specific modules.

Perhaps, even

# inside General
mutable struct conditions
    st::state
end

const used_conditions = conditions(state(0,0,0,0))

function set_planet!(planet)
   planet == :earth && used_conditions.st = state(1,1,1,1)
   planet == :venus && used_conditions.st = state(1,2,2,3)
end

:fireworks: :fireworks: :fireworks:

12

Great! Note that there could be a performance hit to using a mutable struct for this (could be minimal; hard to say), so for any high performance stuff, I suggest passing used_conditions.st in as early in the stack as reasonable.

13

This is perhaps analogous to how Plots.jl does it. You load Plots, and then you set the backend (unless you want the default):

using Plots
plotly() # select backend

This function call presumably sets up a lot of ā€˜constantā€™ state.

2 Likes
14

good point.
Still, I guess it should not be slower than the general call calculation(x, st).
The fastest way, I suppose, would be to compile with const st = state(1,1,1,1). Not clear how to achieve that without code duplication in the original post, tho.

Nevertheless, I think I am satisfied with the solution for now.

1 Like
15

Thatā€™s an interesting comment.
Could you point me to a package that follows those principles?
Thanks

1 Like
16

I would start with

https://docs.julialang.org/en/v1/manual/interfaces/

as a nice example of API design.

17

Thanks for the quick answer.
Iā€™ve read that part of the manual few times already. It is indeed a very nice example. But I always struggle a bit to relate that to scientific computing.
If I would look at one of your packages, which one would you recommend?
Thanks again

18

I donā€™t think my packages are in any way exemplary, but eg

https://tamaspapp.eu/LogDensityProblems.jl/dev/#log-density-api-1

has a very sparse API.

1 Like
19

Thank you very much!