Combining two co-dependent modules into a single program

[Niall]

I’m struggling to understand how to combine modules at the moment, and am clearly missing something crucial. I have a module that uses another module, and also a method that calls another method of the same name but different signature, and I can’t get it to work. The simplified code is this:

module Hands

#----------------------------------------------
module Fingers

export Finger, wiggle

struct Finger
end

function wiggle( f::Finger)
	println(f)
end

end		# ... of module Fingers

#----------------------------------------------
using .Fingers

export Hand, wiggle

struct Hand
	f::Finger
end

function wiggle( h::Hand)
	println(h)
	wiggle(h.f)
end

end		# ... of module Hands

#----------------------------------------------
using .Hands, .Hands.Fingers

wiggle( Finger())

wiggle( Hand())

In actuality, Hands and Fingers are in two separate files, and the Fingers code gets included, rather than explicitly written as here. The warning message I get is that “both Fingers and Hands export “wiggle”; uses of it in module Main must be qualified”.

Can anyone help? Thanks!

[lmiq]

If what you want is to add a method to the wiggle function, you need to import it explicitly in the Hands module:

              function Fingers.wiggle( h::Hand)
                  println(h)
                  wiggle(h.f)
              end

that will make clear that wiggle is a single function with different methods, which is extended in the Hands module, and not a unrelated functions that happened to have the same name.

That said, Julia packages are generally “flat”. Meaning: either the Fingers module is useful on its own and then you should use it inside Hands, but not include it there, or you could just define the Finger functions inside Hand.

Meaning, these would be more common approaches:

julia> module Fingers
          export wiggle, Finger
          struct Finger end
          wiggle(x::Finger) = 1
       end
Main.Fingers

julia> module Hands
          using ..Fingers
          export wiggle, Hand, Finger
          struct Hand end
          Fingers.wiggle(x::Hand) = 5
# or, if you prefer
#        import ..Fingers: Finger, wiggle
#        wiggle(x::Hand) = 5
       end
Main.Hands

julia> using .Hands

julia> wiggle(Finger())
1

julia> wiggle(Hand())
5

or

julia> module Hands
          export wiggle, Hand, Finger
          struct Hand end
          struct Finger end
          wiggle(x::Finger) = 1
          wiggle(x::Hand) = 5
       end
Main.Hands

julia> using .Hands

julia> wiggle(Hand())
5

julia> wiggle(Finger())
1

Nothing impairs you from putting the “finger” part in a different file in the second case and include it, but that not necessarily needs to be in a module.

[Niall]

Thank you very much, Leandro - that makes clear to me where my misunderstanding lies. Coming from OOP, I keep trying to interpret Julia in those terms. I really like the way Julia has deconstructed the entire idea of classes, distinguishing their purpose into dispatch and responsibility, but I’m still learning what that means for practical programming.

I had until now regarded modules as the responsibility component of classes, and felt the need to separate Fingers and Hands into separate modules. However, from what you say, I gather that modules actually capture a third aspect of classes: relevance. I take it that the criterion for putting functionality together in one module is whether they are relevant to each other.

The first of your “more common approaches” seems very strange to me: a Hand might use Finger.wiggle() to wiggle its Fingers, but why would the wiggling of the Hand itself be named from the Finger module? The answer seems to be that since Hand.wiggling is related to Finger.wiggling, they should both be within the same module. Am I getting there?

Thanks once again for the very detailed answer.

[Tamas_Papp]

@Niall

I gather that modules actually capture a third aspect of classes: relevance .

You may be overthinking this — modules are for primarily namespace management, nothing more. See

https://docs.julialang.org/en/v1/manual/modules/

[Niall]

… and yet it seems to me there is still something I haven’t understood. Does this mean that delegation relationships should only ever occur within a single module?! Here, I wish Hand to implement its own wiggle functionality, yet delegate responsibility for the Finger.wiggle part of that functionality across module boundaries to the Finger type.

Or, in the particular case I am implementing, I wish a gene-based Organism (Hand) to delegate responsibility for its own mutation (wiggle) to the individual Chromosomes (Finger) that it contains. As I understand you, this means that both types (Organism and Chromosome) must either be implemented within a single (and therefore very large!) module, or else the module Organisms must name its own mutation functionality Chromosomes.mutate, and so accept its own second-class status with respect to the module Chromosomes.

At the moment, this just feels so wrong that I must be missing something somewhere.

[Niall]

Thank you, Tamas. I needed that bringing down to Earth!

I actually had always had problems with that phrase: “Modules are separate namespaces, [… allowing] the same name to be used for different functions or global variables without conflict, as long as they are in separate modules.” When exactly do I want two domains to have no names in common? Presumably, precisely when they never sub-delegate responsibility to each other, but merely request services. Yes, that makes sense - thank you!

[lmiq]

Yes, probably you are overthinking this in the OOP style. Another way to organize that code could be:

module Movements
  function wiggle end
end
module Fingers
  import Movements: wiggle
  struct Finger end
  wiggle(x::Finger) = ...
end
struct Hands
  import Movements: wiggle
  struct Hand end
  wiggle(x::Hand) = ...
end

Also, there nothing wrong about “very large modules”. They don’t need to be inside a single file, so the code organization is not associated to how many modules there are. This is different from Python, in which each module is a file. (not saying that the Julia model in this regard is better at all, I think the Python way to do this has its advantages).

In the case of chromosomes, I would think it like this: what can be mutated is a chromossome (or a gene, or perhaps a base, but lets say that the minimal entity is the chromossome). Then, the module would be:

module Mutations
  struct Chromosome end
  struct Organism
    c::Chromosome
  end
  mutate(c::Chromosome) = ...
  mutate(o::Organism) = mutate(o.c)
end

[Niall]

@Niall

When exactly do I want two domains to have no names in common? Presumably, precisely when they never sub-delegate responsibility to each other, but merely request services.

… and yet …

I’ve just discovered that I get exactly the same name conflict if wiggle is exported from two distinct modules, one of which contains the abstract type Wiggler, and the other contains the derived type Finger <: Wiggler:

module Wigglers
export Wiggler, wiggle
abstract type Wiggler end
function wiggle( wiggler::Wiggler)
	println( "Wiggler-wiggling!")
end
end		# ... of module Wigglers

module Fingers
using ..Wigglers
export Finger, wiggle
struct Finger <: Wiggler
end
function wiggle( f::Finger)
	println( "Finger-wiggling!")
end
end		# ... of module Fingers

module Hands
using ..Wigglers
export Hand
struct Hand <: Wiggler
end
end		# ... of module Hands

using .Wigglers, .Hands, .Fingers
wiggle( Finger())
wiggle( Hand())

As far as I can see, there is no way to make this program produce the (to me correct) output:

Finger-wiggling!
Wiggler-wiggling!

Yet surely derived types are a supreme example of delegating responsibility. So surely this implies that derived types must always be defined within the same module as their abstract supertype? I assume that cannot be the case, yet cannot offhand find a counterexample. I’m lost! Help!

[lmiq]

@Niall

using ..Wigglers

You need import ..Wigglers: wiggler there.

But that way to structure the code is fine, I just don’t think that you need the Wiggler type in the first module, that module can only be an interface that defines the function wiggler in general, to be extended by the other modules.

[Niall]

Sorry - I hadn’t seen this post before I sent my last one out. I’m going to have to think more about this, Leandro. I didn’t know you could declare a single function without implementation as you do in the module Movements.

I know that my system is large enough that I don’t really want everything gathered together as in your module Mutations: I want Organism, Chromosome and World as encapsulated from each other as I can. However you may be right that modules just aren’t the right tool for the job. Maybe I do just need different files within a single module. Still, any further tips you may have on the subject would be very welcome.

[lmiq]

The key point is think in terms of functionality.

The things defined in the Chromosome module are useful without everything that Organisms define? If it is, then you can define a module (and even a package) for that that does not mention Organisms at all, and you and others can use that as a dependency to other packages. If it does not make sense to use the Chromosome functionality outside Organisms, then just use separate files and include them in the same module/package.

Is the Organisms module useful even if Chromosomes is not defined? If not, develop a module or package which has Chromosomes as a dependency. If it is, it can be also a separate package.

If both modules/packages can be used independently from each other, but they have to extend a function that exists in common for both of them, then you want to have a module/package that interfaces those common functions, like your Wigglers module there, which would be a dependency of both.

Julia encourages you to write packages which provide small but useful functionalities, and can be reused and extended by other packages.

[Niall]

@lmiq

Is the Organisms module useful even if Chromosomes is not defined? If not, develop a module or package which has Chromosomes as a dependency. If it is, it can be also a separate package.

This seems to best describe my situation: Organisms use Chromosomes, but both can be used independently of each other. I just haven’t yet worked out exactly what a package is - I can’t find it defined in the manual. Also, and more to the point here: What exactly do you mean by a “dependency”? An import?

[lmiq]

@Niall

What exactly do you mean by a “dependency”? An import?

from a module point of view, yes (or using), but if you develop different packages for each module (that can be installed independently by the user), then they become dependencies of the packages.

[Niall]

Thank you - that’s the information I needed. So “package” is not a Julia term, but a GitHub term. That’s my next staging post in this bout of learning. Right now, I need to go and chair a boring meeting, but I’ll peruse all this new information and get back when I’ve processed it.

Thanks to all who’ve contributed!

Best wishes,
Niall.

[MA_Laforge]

Confusion of concepts

I think a large part of your difficulties comes from a confusion of concepts. Some of the methodologies that are practical for other languages are sort of a hinderance in Julia. They need to be adapted somewhat to avoid “the square peg in a round hole” situation.

A few helpful mappings:

• “Namespace” or “Named global scope” → Julia “module”
• “Software module” → Julia “package”
  • (though a simple namespace (Julia “module”) might be sufficient in certain cases).
• Loading code into memory (might include “module”-s) → Julia include() statement.
  • (typical application: assemble code from multiple files).
• Loading Julia “package” into memory → Julia import or using statement.
  • (in turn, packages often use include() to assemble code from multiple files).

dispatch, responsibility, relevance…

Correct: Typically, “software modules” are used to enforce “responsibility”. You encapsulate some functionnality within a software module to guarantee proper functionality wilst providing a simple/effective interface for your user.

Typically, you would use a Julia “package” to do this function, though a simple Julia “module” might be sufficient (depending on the circumstances).

At best, Julia packages & modules offer soft enforcement of responsibility. Nothing is really hidden or off limits from the outside world as long as you use the path.to.the_housing_module.and_varible_name = some_new_value.

Probably as a consequence of this lack of information hiding, Julia packages or modules are rarely used for this purpose. Instead, Julia developpers tend to simply exercise caution when writing a feature for their package/module. For example, a communication package might “provide” a messaging queue with the appropriate enque(::MyQueue) and dequeue(::MyQueue) functions, and a bunch of low-level facilities to implement these features. They might also provide user-facing facilities (ex: pack(::MyMessage)/unpack(::MyMessage)), and base objects (MyMessage, MyQueue, …) that are needed for the implementation. Since there is no real information hiding in Julia, it tends to be most practical to place everything in a single Julia “module” or “package” (ex: MySecureMessaging). There is no real need to split a “software unit” like a messaging queue into its own (sub-) “module”.

So, I guess “responsibility” in Julia is more about explicitly documenting your interface & intended use to the user. For example, you might want to prefix functions not meant to be called directly by the user with “_”. Some people rely on what functions/variables (Julia symbols) are export-ed from a module, but I often ommit exporting functions to avoid name collisions (a problem you have already noted). In other words, I often provide user-facing functions that are never get export-ed.

What Julia’s dispatch system does differently than in most languages is allow the same function name to be re-used for different permutations of argument types. This avoids having to write long functions names like addIntInt(x::Int, y::Int), addIntFloat(x::Int, y::Float64), …

And yes: dispatch is no longer the domain of classes as it is in most OO-languages I’m aware of. In Julia, a more flexible dispatch system is available for all functions, and resolved for different (potentially complex) permutations of their argument types.

Mapping of object-level dispatch from most OO paradigms:

• a_circle.draw() → draw(a_circle)

IMO, it is actually more natural to write Julia’s draw(a_circle) than using a_circle.draw() in most cases anyhow.

Relevance: That sounds about right: Julia “modules” (namespaces) are a good way to organize things by relevance. I will elaborate a bit in the following section.

Why use Julia “modules” (namespaces)?

It might be hard requirement:

1. Julia “modules” are required to wrap the contents of each Julia “package” of the same name.

About packages:

• Packages are mostly a way to bundle-up software into units that can be shared amongst projects.
• You don’t technically need to bundle things up into a package if your code is project-specific (needed only for one project).

Otherwise, I find “modules” (namespaces) can be useful to logically separate code. In this case, “modules” act sort of like folders, to de-clutter the “user-facing” namespace, and avoid name collisions:

module MySortingPackage

module BubbleSort
   #Supporting functions here

   sort!(data) = ... #Entry point of algorithm
end

module QuickSort
   #Supporting functions here

   sort!(data) = ... #Entry point of algorithm
end

bubblesort!(data) = BubbleSort.sort!(data)
quicksort!(data) = QuickSort.sort!(data)
export bubblesort!, quicksort!

end #MySortingPackage

But if you don’t actually have high risk of name collisions (ex: not that many supporting functions), most people just put everything in a single namespace (Julia “module”).

[MA_Laforge]

@Niall

As I understand you, this means that both types ( Organism and Chromosome ) must either be implemented within a single (and therefore very large!) module, or else the module Organisms must name its own mutation functionality Chromosomes.mutate , and so accept its own second-class status with respect to the module Chromosomes .

I don’t fully understand this statement, so I’ll try to give an example of what I think you might be wanting to accomplish. Please explain which parts of this solution are inadequate for your particular problem. (Sorry, I’m not an expert in genetics, so I might have gotten some of the terms wrong).

Suggestion (Genetics.jl file):

module Genetics

struct Nucleotide #Place holder for now
end

#-------This stuff could be in its own Chromosomes.jl file if it is too much code-------
struct Chromosomes
	dna::Vector{Nucleotide}
end

#Construct a Chromosomes object with the given amount of nucleotides:
function Chromosomes(num_nucleotides::Int)
	dna = [Nucleotide() for i in 1:num_nucleotides]
	return Chromosomes(dna)
end

function mutate!(o::Chromosomes)
	println("Mutating ::Chromosomes")
	#Swap out nucleotides here to simultate mutation!
end


#-------This stuff could be in its own Chromosomes.jl file if it is too much code-------
struct Organism
   chromosomes::Vector{Chromosomes}
end

function Organism(num_chromosomes::Int, num_nucleotides::Int)
	chromosomes = [Chromosomes(num_nucleotides) for i in 1:num_chromosomes]
	return Organism(chromosomes)
end

function mutate!(o::Organism)
	println("Mutating ::Organism")
	map(mutate!, o.chromosomes) #Basically runs a for loop for you!
end
#-------End of Organism.jl file-------

export mutate!

end #module Genetics

How to try out above module

You would then typically bundle Genetics.jl as a package. But since this is a bit more involved as a first step, we’ll just include() it for illustration purposes below so that you can more quickly try it out for yourself:

include("Genetics.jl") #Load code in Genetics.jl

using .Genetics #Make "export"-ed symbols (mutate!) available in this namespace.

#Chromosomes not exported. Need to qualifiy explicitly with "Genetics":
c = Genetics.Chromosomes(5) 
mutate!(c) #Available in this scope since "export"-ed from Genetics

println()

#Organism not exported. Need to qualifiy explicitly with "Genetics":
o = Genetics.Organism(2, 4)
mutate!(o) #Available in this scope since "export"-ed from Genetics

Slight warning

FYI: Julia doesn’t like if you run include("Genetics.jl") & using .Genetics twice. It will complain that you are overwriting exported symbols (mutate! in this case).

You wouldn’t have this problem if made Genetics.jl into a proper Julia “package” because using does not re-load an already loaded package. On the other hand, include() is a lower-level command that does re-load code.

[Niall]

Thanks MA_Laforge - those were two very informative posts. It therefore looks as though I really have been misunderstanding the relationship between modules and types. I really wanted to match them up 1-to-1, so that each type was grouped together with its operations within one separate module, with an additional module to handle interactions via the corresponding abstract types.

However, it looks like I really need to put all three of my classes (Chromosome, Organism, World) into one single module to enable their interaction. I’ll put them in different files that are then all included into the single module. This still feels wrong to me, but hey, that’s what learning is about, I guess. Thanks all round!

[ericphanson]

@Niall

So “package” is not a Julia term, but a GitHub term

Just to say that I don’t think this is quite right-- Julia really does have a concept of “packages”, and they don’t need to be tied to separate GitHub repositories (or even on GitHub or on the internet at all!). Julia has a package manager (the standard library Pkg) which has a concept of a “package registry” which is essentially a directory of “packages”. Packages are pieces of code in a single module (which can have submodules if you wish) that can have dependencies on other packages and declare compatability with them. These relationships are stored both in the packages’ Project.toml file and in the registry itself. The package manager’s job is to “resolve” a set of versions of packages that are all mutually compatible and retrieve them.

None of this requires GitHub or the internet, but the most common use of this is the General registry which comes preinstalled with the package manager, and has additional requirements about its packages (e.g. that they are accessible at a hosted git repository on the internet, although not necessarily on GitHub). However, LocalRegistry.jl makes it easy to make and use registries locally.

Registries and packages are a very useful way to organize large codebases in a flexible way that allows upgrading one part of the codebase while allowing other parts to rely on older versions of some of the code until they can be upgraded. This modularization is essential for large decentralized projects like the open source community, but also is a great way to manage large codebases even within a single organization.

[Tamas_Papp]

@Niall

However, it looks like I really need to put all three of my classes (Chromosome, Organism, World) into one single module to enable their interaction.

Yes, starting development in a single module and then splitting on demand is usually the best idea.

Again, I would not overthink this. It is better to reorganize/refactor codebases on demand instead of planning everything ahead of time.

[Niall]

Thanks Eric - yes, I’d gradually gathered that after Leandro sent me the link to pkgdocs. So much to learn and so little time to do it.