Multiple Dispatch Across Seperate Files

I am attempting to use the same function forward in different contexts.

Layers.jl

module Layers

export Dense, forward

struct Dense
    n_inputs
    n_neurons
    weights
    biases
    function Dense(n_inputs, n_neurons)
        ### Intialize weights and biases
        ...
        new(n_inputs, n_neurons, weights, biases)
    end
end

function forward(layer::Dense, inputs)
    if layer.n_inputs != size(inputs, 2)
        throw(DimensionMismatch("Input size does not match layer input size"))
    end

    return (inputs * layer.weights) .+ layer.biases
end

end

Activations.jl

module Activations

export forward, ReLU

abstract type ReLU end

function forward(layer::Type{ReLU}, data)
    return map(x -> max(0, x), data)
end

end

main.jl


using .Layers
using .Activations

dense1 = Dense(2, 4)

inputs = [[1 2]; [3 4]; [3 5]]

out = forward(dense1, inputs)
forward(ReLU, out)

On the last line, I get an error saying that forward does not have an overload for ReLU:

julia> forward(ReLU, out)
ERROR: MethodError: no method matching forward(::Type{ReLU}, ::Array{Float64,2})
Closest candidates are:
  forward(::Dense, ::Any) at /Users/user/Documents/test/Layers.jl:33
Stacktrace:
 [1] top-level scope at REPL[106]:

Is there a correct way to pull in functions with the same name from multiple different files to use multiple dispatch?

The problem is that you have 2 different functions named forward instead of 1 function with 2 methods. The solution is to make one (or both) of Layers and Activations import forward before defining it.

7 Likes

I think that deserves a design explanation. Naively I would expect that to work and to raise a warning if a method overwrites the other.

Of course for third party modules the alternative is to import. But it is somewhat scary that that kind of overwrite can happen without notice when loading multiple modules.

2 Likes

Hi, thanks for your reply. Iā€™m still having a bit of an issue doing it this way. I tried following this and reading this documentation, yet Iā€™m still unsure as to where Iā€™m going wrong.

New file NNFS.jl

module NNFS

export forward

forward(layer, data) = begin
    throw("You need to overload base forward for $(typeof(layer)) and $(typeof(data))")
end

end

Modified Layers.jl

module Layers

using Random; Random.seed!(0)
include("NNFS.jl")
import .NNFS.forward

struct Dense
    n_inputs
    n_neurons
    weights
    biases
    function Dense(n_inputs, n_neurons)
        ### Initialize weights and biases
        ...
        new(n_inputs, n_neurons, weights, biases)
    end
end

NNFS.forward(layer::Dense, inputs::Array{Float64,2}) = begin
    if layer.n_inputs != size(inputs, 2)
        throw(DimensionMismatch("Input size does not match layer input size"))
    end

    return (inputs * layer.weights) .+ layer.biases
end

export Dense

end

Main file:

include("NNFS.jl")
include("Layers.jl")
using .NNFS: forward
using .Layers

dense1 = Dense(2, 4)

inputs = [[1. 2.]; [3. 4.]; [3. 5.]]

out = forward(dense1, inputs)

The last line gets the error (which I defined)

ERROR: "You need to overload base forward for Dense and Array{Float64,2}"
Stacktrace:
 [1] forward(::Dense, ::Array{Float64,2}) at /Users/user/test/NNFS.jl:6
 [2] top-level scope at REPL[7]:1

And calling methods on forward gives the following

julia> methods(forward)
# 1 method for generic function "forward":
[1] forward(layer, data) in Main.NNFS at /Users/user/test/NNFS.jl:5

What is the correct way to set up function overloading from multiple different files?

Thereā€™s no overwriting going onā€“there are just two different functions: Layers.forward and Activations.forward which have nothing whatsoever to do with one another. Neither interferes with the other, but they canā€™t both participate in multiple dispatch because they are fundamentally different functions.

5 Likes

The reason for this is that making a new function by default means you only need to be aware of the functions you extend. If a new method was created by default, any package (or Base) adding a new function would be a potentially breaking change.

1 Like

The problem in your case is that youā€™ve now included NNFS.jl in two different placesā€“once inside Layers.jl and again in your main file. Those separate include()s have created two completely unrelated modules which happen to have the same name. Thatā€™s why itā€™s important not to include the same file multiple times. Hereā€™s a post with a similar question on avoiding multiple includes that you might find helpful: Need to include one file multiple times, how to avoid warning - #5 by rdeits

In your case, you should be able to replace:

include("NNFS.jl")
import .NNFS: forward

with

import ..NNFS: forward

in Layers.jl

4 Likes

My blog post may be helpful:

https://www.ahsmart.com/pub/the-meaning-of-functions/

This is also the reason why you sometimes see WhateverBase.jl packages because they are used to define functions that are expected to be extended.

8 Likes

I am sorry, there is indeed an error raised, so that is fine:

julia> module Pkg1
         f(x) = 1
         export f
       end
Main.Pkg1

julia> module Pkg2
         f(x) = 2
         export f
       end
Main.Pkg2

julia> using .Pkg1; using .Pkg2

julia> f(1)
WARNING: both Pkg2 and Pkg1 export "f"; uses of it in module Main must be qualified
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] top-level scope at REPL[5]:1

julia> 

My first impression from the original post was that didnā€™t raise any error or warning.

Also, I do not understand how did you get the no method matching forward... error. You should have gotten a warning on the impossibility of using forward from two modules without explicitly importing them, and then forward not defined errors. Here is what happens here, reproducing your initial post with some simplifications:

julia> module Layers
         export Dense, forward
         struct Dense x end
         forward(x::Dense) = "from Layers"
       end
Main.Layers

julia> module Activations
         export ReLU, forward
         abstract type ReLU end
         forward(x::Type{ReLU}) = "from Activations"
       end
Main.Activations

julia> using .Layers

julia> using .Activations

julia> dense1 = Dense(1)
Dense(1)

julia> forward(dense1)
WARNING: both Activations and Layers export "forward"; uses of it in module Main must be qualified
ERROR: UndefVarError: forward not defined
Stacktrace:
 [1] top-level scope at REPL[6]:1

julia> forward(ReLU)
ERROR: UndefVarError: forward not defined
Stacktrace:
 [1] top-level scope at REPL[7]:1

julia> 

Concerning the original question, here is a simplified version which might help to understand what is going on:

NNFS.jl:

module NNFS
export forward
forward(layer) = "original"
end

Layers.jl:

module Layers
include("NNFS.jl")
import .NNFS.forward
struct Dense
  x
end
NNFS.forward(layer::Dense) = "overloaded"
export Dense
end

main.jl:

include("NNFS.jl")
include("Layers.jl")
using .NNFS: forward
using .Layers

dense = Dense(1)

println(forward(dense))
println(Layers.forward(1.0))  # not Dense
println(Layers.forward(dense))
println(Layers.NNFS.forward(dense))

Results in:

julia> include("./main.jl")
"original"
"original"
"overloaded"
"overloaded"

As you see, you are overloading the forward function from NNFS, but that is visible only on the NNFS version which lives inside Layers. There are two versions of NNFS, because of the double include, as explained.

2 Likes

Thank you! Is there an explanation on . vs .. available somewhere? The only mention of it I can find in the documentation is here.

Typically the difference between . and .. is to specify cwd or parent directory; so, Iā€™m a bit surprised to see it being used like this.

Itā€™s explained at the end of this section: Modules Ā· The Julia Language

The analogy to cwd or the parent directory is intentional: the .. means to look in the parent module.

3 Likes

Here is an example: Difference between "using x" vs "using .x" or "using Main.x" - #2 by lmiq

Note that the modules chapter was rewritten significantly in 1.6/master, see

https://docs.julialang.org/en/v1.7-dev/manual/modules/

1 Like

I recommend against using submodules, for the majority of use cases. (not all, but most)
It leads too problems like this too easily.
Either use seperate packages and load them via using or import.
Or use seperate files and include them, with only the main file having a module declaration.
Not seperate files containing modules includeed into each other.

At least initially, just put everything in one big namespace.
Namespaces are over-rated, lets have less of them.
This may be counter to your experience in other languages (it was for me), but julia is a different language to others.
And submodules just allow for extra mistakes, and require extra book-keeping.
You can always later decide to split things into seperate packages (or seperate submodules if you really want), but until you have a reason to, just put it all in the same namespace.

4 Likes

I think your comment could be summarized to ā€œusing submodules is non-trivial/ā€˜very different from other languagesā€™ so beginners should avoid it if possibleā€. Obviously, any language feature may be wrongly used or overused. However, without talking about specifics it is impossible to draw the line. I use multiple submodules (with ā€œseperate files containing modules includeed each otherā€) and have yet to experience any problems regarding this. Nor I have any reason to break my package into multiple packages.

2 Likes

One of my favourite quotes is from the introduction of
Press, Teukolsky, Vetting and Flanneryā€™s ā€œNumerical Recipesā€:

We do, therefore, offer you our practical judgements wherever we can. As you gain experience, you will form your own opinion of how reliable our advice is. Be assured that it is not perfect!

This is my advise, your mileage may vary.
Be assured that there are many viable ways to do things, and this is just the one I personally recommend.

2 Likes