An issue with dispatching imported types

General Usage
21

Note that (AFAIK) Revise cannot delete a struct definition, just its methods. Thus, if I have a module:

module MyPkg
    struct A end
end

and thus, a struct that is defined inside the package:

julia> includet("./MyPkg.jl")

julia> using .MyPkg

julia> MyPkg.A
Main.MyPkg.A

it does not disappear if the definition is deleted:

module MyPkg
end

# after changing the file
julia> MyPkg.A
Main.MyPkg.A

which means that you need to restart you Julia section if you deleted those definitions inside PeriodicSystems.

ps: To use a clean new environment, start Julia and do:

julia> ] activate --temp

julia> dev PeriodicSystems, PeriodicMatrices

julia> # ... do your tests
22

I performed the steps as you indicated and then the test. The following error message is possibly relevant

julia> A = PeriodicSymbolicMatrix([cos(t) 1; 1 1-sin(t)],2*pi);
WARNING: both PeriodicMatrices and PeriodicSystems export "PeriodicSymbolicMatrix"; uses of it in module Main must be qualified
ERROR: UndefVarError: `PeriodicSymbolicMatrix` not defined
Stacktrace:
 [1] top-level scope
   @ REPL[9]:1

The loaded version of PeriodicSystems corresponds to the current version v0.8.0, which contains the old definitions.

(jl_35CxA4) pkg> st
Status `C:\Users\Andreas\AppData\Local\Temp\jl_35CxA4\Project.toml`
  [dd9cd634] PeriodicMatrices v0.1.1 `C:\Users\Andreas\.julia\dev\PeriodicMatrices`
  [5decd0d0] PeriodicSystems v0.8.0 `C:\Users\Andreas\.julia\dev\PeriodicSystems`

This explains the above error message.

1 Like
23

yeah, the dev command you did, took the two packages in development mode, but from the .Julia/ folder, not from your local hard drive. That is why those are the old ones.

1 Like
24

This could be the issue: the old structures are somehow still present, and this leads to ambiguities. Is any way to get rid of the existing precompilations? I ponder to start from scratch to rebuild PeriodicSystems, with another name, as a new package. This should work without problems, or ?.

25

In principle restarting Julia should be enough.

26

A breaking release should be fine, since you remove Features (like the struct we discuss). No need for a new name.

27

I moved to a new breaking release (for the MWE) v0.9.0

(@v1.10) pkg> st PeriodicSystems
Status `C:\Users\Andreas\.julia\environments\v1.10\Project.toml`
  [5decd0d0] PeriodicSystems v0.9.0 `C:\Users\Andreas\Documents\software\Julia\PeriodicSystems.jl`

The sequence still fails:

julia> using PeriodicSystems

julia> using PeriodicMatrices

julia> A = PeriodicSymbolicMatrix(rand(2,2),2*pi);

julia> ps1(A)
ERROR: MethodError: no method matching ps1(::PeriodicSymbolicMatrix{:c, Num, Matrix{Num}})

Closest candidates are:
  ps1(::T) where T<:PeriodicMatrices.PeriodicSymbolicMatrix
   @ PeriodicSystems C:\Users\Andreas\Documents\software\Julia\PeriodicSystems.jl\src\PeriodicSystems.jl:6

Stacktrace:
 [1] top-level scope
   @ REPL[28]:1

However, the following works:

julia> A = PeriodicSystems.PeriodicSymbolicMatrix(rand(2,2),2*pi);

julia> ps1(A)
PeriodicMatrices.PeriodicSymbolicMatrix{:c, Num, Matrix{Num}}

Is this a step forward?

See also

julia> @which PeriodicSymbolicMatrix(rand(2,2),2*pi)
PeriodicSymbolicMatrix(F::VecOrMat{T}, period::Real; nperiod) where T<:Real
     @ PeriodicMatrices C:\Users\Andreas\Documents\software\Julia\PeriodicMatrices.jl\src\types\PeriodicMatrix_sym.jl:27

julia> @which PeriodicSystems.PeriodicSymbolicMatrix(rand(2,2),2*pi)
PeriodicMatrices.PeriodicSymbolicMatrix(F::VecOrMat{T}, period::Real; nperiod) where T<:Real
     @ PeriodicMatrices C:\Users\Andreas\.julia\packages\PeriodicMatrices\ZwH3j\src\types\PeriodicMatrix_sym.jl:27

Apparently the same function is called in both cases.

28

Oh, maybe I was not precise enough. Just changing the version number does not change the functionality, it is an indicator for users that there was a breaking change, so that they have to adapt their code.

You would still have to decide to move the definition of the PeriodicSymbolicMatrix to exacty one package, otherwise you always have two such types.

Again, I can not see your dev version, so I can just guess:

You currently seem to define the type in both packages still. This still creates two completely different types. It looks like, currently only one of them, PeriodicMatrices exports the type and hence only one is available in global scope.
I conclude that from the fact that you can define A (and that ps1 fails).

The PeriodicSystems tape is not exported, because that would cause an ambiguity and you would have to specify which of the two types A should be of.

But then of course ps1 is defined for the (non-exported, internal) data type of a PeriodicSymbolicMatrix within the PeriodicSystems which due to the two completely different types is not the type of A. so sure the ps1 call fails, unless you specifically use that type for which ps1 is defined.

A good analogy is really again two people, maybe even born on the same day of same name.
One lives in PeriodicSystems, one lives in PeriodicMatrices and that name is also exported (make known to) Main if you are using the package.
They are still two completely different types (people) that just share the same name.

I see really only one solution here. Do not define the type twice. It now confuses you as the developer so it will confuse most of your users.
I would do the following – also since you are “extracting” the matrices into their own package.

  1. Define the type only in PeriodicMatrices and export it there
  2. Make PeriodicMatrices a dependency of PeriodicSystems and use the type that PeriodicMatrices defines therein as well

This can even be done in a non.breaking way, since someone only using PeriodicSystems would not even notice a difference.

Your last type of code confuses me though, because it seems you might have even done this already? Again, all of this post is merely written on guessing what your current code status is. It would maybe be easier to actually do a small zoom meeting instead of this now really really long thread where we try to reverse-engineer from your errors both what you current package status and REPL status is.

The main message and thing to follow is: Do not define the type twice. Define it once and only use it in the other package then (that of course then depends on the one where you define it).

29

Yes, I did.

30

Then your first code is still running on a version of REPL / running Julia where that change had not yet taken effect (i.e. you still need to restart REPL).

31

The type is defined only once in PeriodicMatrices. No type definitions take place in PeriodicSystems.

32

That contradicts what you wrote earlier that

unless that was in a REPL session where the old definition was still present, but again, this is by now merely guesswork and a bit hard to reverse-engineer without knowing the code base and REPL state you run code in.

33

I am not sure I understood this. I am working in VSCode, so any time I restart Julia, I restart also REPL. Or ?

34

Ah yes with restarting REPL I mean to do julia>quit() or control+d and then start Julia again or start a new Terminal in VS code.

The simple reason is, that there is no way to “unload” or “unusing” a package. For functions Revise.jl is super helpful but whenever you change structs (or also move them to new packages) you have to restart julia.

35

The whole PeriodicSystems package consists of the following:

module PeriodicSystems

using PeriodicMatrices

function ps1(A::T) where {T <: PeriodicSymbolicMatrix}
    return typeof(A)
end

export ps1

end

As you can see, there are now new type definitions, and ps1 is the only function in this package (excepting those imported from PeriodicMatrices).
I wonder if using Reexport could be helpful.

36

Well. Again. I can not reproduce the error you report above with the code you post here.

For reproducibility: I start Julia, I activate a temporary environment, I install PeriodicMatrices (I fetch a coffee while it complies :wink: ) and then do the following

(jl_vBrjwG) pkg> status
Status `/private/var/folders/_v/wg192lpd3mb1lp55zz7drpcw0000gn/T/jl_vBrjwG/Project.toml`
  [dd9cd634] PeriodicMatrices v0.1.1

julia> module PeriodicSystems

       using PeriodicMatrices

       function ps1(A::T) where {T <: PeriodicSymbolicMatrix}
           return typeof(A)
       end

       export ps1

       end
Main.PeriodicSystems

julia> using .PeriodicSystems

julia> A = PeriodicSymbolicMatrix(rand(2,2),2*pi);
ERROR: UndefVarError: `PeriodicSymbolicMatrix` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Hint: a global variable of this name may be made accessible by importing PeriodicMatrices in the current active module Main
Stacktrace:
 [1] top-level scope
   @ REPL[6]:1

julia> using PeriodicMatrices

julia> A = PeriodicSymbolicMatrix(rand(2,2),2*pi);

julia> ps1(A)
PeriodicSymbolicMatrix{:c, Symbolics.Num, Matrix{Symbolics.Num}}

So all works as I wrote above. We just have one definition of PeriodicSymbolicMatrix, namely in PeriodicMatrices.
You are right, since PeriodicSystems does not export it, we are explicitly required to load PeriodicMatrices to be able to construct A, otherwise (see first try) that name does simply not exist in Main.

But as soon as I do load PeriodicMatrices I can define A but then of course also ps1 just works fine because in this scenario we just have one such type and within PeriodicSystems and in Main both names refer to exactly the same type.

So I am not sure how you got to an error above, unless your Julia session was not freshly started and there were some old remedies of existing types or you used using PeriodicSystems and have that installed or in dev mode in that environment as well – then it dies not load the module you define on REPL, as you can see my code has to use using .PeriodicSystems to load the module I define on REPL.

1 Like
37

Updating Julia, say to v1.11, could help ?

38

I think that does not make a difference. While I could not install the most recent version on 1.8 or 1.9 (maybe it needs at least 1.10?) on 1.10 the last example I posted runs 100% the same results.

39

I changed the module definition to

module PeriodicSystems

using Reexport
@reexport using PeriodicMatrices

function ps1(A::T) where {T <: PeriodicSymbolicMatrix}
    return typeof(A)
end

export ps1

end

Now it works:

julia> A = PeriodicSystems.PeriodicSymbolicMatrix(rand(2,2),2*pi);

julia> ps1(A)
PeriodicMatrices.PeriodicSymbolicMatrix{:c, Symbolics.Num, Matrix{Symbolics.Num}}

I will continue now with some more serious tests.

40

@lmiq and @kellertuer: I would like to thank for your time and patience assisting me to overcome this issue. I would still like to stress that, although the above solution works, I don’t have the full understanding what is different from the original implementation. Furthermore, I wonder if some side effects still can occur. In any case, I will register a new breaking release of PeriodicSystems, before other developments take place.

1 Like