Should our package export an init! function?

We have two packages, that both export an init! function (working on different types). This is causing problems.

What is the suggested solution?

  • not having packages exporting common names
  • not importing packages with using
  • something else?

The effect is actually quite bizarre:

If I do:

using KiteModels#init_interface, VortexStepMethod

and than try to use the method init! I get the error message:

ERROR: UndefVarError: `init!` not defined in `Main`

Shouldn’t Julia be able to figure out which function to use based on the type of the parameters?)

Julia (to my knowledge) never tries to determine which function to use among alternatives. It could probably be successful at it some fraction of the time but could still be a frequent foot-gun. Julia does figure out which method of a given function to use, but Package1.init! and Package2.init! are totally distinct functions so that isn’t the question here.

Ultimately, the possible resolutions I see are

  1. Make init! be declared once and have each package add a method to that function using their own special type.
  2. Like the error hint suggests, qualify your use like Package1.init!. This is what would be required by your idea of “not importing packages with using” anyway.

“Not having packages exporting common names” is only possible when you control every package in use. This can’t be a general solution because people make and use packages all the time and can’t coordinate names with the whole userbase.

Option 1 is a pain and likely doesn’t make much sense anyway. A function should have multiple methods when they each do the “same thing” in some somewhat-interchangeable way (like Base.:+ applies to a huge number of arithmetic types in an largely consistent way), but otherwise it’s just asking for method ambiguities, collisions, or other issues (this is sometimes called a “pun”, and Julia isn’t entirely devoid of these). If you wouldn’t use the same block of code regardless of which you happen to have (i.e., utilize multiple dispatch), there isn’t much upside (but still possible downside) to making them methods of the same function. I expect that despite the fact that they both initialize something, your two init! functions aren’t really interchangeable in a way that multiple dispatch is useful.

So I would strongly recommend option 2. You might consider whether init! should be public rather than export (so that it would always require qualification), but that’s not precisely the issue at hand. This is what is done among the different Makie.jl backends (for example, CairoMakie.activate!() – although this function does not appear to be marked public in the code quite yet, it is documented to be public).

2 Likes

My current - not final - feeling:

  1. We should not export VortexStepMethod.init! . At least the documentation says it would be private: Private Functions · VortexStepMethod.jl!
  2. We should export a common init! function for our models (which will soon be in different packages) either from a new interface package, or from the base package of our set of packages KiteUtils.jl. Which then can get extended by KiteModels.jl and SymbolicAWEModels.jl .

Question:
Do packages from the DifferentialEquations.jl ecosystem export an init! function? Or from the nonlinear solver ecosystem?

I mean, we could also just use a different name that is less likely to clash.

My point is that you should not base your decision on whether they do. If they don’t now, they could next month or year or decade. Or you might later decide to add a package that does. Trying to choose names to avoid collisions is a never-ending battle that still can’t guarantee safety.

If your packages all create different models (especially if those models share a supertype) that run in the same system then the situation is different than what I discussed above. They all would satisfy the same basic interface and it does make sense to share a function they can all extend to implement that interface so that they can be used interchangeably (with multiple dispatch!). That interface (and its functions) should probably belong to whatever parent package does (or hypothetically would) define that supertype.


Alternatively, in some cases you can resolve an ambiguity by specifically using the function from the module you want:

julia> module A
           export foo
           foo() = 1
       end
Main.A

julia> module B
           export foo
           foo() = 2
       end
Main.B

julia> using .A, .B

julia> foo()
ERROR: UndefVarError: `foo` not defined in `Main`
Hint: It looks like two or more modules export different bindings with this name, resulting in ambiguity. Try explicitly importing it from a particular module, or qualifying the name with the module it should come from.
Stacktrace:
 [1] top-level scope
   @ REPL[4]:1

julia> using .A: foo # specifically ask for A.foo

julia> foo()
1

Not init!, no, but there’s CommonSolve.jl and it defines what it means to init and solve! a model. If your use jives with their definitions, then you can depend on that very lightweight package and share the same function.


In general, you have multiple options as a package author:

  • You can choose to not export names that you expect will collide with packages that are commonly used alongside yours; potentially just making them public instead.
  • You can decide to depend upon the other package and explicitly choose to use their function
  • You can decide to coordinate with other packages and come up with a lightweight “common” or “base” or “interface” package that you can all depend upon and whose functions you share. That’s what CommonSolve.jl is doing.
  • You can simply decide it’s the user’s problem and happily export away

If you run into this issue when using two packages A and B, you also have lots of options:

  • You can import A and then always explicitly use A.init! and A.solve and A.whatever at all call sites
  • You can explicitly resolve some names upfront with using A: A, init!, and, others… and you can automatically generate those lists of used names with ExplicitImports.jl
  • You can also rename as you import — using A: init! as initialize! vs. using B: init! as put_in_it! or whatever makes sense
  • You can manually “fixup” using A, B by defining what you want init! to mean with const init! = A.init!.
  • Or you can just using A, B and just always make sure you explicitly say A.init! or B.init!. Julia will happily remind you if you forget.

Even if the method signatures are entirely non-overlapping between two distinct init! methods, Julia cannot and will not automatically merge them together.

11 Likes

Thanks for the detailed explanation of the options.

I find it a bit sad, though, that there are so many different options, and it is not easy to decide what would be the best practice…

No, and it’s how namespacing works across many languages. init! is a name encapsulated by a module, and any name can only bind to one object at a time, not multiple possible ones. This isn’t the property syntax a.foo(b) where a binding foo isn’t needed in the current module and the behavior of getproperty(a, :foo) depends on what is bound to a. We can do just that with module qualification Package1.foo(a, b) vs Package2.foo(a, b); foo is ideally a public yet unexported function, but the unexported part may be too late for init!.

Multiple dispatch is also a factor, it works by dispatching foo(a, b) over the types of foo, a, and b as entirely independent inputs. If foo atypically depends on a or b instead to mimic properties, then 1) we can’t really do higher-order functions or functors, and 2) we’d defeat the purpose of encapsulation and contend with conflicting methods of separate functions in separate modules e.g. Package1.foo(::A1, ::B1) vs Package2.foo(::A1, ::B1). Multiple dispatch does allow us to dispatch a derivative function over a and b types to forward to particular functions e.g. Package3.foo(a::A1, b::B1) = Package1.foo(a, b), but that’s a lot of work to make more complex and confusing API than just qualifying modules.

1 Like