I am happy to announce CallableModules.jl. By making your module callable, you can, e.g., share the same name as a shell command line interface and as a Julia interface.
So if your app accepts the following arguments when the process is started (typically in the shell):
MyApp int1 int2 [--verbose]
you can now also have
MyApp(int1, int2; verbose)
within Julia.
There are probably other use cases for callable modules. You can play with it to find them. The package is rather small, but it needs to be a package due to the implementation doing type piracy which would clash with a redefine method error if multiple packages would be doing it, but works when they all use CallableModules.jl.
To use it, you need to
]add https://github.com/PatrickHaecker/CallableModules.jl
using CallableModules
You can then tag the method in you module (package), which should be called when the Module is being called, e.g. main, with @module_main to get the interface as shown above.
The implementation uses the design workaround proposed by @Keno in #61256 where Val is used to dispatch on modules to then make (all) Modules callable to then dispatch on the ones which implement something useful.
The package is not yet registered, because I want to find a suitable Github organisation for it. Maybe JuliaCollections could be a home for the package? I’d be happy if someone would offer to have it in their Github organisation and provide me with permissions so that I can maintain it there.
Quite clearly not appropriate for the General registry. Do type piracy in a personal script if you must, but don’t try to register it in General. My two cents, but seems clear.
I really appreciate your opinion, @nsajko. Thanks for stating it clearly.
In most of the cases I agree that we should avoid type piracy. However, in this case, I have the following reasons why I currently think it is the best available option:
Julia uses the same type for all modules. I argued in #61256 why I think it would be better to have a separate type for each module. However, this does not seem to have any traction and even if this might be a good idea, we just don’t have it and at least short-term it does not seem to be in reach.
If Core implemented the relevant method, it would no longer be type piracy. But again, it does not seem that there is any intention to do so. And I see that there are arguments for opt-in here, at least to explore the problem space a bit better by having a package first.
For callable objects type piracy is harder to avoid in general, as there is no function name to distinguish during dispatch.
The package’s purpose is exactly to avoid type piracy problems by being the coordinating instance.
Every package using CallableModules.jl is actively opting in, so there should be no surprises. But please mention if you have an idea how this should be made even more transparent.
The chances of a true type piracy problem, i.e. that a module call will be used in the future for something else than, well, a module call, seem exceedingly low (is that logically even possible?).
The functionality that an app shares the same name for its module, its command line interface and its Julia interface is useful and quite intuitive from a user’s perspective.
But you made me curios, as I know you often come up with really elegant solutions: What do you suggest? How could the problem of name sharing be solved more elegantly?
# This is type piracy, but is fully generic. If every package which wants to have its
# module callable uses this package, no problems with multiple definitions should occur.
(x::Module)(varargs...; kwargs...) = Val(x)(varargs...; kwargs...)
Nope, not only is it unconditionally type piracy, all packages using CallableModules would interfere with the potential Core/Base implementation and likely break something. This gets worse if CallableModules gets versions that aren’t strictly aligned with Julia versions. These sorts of piracy packages are tolerable as personal Base hacks, but not as widespread dependencies.
Secondly, this does not work as stated.
Apps are documented to require the macro call @main, which resolves to a function name main after some background registration for Apps and automatic Main.main calls. @module_main does not do any of that and explicitly supports names other than main.
The premise that the shell command and the underlying module share a name is false. The documented Apps examples ALL specify names in the [apps] section of Project.toml that differ from the names of the package or submodules. Multiple App commands can be based on 1 module by configuring default command-line flags for the Julia process; in the general case, a command can specify much more than a Julia module. The -m flag of the julia command for running a package’s @main does demand the package’s name, but that obviously compromises the visual parallel to a callable module.
I really don’t understand why you claim this package has anything to do with Apps or shell commands when it already does exactly what it’s named: make modules callable by forwarding to an annotated function. If the intent was to justify calling the module MyMod(args...) instead of a documented function MyMod.main(args...) or even an exported alias const approxappname = main among others, then you’ll need to figure something else out that doesn’t misinform.
The above method is unambiguously type piracy, by any definition. There ought to be nothing special to consider here, independently of any intended application of the package or whatever else. It is really a “textbook example” of the worst kind of type piracy.
To spell it out:
Suppose your package gets registered.
Suppose the next day some other package registers the same method.
Both packages get dependencies.
The ecosystem is now fractured!
Alternatively, suppose JuliaLang/julia itself defines the same method.
This does not make sense. Functions are merely an example of a callable object. It is not even clear when should a callable object be (or not be) considered a function. Basically, “function” is not formally defined, sometimes it means “value of type Function”, but largely I would say “function” is just a synonym for “callable object”.
That is like shooting yourself in the foot to be safe from foot guns.
One way to look at this: if your package gets registered, to be fair and consistent, other packages that define the same method ought to be allowed to be registered. So there is really no attempt at “coordination” here. What do you mean by “coordination”?
To achieve actual coordination, you would have to make a PR to JuliaLang/julia and get it merged and released.
I feel with you here. The current solution regarding apps seems overly complicated from the UX perspective. As far as I gather, a design goal for apps is that a single package should be allowed to have multiple entry points, thus the app names were decoupled from the package name. But I agree with you about this making things more complicated. I feel some piece may still be missing regarding the apps design.
Thank you.
I feel I would need a clear picture of a concrete application (or intended user interface) in my head before I could start thinking about this. I guess I have not used or created enough apps yet.
Existing / old packages doing something bad does not give new packages permission to do something bad.
Personally, if I’m aware of type piracy in a package registration, I would only let that registration merge if there was an extensive discussion about the type piracy in that particular context, with a strong consensus that it’s okay for that package. Ideally with a confirmation from a core maintainer or a someone else I would trust to have a deep understanding of the implications.
A comment like Benny’s would be a strong blocker. And actually, automated checks for type piracy would be pretty high on my wishlist for things that prevent auto-merge. I would consider type piracy one of the most pernicious problems in a package, and something we should “ban” almost entirely.
I could see exceptions for “front-end” packages that are not intended as a dependency, e.g., to override the behavior of the REPL, or something like Pluto.
I’m happy that this is just an opinion and not a strict requirement for Julia code / packages in General.
See here JuliaHub for a tiny glimpse of package that do type piracy. This search only finds packages that use Aqua + actively aware of piracy + pattern caught by regex.
For this specific package, it’s hard to recommend anything unfortunately: overloading module calling is its whole point. So, there’s no way to have a first useful version without piracy, adding this overload later after it’s already registered.
I tend to agree that the existence of piracy should not preclude a package from General registration. After all, adding a dependency is always opt-in. that said I would generally advise projects, in their individual capacity, to bias away from depending on packages with piracy, and likewise advise authors, in their individual capacity, to avoid using piracy. But there are lots of ways that a package might be designed suboptimally and General can’t possibly hope to police these all.
I think packages, let alone ones in a curated registry, ought to disclose, explain, and label type piracy or unstable internals. CallableModules is a good minimal example.
It’s not a strict requirement, and piracy can be quite useful. But we as a community discover more issues with piracy almost every day, it can cause serious problems.
This is a pretty severe case:
It pirates for one of the most fundamental types in the language, which is in Core not even in Base.
If the package succeeds by gaining many dependents (which is one of the goals of registration), then it makes lots of potential changes to Modules a Julia 2.0 affair. Julia itself may have good reason to use similar syntax in the future, or perhaps to change the way modules work more fundamentally.
There’s no way to disambiguate this function if multiple packages want to do this, the interface is untyped and vararg so if another package (or just a user futzing about in their own personal project) defines an overload they can break any dependents of CallableModules.jl
The sole purpose of the package is this piracy
On the other hand it isn’t changing existing behavior so it avoids some of the worst sins of the common LinearAlgebra pirates.
IMO a far better solution to this is to just export the app name and mildly mangle the module name. Or best practice is typically that functions are lowercased, just export myapp
If there’s an unambiguous statement from the Julia maintainers that this signature is unlikely to ever be used in Core or Base then that solves one facet of the issue. That leaves composability problems still…
While this is true, the question might be whether this case is special. As far as I see it, all potential arguments are covered by the solution. So I think there are three possible cases to consider here:
Core will implement exactly this method: Fine, everyone can just delete the dependency on CallableModules.jl and be done with it or CallableModules.jl can only provide the macro. We can define the method in CallableModules.jl only if it is not yet defined in Core if you think this is likely despite the point above.
Core will implement the method with a different implementation. It’s hard to imagine how this would look like and unlikely due to the point above.
This gets worse if CallableModules gets versions that aren’t strictly aligned with Julia versions.
We can add a versioned Julia dependency. It would not change too much, because “package not working because dependency not met” is not too much better than “package not working because of method redefinition error”. But it would happen at instantiate time, not only at precompile time.
Secondly, this does not work as stated. […] @module_main does not do [what @main does] and explicitly supports names other than main.
Yes, it is in general expected to be used in combination, i.e. @module_main @main, if you want to do this. I didn’t want to limit the user, because the package can be used for non-CLI cases which might or might not prove useful. There you can still align the Julia interface with the module name even without generating an entry point for the executable (i.e. what @main does).
The premise that the shell command and the underlying module share a name is false.
I am not sure whether I fully understood your point here. The fact that earlier examples, where it was not possible to share the name, have not shared the name, does not seem to conclude anything one way or the other. And while I fully agree that there are cases where this name synchronization is not useful, e.g. as in your example where one package creates multiple app commands, this does also not seem to imply that there are none.
I really don’t understand why you claim this package has anything to do with Apps or shell commands
We all know that naming is hard and I am totally open for suggestions here. My reasoning was: Let’s name it according to what it does (i.e. making modules callable) and let’s describe for where it might bring a benefit (i.e. have a common name in apps and in Julia).
One alternative to trying to make the module itself callable would be to create a macro in the module with the same name as the module, that just acts like a function, i.e.
module MyApp
export @MyApp
function main(args...; kwargs...)
@info "got" args kwargs
end
macro MyApp(args...)
:($main($(esc.(args)...)))
end
end
That said, I will say that while I think the way you’ve gone about this seems rather dangerous, I do kinda think the concept of callable modules is kinda nice. It’d be cool if there was a better way to achieve that.
This seems pretty harsh and argumentative, I think these forums would benefit from you engaging more gently, as psychological safety leads to better collaboration and outcomes.
I don’t see how and certainly didn’t intend it, but if you believe I violated the community standards there, you are welcome and encouraged to flag it.
kind of a side note but I don’t think this is quite true. just because packages do things doesn’t mean Base can’t change it (unless the docs specifically promise not to)
It certainly raises the friction to do so. You’re right Base explicitly doesn’t promise that this signature will be stable but we have tools like PkgEval for a reason. If a future Base feature wants to change this they’ll have to balance breaking the public interface of dependents of CallableModules vs making the changes in Base.
This is mooted if this can be added to Base or we can be somewhat reassured that no one in Base will want this signature.
As another aside I realized just now that I don’t have any CamelCase binaries (on my PATH) on any of my machines… The convention of lower case binaries matches Julia functions.
While this is certainly true, I think there are two things to consider:
Adding a different functionality for callable modules or implementing it differently compared to what is done in CallableModules seems unlikely. At least up to now no-one came up with an example (this is probably too tempting as a challenge, but then we at least know).
Your reasoning considered a static package system. So even if such a package should come up (note: it does not seem to have come up in the last ~10 years), both package maintainers can (and hopefully do) collaborate and hopefully find a backwards-compatible solution. And even if it should not be possible to find a backwards-compatible solution, it is still possible to have a backwards-incompatible package release. So even if Julia can’t move too much due to its compatibility constraints, the package system can (although I admit that there are practical limits).
For callable objects type piracy is harder to avoid in general, as there is no function name to distinguish during dispatch.
This does not make sense. Functions are merely an example of a callable object.
Thanks for the correction. I meant that if I wanted to add a “regular” function to Module I were free to choose a new function name and could therefore make it non-piracy. I could even use an existing (relative) name, but put it in my package’s namespace to make it non-piracy. I can’t do the same with the “functor”, as they share the name with its type (that’s their purpose) and in that sense when doing it you can only avoid doing type piracy with the type of the argument Tuple and not with a combination of the function name and argument Tuple and in that sense type piracy is harder to avoid.
if your package gets registered, to be fair and consistent, other packages that define the same method ought to be allowed to be registered.
I value fairness and consistency, but I think if CallableModules would ever be a frequently used package (which would surprise me, as it seems rather to solve a niche problem compared to some other packages), I think it would not be too much to ask the author of a registering package to ask for collaboration first. That is, however, for the general registry maintainers to decide. And I understand that there might be no processes that detect this and if so this lead to a problem which would need to be solved by collaboration afterwards when it is detected (see above).
So there is really no attempt at “coordination” here. What do you mean by “coordination”?
I mean that every package which wants to have its module callable should depend on CallableModules. This would solve the obvious problem of type piracy in this case, although I admit it can’t solve the less likely (unlikely?) other problems of type piracy following from this definition.
To achieve actual coordination, you would have to make a PR to JuliaLang/julia and get it merged and released.
I will do this if there is some indication from Julia maintainers, that this is the preferred way. Up to now we have the opposite indication (see above). Additionally, the typical stance with such a feature seems to be “first try it as a package before we commit to it in Base/Core”.
I feel some piece may still be missing regarding the apps design.
I feel the same. I had the hope, that CallableModules might be (one of) that piece(s).
To add another perspective as a side note: We are using the (functionality of the) package for closed-source apps (in that sense it is a “front-end”). The risk of such a use on the (public) Julia ecosystem seems negligible. Although that’s obviously not the only way a package like CallableModules.jl can be used (hence this discussion is useful beside this point which is why I hadn’t mentioned it), too strict requirements for the general registry would make it harder for such non-OSS developments (leading to the more than one “official” registry again). There are pros and cons for us as a community to support open source code which will be (in the general case maybe even: primarily) used in closed source code, which does not fit the scope of the current discussion. I only wanted to highlight that it is a trade-off even if I agree that a higher bar might make sense (although I would personally try more transparency first).