[ANN] UseAll.jl – Temporarily Demodularize Julia Code

I hereby announce the public release of UseAll.jl providing @useall which is effectively a small tool to ease REPL-based worflows by pulling the content of modules/packages into the current namespace.
This simplifies developing code inside of a package from the start and thus shares the spirit of (and combines well with) Revise.jl and BestieTemplate.jl.

This is already used internally in our company for some time as code run in startup.jl now made into a proper package, with some features added and released to the public.

The following would normally be done with package code, i.e. having the code in a file. But for convenience, the following example will be REPL-only:

module MyModule
    f(x) = g(x)
    g(x) = 42 + x
end

You might want to test/debug/run this with

julia> x = 12
12

and copy lines from the module/package into the REPL to execute them. Unfortunately, you can’t:

julia> using .MyModule

julia> f(x)
ERROR: UndefVarError: `f` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ REPL[6]:1
 [2] top-level scope
   @ REPL:1

Instead, you need to modify each line (the function name and all method/constructor calls used to create method arguments):

julia> MyModule.f(x)
54

Alternatively, you can switch the currently active module in the REPL by entering MyModule and hitting <alt>+<m>. But this often does not work, too,

(Main.MyModule) julia> f(x)
ERROR: UndefVarError: `x` not defined in `Main.MyModule`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ REPL[6]:1
 [2] top-level scope
   @ REPL:1

as your code depends on previous definitions in your REPL session or in your startup.jl which is not available in the module/package you want to run code in.

With UseAll.jl, you can simply call to make it work

julia> @useall MyModule

julia> f(x)
54

and use everything afterwards as if all definitions of your package/module were done in Main.

Note the missing dot, i.e. @useall MyModule instead of @useall .MyModule which is needed, as the latter one does not parse and therefore cannot be used. This is normally not an issue, as most of the time you will be working with packages anyway which never need the dot prefix.

This, of course, assumes, that you have the package installed and have executed using UseAll explicitly or added it to your startup.jl (the latter one being recommended).

It should be clear, that the demodularization which @useall effectively does is a terrible idea in general when used in packages. This is a development tool meant to be used in the REPL. @useall can be useful, however, if you have something closely coupled to a package, e.g. a runtests.jl which should test many internals of the corresponding package, where it might make sense to just @useall on the package to have all identifiers readily available in runtests.jl.

UseAll.jl tries to integrate well into your development workflow.

  • When used in the REPL, it tries to support tab complete for package names. It “tries” because private REPL API needs to be used, so this is not guaranteed to work with all Julia versions, although it at least should fail silently without disturbing anything else if API changes.
  • When used with Revise.jl, it tries to keep bindings in sync, i.e. if you load a package with @useall in Main and add a binding to the package later (in the same REPL session), the symbol will also be added to Main. The caveats with “tries” are again the same.

The package is not yet registered, so currently you need:

]add https://github.com/PatrickHaecker/UseAll.jl/

I hope to find a suitable Github organization to maintain it there to avoid a single point of failure (maybe someone from JuliaDebug · GitHub is interested?).

13 Likes

I’ve wished for something like this for a while, are you planning to register UseAll?

The caveat in the README

Do not use UseAll.jl in a regular package.

would be a strong argument for not putting this into the General registry.

I can see how this package could be very useful for certain REPL-based development tasks, though. So the argument for registration would be more visibility.

But packages don’t always need to be registered to be useful. Installing from GitHub is a perfectly fine thing to do.

So I’m a bit undecided as to what I would recommend in this particular case.

3 Likes

Yes, it’s obviously a REPL / debugging / … package. But we have quite a few such packages that are registered, even though they are not meant to be used as dependencies.

8 Likes

Is there probably any technical way to exclude using package as a dependency from inside of that package?

Yeah, I’m wondering about that: Is it possible for a package to detect whether it is being used/imported in the REPL or inside another package? If that’s possible, UseAll could print a big warning if it is being used in another package. In that case, I would say: Go ahead with registering it.

[P.S.: I’ve spun that question into a separate thread: Can a package detect whether it was imported in the REPL vs inside another package?]

Either way, I’m not strongly opposed to it being registered. But I’d also point out that registration in the General registry isn’t may quite as important as people often make it out to be, and, since we have the requirement that registered packages can only depend on other registered packages, not being registered is good way to enforce it being a REPL-only tool.

I agree with the logic, but isn’t the caveat a bit extreme to begin with? Say there’s a package that does @useall its dependencies and does all the crazy things that PatrickHaecker is warning against. That mostly doesn’t matter to the package’s users unless ExportAll.jl, which is in General, is also used to expose all those names to them. There’s an argument here that a package developer is free to do generally bad practices (like relying on internals) when it makes sense and isn’t exposed, and a runtime warning wouldn’t really help then. It’s already quite easy to check a package’s Project.toml for UseAll, and it’d show up in Manifest.toml if an indirect dependency ever uses it (but factcheck me if test dependencies also show up, which we may want to ignore). Dependency tree reflection for a particular dependency would be nice though; I think there was a related package announced recently.

You could support a secondary syntax @useall using .MyModule like Reexport.jl.

That would actually work great in conjunction with this package because it’s easy to run into import name conflicts upon multiple implicit imports, and @useall introduces more names per module that could conflict. There is an element of chance here, but the chance is very high for the standardized @main() resolving to :main. Dumping Main into each package would be less conflicting than dumping many packages into Main, but namespace baggage in our actual projects is a session-irreversible source of bugs. Either way, it’d necessitate a time-consuming session restart.

To minimize the number of possibly conflicting modules and to partially put the “temporarily” in the title, we can make and switch to throwaway submodules in Main just for REPL experimentation, and each one can isolate a pair of @useall Main; @useall MyModule. If that still causes issues, we can make another Main submodule and rename some imports instead of @useall Main, maybe work with new isolated globals. Those modules aren’t really temporary because they’re all kept alive in the session, but they don’t affect other modules and we can switch away for the rest of the session.

3 Likes

Thanks! I think it would make sense to register it, similar to other (mostly) development packages like Revise.jl, BestieTemplate.jl or JET.jl.

However, I’d prefer to maintain it in a Github organisation, as I have seen enough packages where the maintainer has disappeared and PRs could no longer be accepted. I am still unsure how to best find such a Github organisation and why there are still so many packages in personal repositories, but this is probably stuff for a different topic.

Of course, whether it actually succeeds in being registered is for the general registry maintainers to decide.

1 Like

Hoo yes thank you! This is perfect for people coming from R who are used to devtools::load_all(). I was missing this in my workflow and was frustrated, but not enough to create a package ^^. So thank you for that, will test soon.

1 Like

I very much applaud that attitude.

seems like a great fit to me. Don’t hesitate to just reach out to some org members to help you facilitate the move. Maybe @aviatesk

Otherwise, you can also always create your own org, but it’s best of an org has at least two (public) owners. If there’s no natural second maintainer, we’ve sometimes invited “outstanding community managers” as owners of an org, even if they have no direct connection to the packages in the org. This by itself helps to reduce the “bus factor”: If the original owner/maintainer disappears, there’s at least a point of contact to put in a successor.

I found the arguments in [ANN] UseAll.jl – Temporarily Demodularize Julia Code - #7 by Benny pretty convincing, so you won’t get any complaints from me about registering this, at least.

1 Like

Maintenance would be great. The macro itself doesn’t appear to be doing anything beyond base Julia v1 API, so that part will last without any activity. The problem is the extensions with REPL internals and Revise integration, which change a lot alongside base Julia even when it’s not breaking.

Some nuance though. You don’t technically need an organization (Revise.jl doesn’t), but those packages usually well-connected enough to find maintainers and have a lot of contributors from invested organizations anyway. An organization can also fail to maintain or even archive a repo if they and the public lose interest. There’s a real risk of that because @useall is basically using unfettered by export statements, like Python’s from mymodule import *, which is widely considered impractically dangerous. People would likely still be more cautious about name conflicts and Revise temporary globals in the developing modules before trying anything mentioned here so far.

Sure, but JuliaDebug in particular has several core contributors as owners, I think, so even if Boston gets hit by a asteroid, it seems pretty likely that there still be someone left who could at least assign a new maintainer to any package in the org, if that ever becomes an issue.

Nothing is ever guaranteed to last forever, but the more packages we have in more organizations with more owners, the less we have to worry about having to jump through hoops for resurrecting abandoned packages.

Going of on a tangent…

Personally, I consider the Julia’s standard using MyModule as “impractically dangerous” as Python’s star-imports. After all, Python’s star-import import only exported members (things listed in __all__) as well. Granted, the default __all__ is “everything that’s doesn’t start with an underscore”, and that makes a difference.

But I wouldn’t mind if in a hypothetical Julia 2.0, we replaced export with public entirely, and made using MyPackage an alias of import MyPackage. Probably never going to happen, of course, but maybe we can push more for universal adoption of ExplicitImports, have more Linter support for flagging imports, normalize package READMEs explicitly importing symbols from the package, etc.

In that context, a @useall macro would be a nice feature (even a core feature!), because implicit imports can be useful in the REPL or a throwaway notebook. They just never should have been such a normalized part of the language. It’s really a shame that we didn’t learn from the experience of the Python community that took years to culturally eradicate ubiquitous star imports throughout the entire ecosystem :wink:

4 Likes

Thanks, I didn’t know this syntax. Although using this syntax is more effort, it is more generic (it can distinguish a submodule from another root module) and users from Reexport.jl probably expect it, so I implemented it, too.

To minimize the number of possibly conflicting modules and to partially put the “temporarily” in the title, we can make and switch to throwaway submodules in Main just for REPL experimentation, and each one can isolate a pair of @useall Main; @useall MyModule.

Name conflicts can limit the usefulness of UseAll.jl. Yet, we haven’t observed any practical limitation in the last ~1.5 years of using this. If there are conflicts, they are shown, but up so far there was nothing, which was more work than adding a single module prefix.

However, some usage patterns most likely are more prone to a significant number of conflicts than others. Therefore, your idea of a temporary namespace is very interesting. Note, however, that it already exists, it only lacks a good user interface:

# Create temporary namespace
julia> module TempNameSpace end
Main.TempNameSpace

# Set active module: TempNameSpace<alt>+<m>

# Prepare temporary name space
(Main.TempNameSpace) julia> using UseAll

# Fill temporary name space
(Main.TempNameSpace) julia> @useall Main MyModule

# Work in this temporary unified namespace

# When you are done, reset active module: <alt>+<m>

# Optionally empty temporary namespace (assuming Revise.jl):
julia> module TempNameSpace end
Main.TempNameSpace

I would wait with an implementation until I get reports that conflicts are really relevant. But if they come in, we already have a solution. Thanks again for this, @Benny.

1 Like

People have a natural tendency to avoid conflicting names across different projects, and natural languages tend to provide enough synonyms for this to be feasible. After all, it would be terrible for the convenient using A, B to run into overlapping export sets. But there are definitely exceptions to that like main and parse (base Julia alone has Base.parse !== Base.Meta.parse).

By name conflict however, I don’t just mean importing conflicting names. Increasing the number of global variables by merging namespaces, even disjoint ones, can get in the way of making more global variables in development or hide unintended global variable accesses. For example a global const e = float(ℯ) from a different module would collide with making a @enum Alphabet a b c d e f ..., now silently across world ages in v1.12+ (whoops, not for imported names), or it would hide a forgotten local e assignment that could’ve been caught with a runtime UndefVarError. But these are all expected risks.

I would mind a lot because I don’t think most public names ought to be exported, just the most convenient ones for easy implicit imports (bare using in v1). Unless I’m misreading this and you mean we don’t do implicit imports at all. Implicitly importing all public names actually feels like a somewhat dangerous practice that UseAll could support now; naively I’d think it’d just need an extra Base.ispublic check (would need to change the [compat] to v1.11+ though).

The number of conflicts will increase the longer you work in your REPL and the more modules you @useall. This is parallel to other usage patterns where you copy-paste code types and methods into the REPL (possibly editing them until they work), but of course with an increased speed.

Nevertheless, we have not observed this as a problem yet. Maybe the reason is that @useall is more like Pkgs develop while using is more like add and typically you use a lot more packages than you develop, so in practice it seems to be rare to @useall more than a handful of packages in a session. Combine this, with at least the VS Code workflow, where each repository typically has a separate window and therefore a separate integrated REPL.

Conflicts like main do happen, but these are trivial. More tricky conflicts can happen, but the good thing is that we all write our test cases and tests are run separately anyway, so you have the safety ropes there independent of which conflicts you created in your development workspace.

I don’t think most public names ought to be exported

I read both of you statements such that you both agree. I use using without specifiers as

  • a REPL feature
  • when creating a package and want to make fast progress and clean it up later
  • when the used package only has a single export and it’s not too likely to change drastically

and like its simplicity in these cases.

Implicitly importing all public names actually feels like a somewhat dangerous practice that UseAll could support now; naively I’d think it’d just need an extra Base.ispublic check

I think I didn’t get this one. UseAll.jl should not be used in a regular package, so the problem should not occur. And even if someone (mis-)uses the package for this, I would not worry about the implicit import of the exported and public bindings, but of the private ones, which are typically dominant.

Or are you proposing a @usepublic which adds all exported and public bindings, but not the private ones, to a module? That sounds interesting, but I am not sure what the use cases for this would be. In packages we do not want implicit imports and in the REPL this is probably only useful in case of relevant conflicts.

3 Likes

Basically only experiment with packages’ public names, reduce conflicts (for example, it’d solve the prior e-Alphabet conflict because the @enum is likely internal), and leave more possible names open in the @useall module. Revise-ing the non-public names in the package would still happen, you just couldn’t reference them in the @useall module. Granted, you’d often want to directly call non-public functions in development, but that’s not always necessary. @usepublic might also be convenient or more palatable for people who refuse to directly test internals (there’s debate on this but some swear it’s less brittle), though I personally prefer announcing what I’ll test.

The usefulnames function also just made me think broadly about filtering names for import sets. So far, we have 1) export set via using, and 2) all names except # via @useall. We have many other potential sets of names on the language level: public, const, function, struct, module, etc. or their complements. If this were an uglier eval-loop function instead of a macro, it’d even be possible to let users input their own name-filtering function, though I couldn’t imagine a practical use for that. public was the only one that sounded possibly useful for development and testing.

1 Like

Indeed, we do! :slight_smile:

In addition to something like ExplicitImports.jl, reducing or even eliminating export declarations in favor of public could go a long way at shifting cultural norms towards more explicit code. Of course, that might make UseAll all the more useful!

2 Likes

Note that you can also set the context module in the REPL with super + m:

julia> module MyModule
           f(x) = g(x)
           g(x) = 42 + x
       end
Main.MyModule

# MyModule in REPL buffer then super + m:
(Main.MyModule) julia> x = 12
12

(Main.MyModule) julia> f(x)
54
6 Likes

This is also mentioned a bit in the original post, but the complaint is that sometimes you have x defined in MyModule and f in Main, which is annoying.

1 Like

Additionally, if you aren’t restricted to the non-public code of a single module, you constantly need to switch the module which amplifies the problem of your variables defined in the “wrong” (i.e. not the currently active) module scope.

With @useall you just bring both modules into your module scope and can happily copy-paste ever after. :grinning_face:

So while switching the context module works for some cases, I personally stopped using it when I had @useall, because I valued the simplicity of a single approach and @useall scales much better for my use cases and reduces the mental load.