With this, you register local path dev/UsefulLib as the source for package UsefulLib within the context of the TopProject environment. And so using UsefulLib would work anywhere in the sources of TopProject.
Is that not what you wanted? Or maybe is the process too “heavy” and not convenient enough?
module MyTopLevelPackage
module MyUsefulLibrary
end
module FiniteElements
module Geometry
end
module Mesh
using MyTopLevelPackage.MyUsefulLibrary
using ..Geometry
end
module Assembly
end
end
end
Hmm, interesting. That would indeed provide a solution for at least a certain type of use case. It still feels like a lot of effort for something that should really be easy and simple, though.
The reason it is useful to know the “why” is because then we can point to a solution which is idiomatic Julia.
This is true for code loading, and everything else you want to do in Julia or any other language.
As an example, lets take the typical OO notation object.some_function() in Python. In python it is idiomatic to use this. In Julia it is not. So if someone asks how to do object.some_function() in Julia, the answer is usually : “What are you trying to achieve? Because even though it is technically possible to do it, it is not recommended. You will have better performance/structure/reusability if you do something else.”
If we knew why we might be able to suggest alternative approaches to solve the problem, but if it is a trade secret we’ll have to leave it there.
For what it’s worth we have a C++ GUI application at work which can call Julia, via embedding, for certain functionality. There we can update the Julia code while the application is running with the use of the Revise package, subject to its limitation that types cannot be changed. But this is maybe completely different from your use case.
Also, besides dev you can also just use add to reference unregistered packages by path (then revise won’t track them though) [EDIT: That only works if the package to be added is in a git repo].
From 1.11 onwards that’s even easier with the sources blocks in the Project.toml
There is really no difference between a package and a module: each package just defines a module. The way of using a module in julia is with, well, using or import.
But somehow the julia process needs to know of the existence of that name/modules. There are two ways to achieve that:
Making use of Pkg by marking your module as a package and declaring the dependency in the toplevel Project.toml.
Evaling the module code in your running process (i.e. include), thus making Julia aware of the existence of the module. This enables subsequent using/import.
Personally, I think there is rarely a reason to use multiple modules within a single package. I don’t care too much about the namespacing and if your goal is composability/reuse, it needs to be a package anyway (otherwise other projects cannot refer to it). In my experience submodules are rarely used, even in big julia projects. Of course, this is subjective and somewhat a matter of taste.
You’re putting things out of context. This was in response to @mhinsch saying that “the module system should be completely independent of include”. And I argued that it already is: for any arbitrarily complex module hierarchy, you can decide to define it all in one file, or to split it in as many files as you want, with the filesystem layout that you want.
Which I think is the very definition of independence: the choices you make for module structure do not limit you in the choices you make for source files.
I strongly disagree with that. There’s a reason why encapsulation and separation of concerns are important concepts in software engineering.
More practically speaking - one of my projects is a simulation platform that consist of many different subsystems. For a given simulation project the person who designs the model should be able to pick and choose from these subsystems and combine them with as little overhead as possible. I started out with a simple collection of plain Julia files that were loaded into the main simulation with include, but quickly realised (what software engineers have been realising for decades) that encapsulating the subsystems into modules made for much more robust code and avoided many subtle errors. For now I am getting by with an adjusted load path to make the system work, but if I ever wanted to release this as a package, keeping the modularity would be a PITA.
I do agree that there is a bit more to a package than to a simple module. But frankly I think you should forget about LOAD_PATH. I tend to think anything relying on fiddling with LOAD_PATH should be discouraged nowadays, especially since a lot of recent developments in Pkg have been devoted to providing new and safe ways to manage dependencies in almost all cases. This has already been mentioned in this thread, but to be clear I’m referring here to things like Pkg.develop or the new sources blocks in Project.toml files.
There is one thing I do empathize with the OP is that indeed modules and packages are not identical. To make a module loadable, it should be either a package (containing a Project.toml and uuid) that you can then dev, or a module you have to bring in scope using include. Making a package is slightly more overhead than just making a module.
I always wished it would be allowed to make a simple module loadable using dev or add path/to/module. This would make fiddling with LOAD_PATH totally unnecessary
As a first step, maybe we could simply develop a small tool that automates the creation of a minimal Project.toml with a proper name and UUID? this shouldn’t be too hard.
(I have to admit I don’t entirely understand what’s wrong with the usual include-based technique for source code that isn’t really meant to be used outside the current project. But I can understand others having a different viewpoint and more tooling in this area can’t harm)
When someone asks how to do something strange (like replacing an internal module of a package at runtime), it’s all too often an XY problem, and there is a better / more native way to solve the underlying problem.
If you won’t/can’t tell us the underlying motivation, then it’s also less compelling for others to spend their time coming up with weird solutions to weird problems. (This is doubly true for implementing new features or packages, without a compelling use-case.)
In that case, I’ll simply say that Julia is less dynamic than Python in certain ways — you can’t monkey-patch it to the same extent at runtime — often for good reasons having to do with efficient compilation. Digging into the loading system to figure out a way to jigger it to do what you want here with runtime modification of paths might be possible, but doesn’t seem worth the time absent a real application — it’s certainly not a typical, documented thing to do. The closest system is Revise.jl, but that hooks in to the loading system at a fairly low level.
this is not at all commentary or blame to the current participants of this specific thread, and I intend it more to be a summarizing observation of the several 100s of cumulative comments from the times this topic has come up in threads throughout the last several years:
every time code loading & module namespacing is discussed, it seems to me that the conversation quickly starts spinning between the same two points ad infinitum
“imports in Julia feel really clunky, and I’m struggling to structure my project in a satisfying way”
some variation on “you’re holding it wrong” / “you don’t understand the tools available” / “actually language XYZ’s approach has drawbacks you’re not considering” where most frequently XYZ is Python, Go, C++
my impression is that, letter of the law, the latter camp is generally technically right. but it also seems overwhelmingly obvious (to me, although I’m sure a strong statement like this will draw disagreement) that somewhere in this design space that Julia occupies of code loading and module management — although difficult to pinpoint precisely where — there are genuine clunkiness and usability issues.