So, answering more specifically: I would check how many modules in your package can be translated into independent packages with a well defined functionality. That turns out to be very useful later, sometimes (happened to me) a subpackage ends up being more useful to the community (and to you) than the original package.
Then, structure your project into a set of packages, and try (using good sense) to keep a single module in each package, with justified exceptions.
This is a noble objective. But you won’t get that with Julia easily. (and ps: I do like how this works in Python).
I have a 100% success rate for functions that are defined inside my own module/project.
It is true that it is not the case for navigation inside external packages. Usually I can click/jump into the source code on function from an external package but then I can’t continue to navigate further.
What I often do when I need to make a deep inspection of an external package is to install it’s repo and develop inside this package for a while. Hence I recover my 100% success rate.
This is precisely what I do as well (even though ideally it would be readable from the public repo alone…), and I still have far less than 100% success rate. possibly I should start reporting more of those instances as bugs when I observe “jump to definition” not working. I had assumed it was common knowledge that it was flaky functionality
I don’t have much recent experience with largish Julia projects. But I am working on one now. At the moment, it has only one level of modules. Modules are more or less namespaces in Julia. Structs tend be be lighter weight than Python or C++ classes. Perhaps more like C structs. So even people who like to use scopes to prevent chaos often use more than one struct in a module. What I am trying now is: PackageName.jl has one module, it includes all of the other files in one place. As you found, a file can’t refer to things that have not yet been included, which is unfortunate. However, at least for my projects, I don’t often have to shuffle them. Then, for the most part, every file has a single submodule, and of course it explicitly imports things from the other submodules.
I default to a submodule for every file, even if it is a small file. A lot of people, myself included, find this much better for at least two reasons. Navigating and understanding the dependencies in your project. If the project is big, this includes even my own project that I am actively working on. The other reason, which is closely related, is to ease refactoring/reorganization. Empirically, I find it easier to reorganize, refactor, if the dependencies are explicit. I should also say that it does not make sense to split these small file/modules into separate packages. The reasons for making a bit of code into a package and the reasons for making it a file containing a single submodule often have significant overlap, but they are by no means the same. Sometimes one is clearly better.
I don’t think I saw the organization that I’m talking about it your list of options. But I think I may have somehow overlooked it. Basically it is: Include all the code from one within one main file, deal with the dependency problem by hand :sad-face: . Do use a single module per file and put import and or using statements at the top of each file.
Finally, I wonder if it is possible to do an iteration on the design of FromFile to get something that would work for you. Of course, this is a big diversion from your immediate goal.
I am glad to know this now, but I still feel strongly that such tricks (and these really are “tricks”) should not be necessary for something so fundamental as to discover “where is this code defined.” Especially given that not everyone wants to use VSCode.
Unfortunately this is not one of the major Julia pain points pushing people appart.
Paraphrasing Tamas Papp, Julia is not an easy language. It is based on a different paradigm than most other languages. That comes with bonuses and onuses, as it is currently implemented. I wouldn’t judge it before having digged a little bit further into it to grasp what’s good and what’s not for your project.
@ffevotte explained to me a while ago how to place a main function inside the module sources during the development of a given package (leaving the launching scripts for later use). This technique should be better known.
P.S. It is not only for vscode users (it is related to LSP).
I see several reasons why this setup is often used and works reasonable well in Julia
Julia code bases tend to be smaller than in many other languages, especially compared to C++.
Methods are open for extension and compilation is deferred until they are called. Thus, there are few hard dependencies – mostly types need to be defined before methods can dispatch on them.
It might also be a historical relict from Common Lisp
Except for things introducing compile-time instances, i.e., types or macros, you can actually refer to them before they are defined. In particular, if your files only contain methods they can be included in any order without issues. Thus, it’s often rather easy to align the inclusion order of files, i.e., first macros and types and finally methods (in any order). In particular, Julia does not force you to define methods in the same file as types – yet, in smaller code bases I tend to keep types and methods on these types in a single file.
To me it looks like the Julia design encourages one developer per package (which becomes a monolith with tight coupling); a team developing in parallel would be working in a package ecosystem (perhaps with a local registry).
This may have a bit of overhead but has the advantage that relies and benefits on the package manager for dependency handling, version checking etc) and may allow for better scaling than traditional options (intra package imports and the likes). The downside from a single developer perspective is that following local dependencies is a bit of a mess.
I think you can use Base.functionloc for that… It is used by the @edit and @which macros if I remeber correctly. Works only for functions/constructors though, for constants one has to grep.
As the only Julia dev in my team (I’m the one doing mathy stuffs), one common issue is to propagate the updates “up” across dependencies.
In such case, I typically end up having a sandbox project, where I “dev” all relevant packages. This adds a lot of flexibility, to do a lot of trial and error with the REPL. But this can also quickly generate some mess if you want to register new versions of each package. It should be done in a clean and ordered way (you have to “free” each package in order, resolve, and test regularly that everything is OK).
I don’t pretend this is the optimal workflow (especially in regard of the “team” structure), maybe I went too far in the “spit package” approach, or maybe this is the sign of some tight coupling (not in terms of definitions, but in terms of usage). In any case, I wish there’d be a simpler way.
Conversely, I like the approach used in Go, with packages defined separately inside a module, that can be included separately (mind that meanings of “module” & “package” are swapped compared to Julia).
I can’t find it now but Matthijs Cox at ASML just published a blog post on their internal workflow (as I understand it they have hundreds of Julia users internally and since the demise of Invenia are now probably the biggest production Julia users) and it was based around a local registry.
Tribe was slashing its internal valuation of Canadian-British startup Invenia, on which it had bet $30 million, by 95%. Invenia co-founder and Chief Executive Officer Matthew Hudson had been “terminated” and a board-led investigation found he’d “secretly, systemically and repeatedly inflated the revenue and profitability of the company,” according to the memo, which was sent by Invenia board member and Tribe CEO Arjun Sethi.
this is not one of the major Julia pain points pushing people appart.