This isn’t really the case in a real world use case when I add some new features for a big package and want to seperate things in a submodule I will have to register the submodule to get it into the big package.
This is a strawman argument. Modules can be used without becoming packages.
1 Like
Then you don’t have the nice automatic dependency handling provided by the package system for a large package. I’m mainly against the saying of just do a package it’s not necessary a good idea
1 Like
You can do this. However you will end up with lots of scripts of the form
module B
using ..A
end
and you need to make sure A.jl is run before B.jl. This gets complicated very quickly, manually ensuring the exactly correct order of modules is loaded. It would be easier if a resolver did that, similar to what’s used when using various packages.
1 Like
And note putting submodule into a package is what is recommended above by various people .
This is the part I don’t agree, one shouldn’t just do a new package - it has to be registered to be used in a registered package and once registered as a package it will be something forever in the registery
I believe the main benefit of doing whatever is proposed in issue 4600 is to have a way resolving file dependencies automatically instead of manually. This is crucial when a project contains a lot source files.
A project that has a lot lines of code could still have one single file but a project has medium lines of code could also contains a lot component files. So I think the problem is not about large project contains a lot of code but for large project contains a lot of files (each might be written by different people) providing a mechanism of resolving file dependencies automatically becomes crucial. Since it’s not easy to know the dependencies manually anymore currently.
In this case registering packages is the only solution at the moment. Unless one manage the files in a centralized but hard to maintained way.
To summary this is mainly about the include twice problem.
Packages don’t have to be registered. I use plenty of unregistered packages.
1 Like
That has been said if you are working on registered packages and want to extend it by a bunch of complicated submodule you will have to register it to use it in a registered package
1 Like
If I understand correctly what it is you have in mind here:
If I extend an existing package with a bunch of functionality, I can go two ways: if the functionality is separable and generally useful, I could make it into a package of its own, and I register that package.
Otherwise, I simply use modules to provide that functionality to the package being extended. No need to register anything here.
That’s it, I really don’t see what the fuss is about.
2 Likes
if you don’t register the package, then we are back to the submodule problem:
how do you solve the include twice problem? with packages one can just do using whenever they want, the loading system would resolve duplicated using
one may say: oh ok, I can manage the files by hand and remembering all the file dependencies, but why should I? this is something that can be easily done automatically. Not to say when a project contains a lot of files, no one wants to maintain a file structure. And sometimes it’s not possible
maybe think about the following example:
- person A is doing some work in file
A.jl - person B is doing some work in file
B.jl - they both include file
C.jlbecause they think they need it, butA.jlis not yet committed to the existing package but stays in a pull request, same forB.jl
then when these changes get merged, potential problems will be
- whose include should be used? who is responsible to include things?
- if reviewer John Doe and Black Smith both think the PRs are correct separately, and there is no conflict, both PRs are merged in a short time (since the CI only runs on
mainbranch + PR, and it will say yes this PR looks good), and it breaks themainbranch
It’s usually not possible with a private project in a private repo for a private company. Things have to stay in a mono repo sometimes.
settup one’s own private package registry still has some auth issues need to resolve on the PkgServer side, which we have been working on. but not to mention this is another new extra burden on the maintainance at early stage of development. There are certainly benefits of keeping things in one mono repo.
3 Likes
I’m afraid we go round and round. What is needed is a proper for-example: a concrete case to talk about. all of these vague hypotheticals are not very useful.
5 Likes
why is my example not very useful? I’ve stated the file structure and what will happen. I certainly cannot share the actual code of a private repo right? that violates my NDA.
And I think the include twice problem has been mentioned many times on this discourse. If you would like me to elaborate more I can certainly put on more details and even make up a fake example of this.
on the other hand, I’m pretty sure @patrick-kidger has more examples
4 Likes
The question is how do you solve the include twice problem - this is that simple to anwser and can we just answer this question?
Or we just discard this problem and say this is not a problem, and you should just do it by hand?
also maybe read the discussion here first, I think the include twice problem has been mentioned quite a few times, and not just me thinking this should be done automatically
and for whoever is looking for an actual package suffering from the include twice problem, I’m afraid there is none. When a package gets registered one has already workaround this by either splitting things out into packages or doing it by hand, I still remember some package authors would left a comment in their top module file like # do not change the include order, but I cannot remember which package is this, maybe someone could remind me.
I mean again isn’t this simply a human brain burden? why are we doing something that can be done automatically?
2 Likes
As has already been pointed out: include once, import however many times you wish or find necessary.
include("A.jl") # module A ... end
import .A
import .A
import .A
include is meant to be used for “stitching” together separate files as if they were one. It is not meant to deal with dependancies or whatnot. It is not meant to be used to separate functionally distinct code blocks that can work by themselves (even if they may depend on each other) but rather to split too long files into semantic* blocks that still belong and function together. This is the very reason why order matters for include.
(* I am very probably using the wrong word here. I hope it still makes sense.)
2 Likes
Yes, this is exactly why we should have this done automatically by whatever is proposed in issue 4600 isn’t it? One can use include if they likes it for sure, but we should at least provide a way to resolve the file dependency automatically, I mean this is doable and release people from brain burden, why are we doing it manually?
also note, your solution requires one to include all the module files in a top level file (e.g the package module file) one still cannot resolve the problem when seperate people include the same file at different level.
In a mono repo contains some sub-projects, it is very likely one would only look at a sub directory, e.g
.
├── LICENSE
├── Manifest.toml
├── Project.toml
├── README.md
├── src
│ ├── A.jl
│ ├── B.jl
│ ├── C
│ │ ├── C.jl
│ │ └── D.jl
│ ├── D.jl
│ └── MyPackage.jl
└── test
└── runtests.jl
without a strict requirement of everyone working on the same project, this would either results in creating the same function twice in two different modules, or results in creating a warning of one module replace another during precompilation.
2 Likes
The issue that two people might accidentally write the same code independently exists independent of implicit modules. You don’t want two people to write the same method in different places, and you don’t want two people to write the same include statements in different places.
I don’t think I’ve seen a Julia project where include weren’t all collected in one place in the main source file. If you think you’re at risk from double-include, just follow that convention.
3 Likes
yes I don’t want two people to write the same include, but if this happens at the same time, you cannot prevent it. It must take extra manpower to resolve this by either doing it by convention or doing it by one same reviewer. I’m just saying this is not necessary, since it can be done automatically.
Isn’t this because we don’t have a way of resolving file dependencies?
People, the advocates for this change are clearly aware of the fact that you can do this manually. It’s not very helpful to continue pounding on that point.
For examples of packages that have included comments about include order, JuliaHub search pulled up:
The last is most interesting.
16 Likes
oh nice! You just find what exactly I was looking for! I didn’t know this feature of JuliaHub until now.
Yes, exactly, if machines can do it for us, why are we humans doing it? If doing things machines can do is preferred why are we still programming? We should go back to the stone age and do everything by hand.
1 Like
This is simply not true. You include the definition of your submodule once, then you import it however many times whereever anyone may require it. No repeated code necessary.
But I do get your point about manually dealing with the order of includes. You do have to make sure that submodules are included from top to bottom in their hierarchy order, otherwise submodules can’t load properly. But, sincerely, how difficult can that be? If B depends on A, then include B after A, and so on. You must always have a non-circular dependency, so their is always a correct order.
1 Like