I do not use conditional include statements, but I use Requires.jl that may be seem as something similar (but I think it does conditional evals instead of requires).
The use case is simple. I have a command-line tool provided by my package and that uses the functionality the package provides. However, my package/script depends on a solver to work, only one, of about 4 it supports. I do not want to force the user to have a specific solver, or the four of them (some are paid and cannot be installed easily), so Requires.jl allows me to only load the code that interfaces with some solver if the solver is loaded in the environment the script runs. And the only thing the user must do to use the tool is to enter the tool environment a single time and add the solver package they intend to use. Or, if they want to write their own script using my package, they only need to load one of the four solver packages for things to work seamlessly.
I think that looking at Requires.jl use cases will probably reveal which packages depends on something similar to conditional include statements.
I use this feature heavily. I have a big package that provides multiple mathematical formulations for a specific problem. So each mathematical formulation has their own submodule. However, for each formulation module I do not have only the code for model creation but also for plotting the results (that change with each formulation), etc… I do not create a second-level of submodules (FormulationName.Build, FormulationName.Results, …) because it is overkill, the Results method need to know the auxiliary structs used by Build and I would need some extra imports between submodules that serve the same formulation and have no conflicts with each other. So what I did to better organize my code was to split these different features in files, and include directly into the submodule by require. I am happy with the results, I find what I want faster now, without any overhead of headers and imports.
A lingering question I have: is it ever the intent for the definition of subcalc upstream from calculate to be different in two different places where calculate.jl gets included? For instance, does subcalc do sqrt in one context and x^2 in another, or can I always assume when I read the text of calculate that subcalc will refer to the same function?
For example, like this: distance.jl
subcalc(x) = sqrt(x)
include("calculate.jl")
norm.jl
subcalc(x) = x^2
include("calculate.jl")
If something like this happens, then the meaning of subcalc in calculate.jl is ambiguous. As far as I can tell, nobody is suggesting that there is a valid use case for something like this. It sounds like the intent is always to have the same context wherever calculate gets included – even if the include happens in multiple places, it is assumed that all the missing identifiers are the same. But there’s no guarantee that the context will be the same. By putting a namespace around calculate, you provide that guarantee. IMO, unless the intent is indeed to change how calculate works in different contexts, then including instead of using leaves the door open to undesired effects.
I played around a bit, and it looks like Julia generates warnings for pathological cases (ex: including both norm.jl and distance.jl into the primary module, or even if I turn those into submodules and try to export calculate from both.) That covers most of what I’m concerned about. If neither submodule Distance or Norm exports calculate, then there is no warning. That still bugs me a little, but I guess I can get behind it because the two versions of calculate are appropriately isolated in their own namespaces, so there’s no reason they should mean the same thing.
Not to be too disrespectful because some questions are genuine but it seems to me that an aweful lot of people try to construct examples of misuse of includes and then cry about errors it would cause.
That behavior is equivalent to standing in the kitchen and figuring out that knives can’t just cut bread but also your arm. And then cutting your arm. And then proudly showing your cut arm. And then arguing knives should be prohibited.
How about not cutting your arm?
Just because Julia allows you the freedom to be a bad programmer doesn’t mean you have to be one. There is nothing wrong with includes. If you desire a simple way to import single files in their own namespace, then by all means, demand just that. But don’t equate that to includes being wrong or harmful.
I don’t really see how you can look at that issue and conclude that there is “no proper discussion” or that dissenting voices are “slapped down”. Moreover, how is this kind of comment constructive? It is dismissive of the work and thought that have gone into that thread from all parties — which apparently doesn’t warrant the label “proper discussion”. It is also discouraging to core devs like myself who have been trying to move it in the direction of an actual solution, since we are apparently “slapping down” dissenting voices. In general, this kind of meta-complaint is not only completely unhelpful, it’s also one of the more exhausting things about open source. Please don’t do it.
I am sorry if I offended. The comment was really intended to adress the last week or so, not all the work that went into this thread.
I offered some opinions in the issue thread in the past few days, which could be classified as “dissenting”. To my several posts I got a bunch of “thumbs down” with no response/discussion (ignored), and only two meaningful, politely disagreeing, replies. I don’t know, it seems like a critical voice is not welcome?
I thumbsed down a few of your comments because they were steering the discussion away from what I saw was a rather productive brainstorm.
Personally I’m still not sure if I’m on board with a change — partly because I don’t even know what the change would be. I find the status quo perfectly sufficient and dislike python’s system, but I don’t even know what this will end up looking like in a possible future Julia. No one does. I’m curious to see where this goes.
Your view has long been abundantly clear and some of the successive comments weren’t adding much. It’s not that your view isn’t welcomed or that dissenting voices are silenced, it’s that we’ve already heard your voice loud and clear.
Thank you for your perspective. I am afraid I rather gained the impression that this was a done deal and raining on the parade was undesirable and hence discouraged with disdain and .
Just to briefly chime in on the readability point of view, as a user of @patrick-kidger’s FromFile.jl approach: I’m finding it quite a bit easier to work on my SymbolicRegression.jl package after the switch at version 0.5.0. Here is a glimpse of the old code:
At this point in time I used include(...) in my main file. Even as the person who wrote the package, oftentimes I would lose track of which type or method was defined in which file (which perhaps is because I don’t use any IDEs to manage this, just vanilla vim). This happened enough times that I wrote a script to (approximately) help keep track of which file declares which token:
for f in $(find . -name '*.jl'); do echo $f && cat $f | vim - -nes -c '%s/#.*//ge' -c '%s/"""\_.\{-}"""//ge' -c '%v/^\S\+/d_' -c '%g/^\(end\|@from\|using\|export\|import\|include\|begin\|let\)\>/d_' -c '%g/.*/exe "norm >>"' -c ':%p' -c ':q!' | tail -n +2; done | less
But after making the switch, it’s much easier to step through the codebase:
Of course, I could have written some documentation about the file dependencies, or an IDE to keep track of these, and would use fewer lines of code sticking to the include(), but I do like how this system keeps explicit track of dependencies like this.
I should mention that I do still use include() for smaller scripts where I only want to split up a few hundred line codebase into component files, but I think when the codebase grows to a certain size and complexity, it is really helpful to have this hierarchical approach available.
I haven’t surveyed the ecosystem, but anecdotally I have never included the same file in two modules or seen it done after working on a lot of packages. I don’t know if this ever actually happens?
I guess you can include the same file in different modules, in which case the code in it may have a different meaning. While I’m not sure why you would do that, you would also be fully aware that you had done it for some reason. And even your example is highly unusual. Because we have multiple dispatch, subcalc would nearly always be methods of the same functions dispatching on different types.
So, I can agree that something odd is vaguely possible to do in very unusual code. The question is then how much harm does this cause. If it doesn’t cause any, does it matter? Other languages are riddled with worse foot guns lying around that people just work around, this is more like a foot water-pistol hidden in a box somewhere, that only python people seem to look in.
Also, it is kind of possible to use files as modules by tweaking the LOAD_PATH:
shell> ls
A.jl B.jl
shell> cat A.jl
module A
import B
print_a() = print("A")
print_b() = B.print_b()
end
shell> cat B.jl
module B
print_b() = print("B")
end
julia> push!(LOAD_PATH, ".")
4-element Vector{String}:
"@"
"@v#.#"
"@stdlib"
"."
julia> using A
[ Info: Precompiling A [top-level]
julia> A.print_b()
B
This doesn’t really work from packages where you can’t mutate the LOAD_PATH but it could be something to think about if something along those lines would be useful.
Sure, if I had done it, I would know, but how would you know if I had done it? Try to put yourself in the shoes of someone who has to pick up and maintain a legacy codebase. I know Julia isn’t old enough to have much legacy code yet, but it’ll get there. Most Julia devs are doing greenfield work right now, which seems to be reflected in prioritizing features that make it easy to write, but possibly harder to maintain. I always find myself on brownfield projects where I wish the language didn’t have quite so many attractive nuisances for science grad students who taught themselves just enough programming to be dangerous.This experience colors the way I view the tradeoff between power and safety.
To give a concrete example, I once worked with an IDL code where a student had written a single run-on procedure of several thousand lines. Then they broke it up into separate files by using a single common block as a ‘God object’ to schlepp all the variables in the namespace from one file to another. When I see a long list of include statements in a package, I start having flashbacks to that nightmare.
That’s what I was trying to reason out. As I said above, though, it looks like Julia has safeguards against including the same thing twice into the same namespace. Python has no warning mechanism if you use exec to overwrite definitions, which I think goes a long way toward explaining why it’s considered dangerous by folks with Python backgrounds.
That’s a good point about multiple dispatch. It reinforces my point, which is that one rarely needs the full flexibility that include offers.
In brief: the current approach to organising several files worth of code is something like:
# master.jl
include("utils.jl")
include("A.jl")
include("B.jl")
# utils.jl
common = 1
# A.jl
foo = common + 2
# B.jl
bar = common + 3
in which A.jl and B.jl implicitly depend upon utils.jl.
Being implicit in this way introduces various issues. For example it’s hard to tell, whilst reading A.jl, what common is, where it came from, what it does. (Much of this discussion has focussed around the task of reading or maintaining existing code.)
The hope is to switch this out for a syntax looking something like:
# master.jl
import "A.jl": foo
import "B.jl": bar
# utils.jl
common = 1
# A.jl
import "utils.jl": common
foo = common + 2
# B.jl
import "utils.jl": common
bar = common + 3
to make the dependency structure between files explicit. The main motivation being to try and address the drawbacks previously stated.
includeing the same code twice is bad, but so is typing the same code twice, and there’s nothing to stop that. Somewhere there must be a line where you’re allowed to be a bad programmer.
.