I’m afraid that is most likely not true. At the moment users are happily loading functionality from packages and modules whose internals they had never seen before. Of course, it is possible that what you propose would be enabled only for package developers, which I believe would be safe(r) than opening it up to any and all.
Fortunately, that’s not what’s being discussed.
An external package can (and should) still be imported in the way that you’re familiar with. The use of from is specifically for relative imports within your own package.
if you’re skeptical then I’d suggest familiarising yourself with Python’s tried-and-tested import system. In a case of convergent evolution rather than explicit inspiration, the two systems are very similar.
2 Likes
I think flat out preventing everyone from importing files on a whole in order to stop some from possibly doing things wrong is getting dangerously close to starving yourself in order to prevent choking hazards. If anything it probably encouraged an overuse of include() for those that didn’t know how to import from files properly in the first place. For those that wanted to know how to do things properly the only thing it does seem to be a whole lot of hassle or forcing them to adopt a very flat project structure. As I outlined above, a flat project structure doesn’t always make sense. In fact, one can argue that since problems in general are solved by logically breaking them down into smaller and more specific chunks, a flat project structure rarely makes sense except for fairly small projects.
I don’t think that anyone is trying to force you into anything 
The opinions might sound hard but in the end people care about the succes of your project.
The main benefit, I think, of a flatter structure is that you can more easily re-use functions. In OOP, your methods are linked to your objects. So, those can be placed together in a deeply nested file. In Julia, those methods can be much more generic, so you can more often move them further up since you can use them for more objects (types).
4 Likes
Nothing wrong with a deeper structure of the project. My own projects are often based on modules and submodules. I’m just saying that organization based on files may be fragile. Python’s structure is also based on modules, even though superficially it might look like it is based on files.
What do you mean by fragile?
Well, it seems to me that files are more in flux than modules. Let us say I want to use using Mod1: foo. No matter in which file I placed this function, it will be loaded correctly. However, if I refer to a specific file in which this function supposedly resides, whenever I move the function into a different file, I have to visit all the places that load this function to change the reference.
3 Likes
This I do agree (and it’s one of the criticisms I raised against Python’s filesystem based module in my first post). However doing this has its benefits, namely that it’s very clear where the code resides in and therefore very easy for the user to look up the actual implementations. As it is Julia’s module import only shows the module name. Combined with the lack of proper code tracking by ANY IDE it effectively renders nested structure useless for prototyping purpose. This is why I said this system is forcing us to adopt a flat structure - because while it is possible to nest it is not practical.
Also, I’m not sure if I agree a flat structure is inherently a better fit for multiple dispatch/data-only OO Julia uses. The reason is…well, why? Here’s a toy example I cooked up last night to test nesting: https://github.com/fengkehh/julia_nesting_test
It’s meant to be convoluted and ridiculous because I wanted to break things by nesting. You can try running main_test.jl and modify things in all the modules to your own liking. AFAIK I could arbitrarily import type defs and functions from any level to any level regardless where the files reside and everything still works as expected. If it wasn’t for the lack of proper code inspection support I would happily take that as a good enough workaround and use it in commercial development already.
2 Likes
Reading this thread a year and a half later, it seems like (relative) imports are major pain point of Julia for package authors? I’m comparing to something like Rust, where you just write use path::to::module wherever it’s needed and boom it’s loaded, no need to worry about multiple inclusions conflicting or struggling with your IDE not understanding your code because all its imports are missing because they’re imported only in the “main” file, where the current file is included, but not in the current file itself.
I think Julia needs some sort of officially supported “import once” functionality, where, say @include "path/to/file.jl" or @include "path/to/file.jl" ModuleFoo ModuleBar that includes the file at the top level if it hasn’t already been, and if using the second form brings the specified modules into scope.
It’s just too painful as a package author to deal with imports based on C’s #include directive without even having the static guarantees of C’ compiler to help out. Is this something that could be slated for Julia 2?
2 Likes
I think you are misunderstanding how imports work.
1 Like
I understand how they work. The point is that how they work is incredibly hard to to actually be productive with. There’s a reason all languages abandoned the C-style copy-paste #include directive ages ago, which is that it’s really really hard to work with.
If you look at most widely used Julia packages, they break their source tree into several files. In most cases, the non-“main” files don’t include/using their dependencies because those dependencies are brought into scope in the “main” file. Trying to edit a file none of whose dependencies are in scope in the file itself is a major headache. Similarly, trying to chase down the origins of exports in the “main” file is a Herculean effort. You should never have to ctrl-f your entire workspace for the origin of a symbol that’s used in the current file – that’s a sign of bad design.
2 Likes
This is actually really painful when you try to read others’ packages. Sometimes it takes ridiculously long time to find where does a method (or any term) comes from. Is it comes from an external package imported in the main file, or is it defined else where in the current package?
2 Likes
I don’t think you do. Imports are not #includes.
Correct. includes are #includes. Therein lies the problem. in order to import a module, it has to first be in scope. This means it either has to be a package declared as a dependency in your Project.toml — in which case, great, easy peasy, no problems there — or it has to be brought into scope by includeing its source file. If the module is in your own package, and not a dependency, then you have to include it somewhere. Correct me if I’m wrong but there is no third way to import a module.
I have no problem with imports — I think they’re a fine way of making an in-scope module and its items accessible. My issue is with how you get that module into scope in the first place. For that, you need include. And that’s where things get complicated.
1 Like
To make this point concrete, I’ll give a real-life example. Here’s a file in Makie: Makie.jl/display.jl at master · JuliaPlots/Makie.jl · GitHub . Where do the following symbols come from?
@ffmpeg_envget_sceneFileIO
Answer: they’re made available by importing the relevant stuff into Makie.jl/Makie.jl at master · JuliaPlots/Makie.jl · GitHub , includeing display.jl in that file, and then crossing their fingers that everything was wired up correctly across file boundaries.
Package authors shouldn’t have to keep track of the imports in one file in order to reason about a second file, especially when the second file doesn’t tell you which file is the “first file”! If you’re editing display.jl, how are you supposed to find out the myriad places where get_scene is defined? Are you really expected to go through all of the includes in Makie.jl (the file) and check for the presence of that function definition?
I think this qualifies as Action at a distance (computer programming) - Wikipedia , which is universally recognized as a bad quality of a program.
4 Likes
This is a consequence of multiple dispatch. If you see export foo, it means that you are exporting the variable foo. If foo is the name of a function, then you are exporting a generic function. It’s not possible to export individual methods of functions, because they all have the same name. There could be a different foo method defined in every file in your package, so you have to “ctrl-f” to find all the different method definitions. That’s just the way it is—I don’t see a way around that. 
Generic functions are actually global objects, because they are defined by a global method table. So trying to define dependencies with Python-style from pkg import foo doesn’t really work.
Rather than resurrecting this old thread on a contentious topic, I recommend starting a new thread with concrete suggestions for improvement.
9 Likes
This pattern of separating code into different files that are then patched up with include seems to be common in the Julia package ecosystem, and I for one agree about it being harmful and ugly.
I think the situation would be better if includeing arbitrary code was disallowed, but includeing modules specifically was allowed.
But even that seems like only a partial solution.
3 Likes
I strongly disagree. IMO doing this is just bad design.
Adding methods to a function defined somewhere else is mutating global state, and this is a Bad Thing.
If I could, I would have this be prohibited by the compiler. And likewise prohibit type piracy, which is of similar character. (e.g. Rust prohibits the analogous “trait piracy”)
It is completely possible to use Python-style from pkg import foo, as the success of FromFile has demonstrated. (Evidence: 1.1k new users in the past 30 days; for reference Pluto has had 5k new users in the past 30 days.)
To those bitten by this issue, I would recommend using FromFile for your work. Or just using a tech stack other than Julia. (Personally I’ve now made the move to JAX, for this reason and others.)
3 Likes
Would you consider it equally poor behavior to add such a method when one of the types is owned by the module in which that method is defined?
I definitely agree that having the module own the type is necessary. In most situations I think that additionally subtyping an abstract type would really be best practice.
That is, like the following example.
(Using FromFile rather than modules since that’s my preference, but mentally substitute the usual include/import boilerplate if you prefer.)
# foo.jl
abstract type AbstractFoo end
function accepts_foo(x::AbstractFoo)
error("Must define `accepts_foo` for type `$(typeof(x)) <: AbstractFoo`")
end
# foobar.jl
using FromFile: @from
@from "foo.jl" import AbstractFoo, accepts_foo
struct Foobar <: AbstractFoo
...
end
function accepts_foo(::Foobar)
...
end
This gives at least some discoverability to track what’s going on.
This analogous to overriding methods of a class in typical single-dispatch OO; just extending it out to multiple dispatch in the obvious way.
Don’t forget that proper importing of symbols helps with a lot more than just this. Even if you don’t know the method you at least know which function is being called. Code exists in smaller namespaces so it’s harder to introduce various classes of bugs. Topologically sorting dependencies is no longer a burden placed upon the developer. And FromFile, at least, additionally handles deduplication (no include-ing a module twice). Etc. etc. you get the idea.
4 Likes