Confused about package loading

General Usage
1

My understanding is that Julia visits each of the locations defined in LOAD_PATH to load package X until it finds one by the name. So if the default LOAD_PATH is

julia> LOAD_PATH
3-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"

and if you do using PkgA , Julia will look for the current active environment’s project.toml first, and if PkgA is not listed in [deps], then move on to the next location, which is @v#.# (such as environment/v1.8) and again look inside environment/v1.8/project.toml, etc.

As a concrete example,

(@v1.8) pkg> dev Example
   Resolving package versions...
  No Changes to `E:\.julia\environments\v1.8\Project.toml`
  No Changes to `E:\.julia\environments\v1.8\Manifest.toml`

julia> import Pkg; cd(Pkg.devdir()); cd("Example")

(@v1.8) pkg> activate .
  Activating project at `E:\.julia\dev\Example`

(Example) pkg> st
Project Example v0.5.4
Status `E:\.julia\dev\Example\Project.toml` (empty project)

The active env is Example. Now I issue in the REPL, using DataFrames:

julia> using DataFrames

julia> df = DataFrame()
0×0 DataFrame

DataFrames successfully loaded even though DataFrames is missing in the current environment This is because, Julia has discovered DataFrames from the default env v1.8 instead.

Now here is my real question:
I edit Example.jl and import DataFrames within from the module Example:

module Example
using DataFrames

function test()
   df = DataFrame()
end
end

After switching to (Example) pkg> env, Now I issue using Example

julia> using Example
[ Info: Precompiling Example [7876af07-990d-54b4-ab0e-23690620f79a]
ERROR: LoadError: ArgumentError: Package Example does not have DataFrames in its dependencies:

Unlike the first experiment, strangely, Julia failed to locate DataFrames.

Why is this so?

2

What’s the directory structure of your Example? I think you have (inadvertently) structured your code as a package, which have to have all modules they’re using declared as dependencies in Project.toml for dependency resolution in the package manager. This is also used for precompilation.

The stacked environments of other people may not have DataFrames installed after all, so relying on them to have the package available doesn’t work.

1 Like
3

Agreed. I think you are correct here (assuming I understand your setup).

Again, if I understand your setup correctly, this is because:

  • Package Example is using package Dataframes…

And yet:

  • Package Dataframes was not "add"ed to package Example’s project.toml file (dependency list).

Why does Julia treat these cases differently?

Considering the situation from your first example:

  • when using Dataframes is launched from the REPL.

Your current Julia session is clearly using an environment stack that defines Dataframes, so the session should be able to access this package.

On the other hand, in your second example:

  • using Dataframes is called from within package Example.

To Julia, this makes a huge difference. Julia wants to ensure all dependencies are clearly included in each package’s project.toml file.

If I were to guess, I would think this was done to ensure a smooth package installation experience in Julia:

  • By ensuring that all dependencies are present in the project.toml file, the package manager can correctly assume a package will work if all dependencies from project.toml are installed.

…Barring any mistakes in how the version compatibility ranges were specified (and other similar potential issues).

2 Likes
4

@Sukera Thank you for your answer! Example package is a registered package provided by Julia, for educational purpose.

5

@MA_Laforge Thank you for the detailed explanation! I did not know using or import behave differently between when I have them run in REPL or from within package. It really makes sense that the reason for the difference is ensuring all dependencies and thereby helping with smooth package installation experience.

The environment stack (mentioned in Code Loading · The Julia Language) is relevant only when we do things interactively in REPL then? The manual doesn’t seem to state it explicitly.

6

The manual might state this explicitly, but if it does, I haven’t noticed it either.

I suppose this is one of the big problems with communication: How to transmit ideas in an effective fashion? If your explanations are too verbose, finding pertinent information becomes very laboured, and you end up loosing your audience. On the other hand, if your explanations lack sufficient detail, the essence of your message is lost.

If you have a better way to explain how projects/environments operate under Julia, feel free to write something up and submit a PR. I am sure the Julia team will integrate it in their docs if they also find it easier to understand.

I must admit that I too feel that a few key components are missing from the current documentation, but I haven’t yet found a better way to describe these concepts.

I also am very hesitant of submitting my explanation above as I feel it is both too specific, and too verbose. I specifically targeted concepts that you appeared to be conflating - which I consider to be more of a stop-gap solution.

While this approach worked well for your case (I’m glad to hear), I think it would be preferable to present a model that doesn’t accidentally lead people to make these same erroneous associations. Not quite certain how to do that, though. Sorry.

1 Like
7

Thank you for your comment! Yes, I think the code loading part of the manual is well written. It is self-contained and logically consistent. But actually I am not a competent reader so I had to read the chapter over and over gain for two days until I realized the answer to my question was indeed contained in it (and your explanation helped me a lot in the process!)

One person may feel some concept is not explicitly written, while others feel it is apparent. Theorems in math does not tell us every corollary. Many times the reader must prove it from the theorem by himself and many of the corollaries are left as exercises at the chapter end.

In general I feel the classic literature almost always tend to be implicit. :slight_smile: maybe another word for difficult to a incompetent reader like myself. Readers cannot ask Emmanuel Kant to write his concept more explicitly. And maybe that’s the reason we have commentaries or Abc for dummies book, tutorials in the Internal, YouTube videos instead. :slight_smile:

2 Likes