Error using package in local environment

mmv · December 17, 2022, 2:12pm

Hi!

I am using the StringViews package which works fine in the normal REPL, but is not found when I get into the test environment, even though I included a statement using StringViews in the runtests.jl file, as well as an add StringViews statement before launching the test function:

michel@MicMac2:~$ JMtk15
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.2 (2022-09-29)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using JMtk15

julia> using StringViews

julia> juldate = 2.4523965833333335e6
2.4523965833333335e6

julia> datetime = zeros(Cuchar, 28)
28-element Vector{UInt8}:
 0x00
 0x00
    ⋮
 0x00
 0x00

julia> status = MtkJulianToDateTime(juldate, datetime)
0

julia> datetime = StringView(datetime)
"2002-05-02T02:00:00Z\0\0\0\0\0\0\0\0"

julia> datetime = rstrip(datetime, '\0')
"2002-05-02T02:00:00Z"

(JMtk15) pkg> activate .
  Activating project at `~/Codes/Julia/JMtk15`

(JMtk15) pkg> add StringViews
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `~/Codes/Julia/JMtk15/Project.toml`
  No Changes to `~/Codes/Julia/JMtk15/Manifest.toml`

(JMtk15) pkg> add Test
   Resolving package versions...
  No Changes to `~/Codes/Julia/JMtk15/Project.toml`
  No Changes to `~/Codes/Julia/JMtk15/Manifest.toml`

(JMtk15) pkg> test JMtk15
     Testing JMtk15
      Status `/private/var/folders/dx/hlnzrrln4yz8grp1v4385qw00000gn/T/jl_US8uXW/Project.toml`
  [600a0e46] JMtk15 v0.1.0 `~/Codes/Julia/JMtk15`
  [8dfed614] Test `@stdlib/Test`
      Status `/private/var/folders/dx/hlnzrrln4yz8grp1v4385qw00000gn/T/jl_US8uXW/Manifest.toml`
  [600a0e46] JMtk15 v0.1.0 `~/Codes/Julia/JMtk15`
  [90137ffa] StaticArrays v1.5.11
  [1e83bf80] StaticArraysCore v1.4.0
  [354b36f9] StringViews v1.0.3
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [8dfed614] Test `@stdlib/Test`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [4536629a] OpenBLAS_jll v0.3.20+0 `@stdlib/OpenBLAS_jll`
  [8e850b90] libblastrampoline_jll v5.1.1+0 `@stdlib/libblastrampoline_jll`
     Testing Running tests...
ERROR: LoadError: ArgumentError: Package StringViews not found in current path.
- Run `import Pkg; Pkg.add("StringViews")` to install the StringViews package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1163 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [5] top-level scope
   @ none:6
in expression starting at /Users/michel/Codes/Julia/JMtk15/test/runtests.jl:2
ERROR: Package JMtk15 errored during testing

(JMtk15) pkg>

BTW, it is not clear whether the recommendation following the error message to Run import Pkg; Pkg.add("StringViews") should be issued in the REPL or in the (JMtk15) pkg environment…

I suspect this problem may have to do with the general configuration of my computing environment, but I don’t know where to start, because I have not encountered a similar problem with other packages. In fact I have the following specific questions:

Are the packages I am using stored locally on my computer or only accessible in real-time through Internet?
If they are saved locally, where are they? I’ve seen the subdirectory .julia/packages, and it does include a subdirectory StringViews, but those subdirectories appear to be very small in size…
Why are my explicit statements to using and to add StringViews not effective?
If the solution involves JULIA_LOAD_PATH or JULIA_DEPOT_PATH variables, then what’s the difference between those two, and where should those statements be placed so that they are always active (I’m using a MacBook Pro)?

Thanks a lot in advance for your help in this matter.

skleinbo · December 17, 2022, 2:24pm

You will need to add StringViews as a test-specific dependency to the Project.toml. Or create a new environment in the test/ folder.

mmv · December 17, 2022, 3:34pm

Hi @skleinbo,

Thanks for your suggestion, but that does not work, or maybe something else is also missing. First of all, the Project.toml file in my test subdirectory contained only two lines:

[deps]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

although the Pkg documentation at 5. Creating Packages · Pkg.jl does not mention anything looking like [deps]. In any case, I added two more lines to that file which now reads as follows:

[extras]
StringViews = "354b36f9-a18e-4713-926e-db85100087ba"

[deps]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

Following that change, I restarted from scratch and got the same error message:

julia> using StringViews

(JMtk15) pkg> activate .
  Activating project at `~/Codes/Julia/JMtk15`

(JMtk15) pkg> add StringViews
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `~/Codes/Julia/JMtk15/Project.toml`
  No Changes to `~/Codes/Julia/JMtk15/Manifest.toml`

(JMtk15) pkg> add Test
   Resolving package versions...
  No Changes to `~/Codes/Julia/JMtk15/Project.toml`
  No Changes to `~/Codes/Julia/JMtk15/Manifest.toml`

(JMtk15) pkg> test JMtk15
     Testing JMtk15
      Status `/private/var/folders/dx/hlnzrrln4yz8grp1v4385qw00000gn/T/jl_F6TI2w/Project.toml`
  [600a0e46] JMtk15 v0.1.0 `~/Codes/Julia/JMtk15`
  [8dfed614] Test `@stdlib/Test`
      Status `/private/var/folders/dx/hlnzrrln4yz8grp1v4385qw00000gn/T/jl_F6TI2w/Manifest.toml`
  [600a0e46] JMtk15 v0.1.0 `~/Codes/Julia/JMtk15`
  [90137ffa] StaticArrays v1.5.11
  [1e83bf80] StaticArraysCore v1.4.0
  [354b36f9] StringViews v1.0.3
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [8dfed614] Test `@stdlib/Test`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [4536629a] OpenBLAS_jll v0.3.20+0 `@stdlib/OpenBLAS_jll`
  [8e850b90] libblastrampoline_jll v5.1.1+0 `@stdlib/libblastrampoline_jll`
     Testing Running tests...
ERROR: LoadError: ArgumentError: Package StringViews not found in current path.
- Run `import Pkg; Pkg.add("StringViews")` to install the StringViews package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1163 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [5] top-level scope
   @ none:6
in expression starting at /Users/michel/Codes/Julia/JMtk15/test/runtests.jl:2
ERROR: Package JMtk15 errored during testing

(JMtk15) pkg>

So, what else should I try?

ufechner7 · December 17, 2022, 4:28pm

Add a line:

[targets]
test = ["Test", "StringViews"]

Furthermore, have a look at GitHub - JuliaTesting/TestEnv.jl: Activate your test enviroment, so you can use your test dependencies in the REPL .

This is not required, but makes it easier to activate the test environment when needed.

skleinbo · December 17, 2022, 4:33pm

I would set up testing as follows.

Remove test/Project.toml
Add StringViews to your package’s environment. You’ve already done that.

Open the package’s Project.toml and add

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
    
[targets]
test = ["Test", "StringViews"]

mmv · December 17, 2022, 10:13pm

Hi @ufechner7 and @skleinbo,

Thanks for your inputs. I’ve now been able to carry out all tests, though I’ve also had to add the StringViews package uuid in the [extras] section of the main Project.toml file, even though it already was mentioned in the [deps] section. Is this normal?

While I can understand that the testing environment may require additional tools, I would have thought that packages required by the main package being developed would automatically be included in the context of testing it…

The documentation I found for Project.toml files (10. Project.toml and Manifest.toml · Pkg.jl) mentions the [deps] and the [compat] sections, but does not say anything about [extras] and [targets]. So what are the differences between those sections and why do I need to repeat dependencies already mentioned in [deps]?

Thanks again for your support!

rikh · December 17, 2022, 10:45pm

It sort of is, the symbol is available of your top package. Say that your main package is called A and uses B. Then inside your tests, you can load A.B (which is generally not recommended but good to know and sometimes useful)

mmv · December 17, 2022, 10:59pm

Hi @rikh,

Thanks for your input. This is interesting but somewhat contrived. In any case, don’t you think that if package A uses package B to achieve its purpose, then the functions of package B should remain available to A, especially when testing, and without needing to specify this dependency again?

rikh · December 17, 2022, 11:12pm

I actually prefer not to think to much about those things no because at this point of the language maturity it’s probably not gonna change anymore.

mmv · December 17, 2022, 11:27pm

Interesting comment: I’m new to Julia but I wouldn’t have thought that it was already sclerotic at age 10! On the other hand, I see multiple changes and updates every few months with new releases…

Krastanov · December 17, 2022, 11:55pm

My personal preference would be “definitely not” because of the following: If I install package A which depends on some arbitrary package B that I have never heard of, I would expect using A to work fine, but I would prefer if using B to not work, because I have never explicitly told Julia to make B available to me. It feels somehow cleaner and less confusing for dependencies like B to be hidden from the user that only asked for A. The test environment is supposed to “look” the way a user environment looks, so the same thing stands. Of course, this is just my aesthetic preference, which is why I use only the word “feel”, not “should be”, but hopefully this is at least a reasonable rationalization for the current design.

Krastanov · December 18, 2022, 12:00am

Since version 1.0, there is now a commitment to backward compatibility, and this particular behavior can not change without breaking things. However, on the github issue tracker there is a large list under the julia 2 tag, for things that the core developers feel could be done in a neater fashion. There will not be Julia 2.0 anytime soon, but that list would give you a sense of “what would we like to polish, but can not due to backward compatibility constraints”.

mmv · December 18, 2022, 12:14am

Hi @Krastanov,

Thanks for your interesting two comments: In my case, package A really needs package B, in the sense that if B is not available or accessible, then A will fail too. So B is really an essential dependency for A, whether I use A on its own, or whether I am testing A.

I am not familiar with the inner workings of Julia, but I would have thought that the Project.toml file of project A, which does include the required packages like B, could be parsed by whatever process is handling the testing. Or perhaps it could look at that file if a package has not been found in the ‘usual’ places… That would not break any previous programming habit.

BTW, should there be only one Project.toml file in the main directory of a project? Somehow, I saw one popping up in the test subdirectory, though I don’t know why or how that came about… And where can I find a detailed documentation about what to include in a Project.toml file (earlier question)?

Thanks again for your contribution.

Krastanov · December 18, 2022, 12:28am

I think the most typical way to deal with such very closely related package in Julia is to “re-export” symbols. For instance, imagine something like the Makie plotting library, where Makie defines a few convenient interfaces and base classes, but GLMakie has all the functionality to make OpenGL plots and CairoMakie has all the functionality to make static png plots. The latter two depend on the first one. Makie is the one that defines something like plot so naively the user might need to write:

using Makie # so that `plot` is available
using GLMakie # so that `plot` creates interactive plots

However, if GLMakie already has access to Makie, then it can also export Makie symbols. It can be done as follows:

module GLMakie # this is the GLMakie library code
    import Makie: plot
    export plot
end

Now the user can simply write using GLMakie and the function plot defined inside of Makie is available. If you strongly believe that the two packages you are developing should re-export reused symbols, this is probably a good way to structure your code. There is the utility package Reexport.jl which makes that even more trivial.

Edit: don’t take the Makie example too literally. I do not know how Makie is structured internally.

Krastanov · December 18, 2022, 12:38am

This is something that has bothered me too. It seems there are two perfectly reasonable ways to structure your test environment:

Have your testing environment defined in the main project.toml file:

MyLibrary
├ Project.toml # this file contains the test environment definition
├ src
│ └ MyLibrary.jl
└ test
   └ runtests.jl

The project file contains extra test information thanks to this section:

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
    
[targets]
test = ["Test", "StringViews"]

examples: https://github.com/qojulia/QuantumOptics.jl/blob/master/Project.toml

Or have your test environment as a sub-project

MyLibrary
├ Project.toml # this file does not say anything about tests
├ src
│ └ MyLibrary.jl
└ test
   ├ runtests.jl
   └ Project.toml # this file defines an independent test environment

The two project files do not contain any “extra” fields.

Examples:

the main file https://github.com/Krastanov/QuantumClifford.jl/blob/master/Project.toml
the test file https://github.com/Krastanov/QuantumClifford.jl/blob/master/test/Project.toml

By habit and some vague sense of “separation of concerns” I happen to use the second structure. I see some recent discussion of this question here Project.toml in test directory vs extras

I too would like some more authoritative answer in the style of “Use method X because of reason Y”, but I do not have it yet. Perusing the many links in the above discussion would be an interesting archeology expedition through the recent developments in “Julia dev culture”.

Lastly, here is the official documentation that explicitly says there are two ways: https://pkgdocs.julialang.org/dev/creating-packages/#Test-specific-dependencies

I believe your comment about improving the parsing of the Project.toml files during testing is reasonable (including that it would not be breaking compatibility guarantees). I suspect there is no sense of urgency to improve that, while there are a lot of other imperfections of Julia that the core devs consider higher priority. If you have the mental energy to volunteer writing a detailed enhancement suggestion on the bug tracker (and even more so, modify the package import logic and submit a pull request), people might consider it. But that is a lot of volunteer labor with uncertain outcome.

mmv · December 18, 2022, 8:58pm

Hi @Krastanov,

Thanks a lot for this very clear explanation, useful suggestions and pointers to other resources.

I particularly appreciated your description of the two approaches to managing package requirements, with either one or two Project.toml files. The second, with a separate file for testing, makes particular sense if the testing process requires additional packages that are not necessary when using the main package itself.

In my specific case, I do not expect to need any other resource for the testing, as I am only trying to provide Julia wrapper functions to an existing C shared library: all I need to do is verify that the Julia versions deliver exactly the same output as the original C function (but I’m only at the start of that project…) I also like the idea of maintaining a single Project.toml file, so I’ll start on that path, knowing that I can easily switch to the other if future testing requirements involve packages not required to generate the wrappers.

In short, it’s nice to have both options, and I might use them both in different contexts. Lastly, being new to Julia I don’t feel qualified yet to suggest improvements to the language, but I’ll keep that in mind.

Thanks again for clarifying this question.