Using a locally developed package

New to Julia
1

Hi!

I developed a first package JMtk15 and now would like to use those functions in a second package JMTools. That first package appears to be loading correctly because

  • the command using JMtk15 does not generate any error messages,
  • the JMtk15 package appears in the Project.toml of JMTools as expected, and
  • a const definition in JMtk15 is correctly retrieved:
julia> using JMTools

julia> using JMtk15

julia> JMtk15_src
"/Users/michel/Projects/MISR/MISR_Toolkit.1.5/JMtk15/src/"

However, when I call one of the JMtk15 functions, I get error messages:

julia> misr_path = 168
168

julia> misr_orbit = 68050
68050

julia> bool = is_from_misr_path(misr_path, misr_orbit)
ERROR: UndefVarError: jMtkOrbitToPath not defined
Stacktrace:
 [1] is_from_misr_path(misr_path::Int64, misr_orbit::Int64)
   @ JMTools ~/Projects/MISR/MISR_Tools/JMTools/src/is_from_misr_path.jl:57
 [2] top-level scope
   @ REPL[6]:1

julia> bool = is_from_misr_path(misr_path, misr_orbit)
ERROR: UndefVarError: JMtk15 not defined
Stacktrace:
 [1] is_from_misr_path(misr_path::Int64, misr_orbit::Int64)
   @ JMTools ~/Projects/MISR/MISR_Tools/JMTools/src/is_from_misr_path.jl:57
 [2] top-level scope
   @ REPL[6]:1

where the first error message was generated by the code line 57

true_path = jMtkOrbitToPath(misr_orbit)

while the second error message was generated by the code line 57

true_path = JMtk15.jMtkOrbitToPath(misr_orbit)

That function is explicitly exported in JMtk15.jl.

So it looks like I am either missing a step or an instruction to make the functions of the local package JMtk15 actually usable in the new local project JMTools.

However, I also found out (1) that although the source of JMtk15 is located in a regular directory /Users/michel/Projects/MISR/MISR_Toolkit.1.5/JMtk15, Julia appears to load packages from a different location:

julia> pkgdir(JMtk15)
"/Users/michel/.julia/packages/JMtk15/hCUhn"

and (2) that the directory .julia/packages/JMTk15 contains two oddly named subdirectories, each containing what looks like copies of the source code of JMtk15:

MicMac3 ~/.julia/packages/JMTk15 % ls -al
total 0
drwxr-xr-x@  4 michel  staff   128 May  2 10:27 .
drwxr-xr-x  36 michel  staff  1152 Apr 28 16:59 ..
drwx------@ 11 michel  staff   352 Apr 28 16:59 9eq0g
drwx------@ 11 michel  staff   352 May  2 10:27 hCUhn

What is the role of those copies in .julia/packages/JMTk15? When do they get created and what is their purpose? Do I need to keep multiple copies (here 3: the working directory and two copies in .julia/packages/) of the package source files?

Any help will be greatly appreciated.

2

~/.julia/packages/ is a directory where Pkg stores installed packages: The directories inside .julia/packages/JMTk15 correspond to different versions of your package. They get created when you Pkg.add your package. They should not be identical copies.

To your original issue – could you share your Manifest.toml of JMTools?

Also, in search for where JMtk15_src comes from, I checked the source of JMtk15 on github. Note that it should be possible to replace all the lines

include(JMtk15_src * "include/MisrCache_h.jl")

simply with

include("include/MisrCache_h.jl")

Then you don’t need JMtk15_root, JMtk15_src, …

3

This thread has some related discussion which might be helpful: Add vs dev at a local path

1 Like
4

Hi @barucden,

Thanks for your quick response. I looked at the Manifest.toml, and indeed, it contains these lines:

MicMac3 ~/Projects/MISR/MISR_Tools/JMTools % cat Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.8.5"
manifest_format = "2.0"
project_hash = "f480c51fbf8898cfbe1ab1453f805e0d22be89b7"

...

[[deps.JMtk15]]
deps = ["CEnum", "StaticArrays", "StringViews", "Test"]
path = "/Users/michel/.julia/packages/JMtk15/hCUhn"
uuid = "6c71635d-e68b-418f-880a-c8128ca9dc0a"
version = "0.1.0"

...

[[deps.libblastrampoline_jll]]
deps = ["Artifacts", "Libdl", "OpenBLAS_jll"]
uuid = "8e850b90-86db-534c-a0d3-1478176c7d93"
version = "5.1.1+0"
MicMac3 ~/Projects/MISR/MISR_Tools/JMTools %

So I see that the dependency is specifically referring to the .julia/packages/JMtk15/hCUhn subdirectory…

Hi @GunnarFarneback,

Thanks also for your input. I’ve now

julia> pwd()
"/Users/michel/Projects/MISR/MISR_Tools/JMTools"

(JMTools) pkg> dev "/Users/michel/Projects/MISR/MISR_Toolkit.1.5/JMtk15"
   Resolving package versions...
    Updating `~/Projects/MISR/MISR_Tools/JMTools/Project.toml`
  [6c71635d] ~ JMtk15 v0.1.0 `~/.julia/packages/JMtk15/hCUhn` ⇒ v0.1.0 `~/Projects/MISR/MISR_Toolkit.1.5/JMtk15`
    Updating `~/Projects/MISR/MISR_Tools/JMTools/Manifest.toml`
  [6c71635d] ~ JMtk15 v0.1.0 `~/.julia/packages/JMtk15/hCUhn` ⇒ v0.1.0 `~/Projects/MISR/MISR_Toolkit.1.5/JMtk15`

That, in turn, has updated the Manifest.toml file which now reads:

[[deps.JMtk15]]
deps = ["CEnum", "StaticArrays", "StringViews", "Test"]
path = "/Users/michel/Projects/MISR/MISR_Toolkit.1.5/JMtk15"
uuid = "6c71635d-e68b-418f-880a-c8128ca9dc0a"
version = "0.1.0"

However, I still get the same error message:

julia> bool = is_from_misr_path(misr_path, misr_orbit)
ERROR: UndefVarError: JMtk15 not defined
Stacktrace:
 [1] is_from_misr_path(misr_path::Int64, misr_orbit::Int64)
   @ JMTools ~/Projects/MISR/MISR_Tools/JMTools/src/is_from_misr_path.jl:57
 [2] top-level scope
   @ REPL[10]:1

Is there anything else that needs to be setup?

5

Just to make sure – are you really using JMtk15 inside JMTools and not just the Julia REPL session?

4 Likes
6

@barucden,

Excellent point: I have now added using JMtk15 inside the JMTools.jl file, and lo and behold:

julia> bool = is_from_misr_path(misr_path, misr_orbit)
true

Thanks to both of you for resolving this issue so quickly.

1 Like
7

Hi!

When you call

pkg> add /path/to/my/source/JMTk15

It creates a copy of the package in ~/.julia/packages, and each version is placed in a different subdirectory (those are the five random letters). I’m guessing you have two copies there because you added to different versions of your package some time earlier.

This behavior might seem odd when you add a local package, but it is perfectly reasonable. It is the only right thing when you add a package from the network. And even makes sense locally, because once you add a package to an environment, you expect that environment to remain stable, it should not change just because you make some changes somewhere on the disk.

That being said, it often happens that you develop two or more, dependent packages simultaneously, and you explicitly want changes made in one of them reflected immediately in the other as well. You need to use the dev command instead of add in this case:

pkg> dev /path/to/my/source/JMTk15

This tells Julia to use the package right from the source, instead of copying it in the ~/.julia/packages folder.

4 Likes
8

Thanks a lot, @ HanD, for this clarification: very much appreciated.

1 Like