Loading from parent scope in package

lancejnelson · April 2, 2024, 9:18pm

Originally, none of these modules were packages… just modules. That’s where I originally discovered this problem. I’m right in the middle of making them into packages and due to the problem I had with pure modules, I was trying to design my package structure to avoid the problem. So you’re saying that I should be able to do using MatTypes in various modules/packages and it will be the same types everywhere?

mkitti · April 2, 2024, 9:35pm

I’m confused what the parent module of MatML is.

Since MatTypes is an independent package, just do the following in every module you want to use these types. There is no need to reference the parent module.

using MatTypes: Crystal, DataSet

A slightly different style is to the following.

using MatTypes as MT

foo(x::MT.Crystal) = x

mkitti · April 2, 2024, 9:38pm

Correct.

lancejnelson · April 3, 2024, 2:46am

I’m almost home on this issue. I’ve created four packages that all interact with each other and I think I’ve correctly specified all of the dependencies. The packages are:

But when I try and add one of these packages:

Using Pkg
Pkg.add(url = "https://github.com/lancejnelson/CrystalUtils.jl.git")

It claims to have trouble installing VASP. It only works if I install them in a certain order which doesn’t seem right because each of them have the right dependency information.

mkitti · April 3, 2024, 3:46am

Note that you can specify multiple packages to Pkg.add at once. That should eliminate your order issues.

julia> using Pkg

julia> Pkg.activate(; temp=true)
  Activating new project at `/tmp/jl_2AyYu8`

julia> Pkg.add(
           map([                                                              
               "CrystalUtils",
               "MatTypes",
               "VASP",
               "MatML"
           ]) do pkgname
               PackageSpec(url = joinpath(                                              
                   "https://github.com",
                   "lancejnelson",
                   pkgname * ".jl"
               ))
           end
       )

You may want to consider publishing a meta-package with a Manifest.toml that includes all your packages. The Manifest.toml includes the URLs of all of your packages. If someone installs the meta-package, the Manifest.toml will help the resolver discover the URLs of all of them.

julia> Base.active_project() |>
           x->replace(x,"Project" => "Manifest") |>
           x->read(x, String) |>
           println

[[deps.CrystalUtils]]
deps = ["LinearAlgebra", "MatTypes", "StaticArrays", "StatsBase", "VASP"]
git-tree-sha1 = "786a8c27a509440461bc1d1b3ba1a0a0c3291a70"
repo-rev = "main"
repo-url = "https://github.com/lancejnelson/CrystalUtils.jl"
uuid = "03766455-12b7-4a6c-9499-86e52762f3ab"
version = "0.1.0"

[[deps.MatML]]
deps = ["CrystalUtils", "Distributions", "LinearAlgebra", "MatTypes", "StaticArrays"]
git-tree-sha1 = "efb4fd2f705998dec1f6113270d681cd5bc64f56"
repo-rev = "main"
repo-url = "https://github.com/lancejnelson/MatML.jl"
uuid = "272dadef-06aa-45c4-9a1d-b36e12ce8ea5"
version = "0.1.0"

[[deps.MatTypes]]
deps = ["LinearAlgebra", "StaticArrays"]
git-tree-sha1 = "da38da5a0eb2d01a3fb17b0cf1566da52194d8e6"
repo-rev = "main"
repo-url = "https://github.com/lancejnelson/MatTypes.jl"
uuid = "b127de0a-484a-4dc8-ac0a-1765876f1d62"
version = "0.1.0"

[[deps.VASP]]
deps = ["DelimitedFiles", "LinearAlgebra", "MatTypes", "Printf", "StaticArrays"]
git-tree-sha1 = "668dbb354e64f606fc3d361a848c21ce8287ac1f"
repo-rev = "main"
repo-url = "https://github.com/lancejnelson/VASP.jl"
uuid = "3d64381f-12ea-4605-8d7a-6644de55dbc1"
version = "0.1.0"

Something else I wanted to mention is that it is possible to have multiple packages in the same git repository.

https://pkgdocs.julialang.org/v1/managing-packages/#Adding-a-package-in-a-subdirectory-of-a-repository

lancejnelson · April 3, 2024, 2:47pm

I hope you’ll forgive me for saying this is a bit of a letdown. I thought the main advantage to packages is that all of the dependencies are handled automatically. Come to find out… if the packages are unregistered, you have to manage those dependencies manually.

lancejnelson · April 3, 2024, 2:49pm

This was really helpful actually. Thank you. I was “including” the module files in the main file and then using them (with a dot) and the result was different types for different modules.

mkitti · April 3, 2024, 3:09pm

Try this:

Clone this repository:

Then activate the environment contained within.

using LibGit2
LibGit2.clone("https://github.com/mkitti/DemoForLanceJNelson.jl","DemoForLanceJNelson.jl")

using Pkg
Pkg.activate("DemoForLanceJNelson.jl")
Pkg.instantiate()

using VASP

Surprisingly, the below also seems to work.

using LibGit2
LibGit2.clone("https://github.com/mkitti/DemoForLanceJNelson.jl","DemoForLanceJNelson.jl")

using Pkg
Pkg.activate("DemoForLanceJNelson.jl")
using DemoForLanceJNelson.jl

If you want to run this as a new user, set the environment variable JULIA_DEPOT_PATH.

$ git clone https://github.com/mkitti/DemoForLanceJNelson.jl
$ JULIA_DEPOT_PATH=`mktemp -d` julia -e '
using Pkg
Pkg.activate("DemoForLanceJNelson.jl")
Pkg.instantiate()

using VASP
'

lancejnelson · April 3, 2024, 3:17pm

Thank you. I understand your workaround. Now, I’m left pondering which workflow will be less complicated, packages or just modules, considering that students will be collaborating with me on this project. Thanks for all your help.

lancejnelson · April 3, 2024, 3:41pm

This last solution will download all of the packages into JULIA_DEPOT_PATH so that they can be modified and then pushed back to github?

This statement doesn’t seem to work for me:

JULIA_DEPOT_PATH=`mktemp -d` julia -e '

mkitti · April 3, 2024, 3:47pm

What I’m trying to do here is simulate what a new student would experience if they just downloaded Julia.

ffevotte · April 3, 2024, 9:04pm

Yes, working with an ecosystem of unregistered packages is indeed less straightforward than with registered packages.

My personal advice would be to start with only one package. It might not even be necessary to introduce any further structure in it (by which I mean that splitting the sources in several aptly-named files or even subfolders might be enough to organize things, without having to introduce submodules).

It could look like:

MainPackage/
+ src/
  + MainPackage.jl
  + analyze.jl
  + types.jl
  + vasp.jl
+ Project.toml

With the following contents:

# src/MainPackage.jl

module MainPackage
  # 3rd party dependencies (declared in `Project.toml`)
  using LinearAlgebra
  using StaticArrays

  # the API you want to export for final users
  export build_crystal, sample

  # all package sources organized in different files (possibly directories)
  # as you see fit
  include("types.jl")
  include("vasp.jl")
  include("analyze.jl")
end

# src/types.jl

struct Crystal
  # ...
end

struct DataSet
  # ...
end

# src/vasp.jl

function build_crystal(file::String)
  # ...
  return Crystal(...)
end

Now if you really want your code to be structured in isolated namespaces, my personal opinion would still be to structure it (at least at first) in only one package, with a main module and several submodules.

lancejnelson · April 3, 2024, 10:24pm

Thank you. Great advice and thoughtful insight. Side note: My student doesn’t speak Julia yet so he decided he would replicate my code in Python. He was stunned at how slow Python was in comparison to my Julia version.

mnemnion · April 4, 2024, 8:14pm

Since you’re doing this for a classroom, you may consider setting up a local registry, this is more work for you up-front but makes things very simple for your students.

mkitti · April 4, 2024, 11:07pm

Part of the original prompt was this would be a “pretty big library”, which is why we suggested the multiple package route.

I’m unclear if this project will be so that it requires a separate registry, but that is an option. One example for an academic group is @tim.holy 's lab’s registry: