Different Methods of Creating a Pkg?

General Usage
,
1

I’m new to creating pkgs in Julia and have found two methods that work. I’m interested to know if what I’ve written here is accurate and if those with more experience or an opinion on the process would share their input. For this discussion, it’s really about a beginner learning how Julia manages the process, not preparing a licensed pkg for GitHub. No IDE, nor am I using the excellent PkgTemplates.jl method here either.

Both methods begin in Julia REPL - pkg> generate HelloWorld. It is where and how activation is done that differs.

**Method 1: activate . (creates empty project)

  include("HelloWorld/src/HelloWorld.jl")
  • Activation: Sets the current directory as the active environment using activate ..
  • Inclusion: Executes code directly with include from the specified file (HelloWorld/src/HelloWorld.jl).
  • Environment: Dependencies are resolved based on the active environment’s Project.toml file.
  • Portability: May lack portability as it relies on a similar environment.

**Method 2: activate ./HelloWorld

 `import HelloWorld`
  • Activation: Uses activate ./HelloWorld to activate the environment defined by Project.toml in the HelloWorld directory.
  • Inclusion: Accesses exported functionality from the HelloWorld module with import.
  • Environment: Encapsulates the project’s dependencies and environment details within the package.
  • Portability: Preferred for sharing projects, providing a self-contained environment for easy replication.

While Method 1 can get a beginner going and is easy to practice, it is Method 2 that is used in the Julia Documentation (5. Creating Pkgs). Once past the learning stage, it seems that Method 2 is actually required - particularly for sharing projects. It ensures a self-contained environment, simplifying replication for others by encapsulating dependencies and environment details within the package.

Thanks for any input.

1 Like
2

I don’t really understand your post in its entirety, but I’d suggest to forget about the generate command in the Pkg shell (and even the activate command, it’s not essential here) and just use PkgTemplates.jl:

  1. Start/open a Julia REPL
  2. Enter using PkgTemplates
  3. Enter generate(), starting the interactive text-based interface
  4. Use the interactive text-based interface to configure the package skeleton to your desire
  5. Exit the REPL
  6. Inspect the files in the generated directory to learn what is necessary for a Julia package (basically just Project.toml and a .jl file)

Playing with this procedure several times should clear things up.

4 Likes
3

I am not certain I am answering exactly the question you are asking, but you bring up some issues that many students of mine have had, so here is what I usually tell them.

In Julia you use Project.toml files for two slightly different purposes: Creating a self-contained environment in which to work and track dependencies (similar to virtualenv in python); and creating a library that can be used by other libraries or interactively by others.

Creating a self-contained environment

This is somewhat comparable to virtualenvs in python. E.g. you go to a folder, start julia with julia --project=FOLDER_PATH (for instances, --project=. if you want to use the current directory). Or equivalently you start julia without arguments but then go to package mode and do activate FOLDER_PATH. Now, if you add new packages to this environment, you will see a Project.toml files with a list of these dependencies. You can start working in this folder, potentially adding more files, in any subfolder structure you like. You probably would benefit of using the VSCode IDE with the Julia plugin, and setting that folder as your root folder in VSCode.

There will also be a Manifest.toml file that is not meant for you to edit, rather it is exact listing of what versions of various packages were installed (as dependencies of packages you have requested directly that are already listed in Project.toml).

You probably will be using include or Revise.includet for the various script files you have.

This is NOT a library, this is an “environment” in which you can work. Inside of that environment you can install libraries or you can even start creating libraries of your own in it.

For such libraries you would use import or using to import them in your REPL after you have installed them in your environment.

I am being imprecise with the distinction between modules/libraries/packages – there are some caveats about their differences, but they are not too important right now.

Creating a package (a library)

For that you probably want to already be in some environment, and then use generate LibraryName in pkg mode. That will create a new folder, and that folder will have its own internal Project.toml that describes the dependencies of the library you are creating. That folder will also have a certain internal subfolder structure, expected by the package manager (./src folder and maybe ./docs or ./tests).

To have access to whatever is created in that library, your environment will need to have the library added (as any other library). So your environment’s Project.toml file will now have an entry for your library.

You never really need to activate a library the way you are activating an environment.

Example

Consider this folder tree

- my_julia_environment
|-- some_script.jl
|-- Project.toml
|-- Manifest.toml
|-- MyPackage
   |-- Project.toml
   |-- src
      |-- MyPackage.jl
      |-- some_other_internal_library_file.jl

If I want to just run some_script.jl I would do either of these:

  • cd my_julia_environment then julia --project=. then include("some_script.jl")
  • cd my_julia_environment then julia --project=. some_script.jl if I do not need an interactive REPL
  • open my_julia_environment in VSCode, set it as active environment and use the VSCode REPL which has a lot of nice extra features

If I want to make the local version of the library MyPackage available and usable in julia I would do:

  • cd my_julia_environment then julia --project=. then ] dev ./MyPackage – the last pkg command simply tells julia to:
    • record the need for the library MyLibrary in the environment’s Project.toml
    • record the exact version and location of MyLibrary in the environment’s Manifest.toml
    • thanks to all of this now I can simply do import MyPackage

The MyPackage folder does not actually need to be inside of the folder for your environment. Many Julia programmers actually keep such things in ~/.julia/dev (which is also what happens if you do ] dev SomePublicOpensourceLibrary – it gets cloned from github or elsewhere and made available to you)

my_julia_environment/Project.toml by default just contains a list of dependencies in [deps] and no more metadata. You can add extra information, like a [compat] section, listing required versions for the dependencies

The MyPackage folder does not contain scripts, only library files. They are expected to be organized in modules (you can use modules for your scripts, but no need to do that). So it is expected that:

  • MyPackage/src/MyPackage.jl looks like:
    module MyPackage
import Whatever
using SomethingElse
#yada_yada_yada
include("some_other_internal_library_file.jl") # if you want to split your library in multiple files for your convenience
end
  • MyPackage/Project.toml contains more than just a list of required dependencies - it also contains [compat] that specifies what exact versions and other metadata like author and version number for your library. Not all of these are strictly necessary for making a complete package, but if you do not include them, a lot of the tools available to package creators will complain.

You never need to activate a package, only to dev or add (or generate) it. You need to activate an environment and inside of the environment you can add a package.

You can share an environment in a reproducible fashion by preserving its Project.toml and Manifest.toml files. E.g. if you want to share some data processing scripts that generated some plots. No need for packages. You activate an environment with activate SOME_PATH

You make a package when you want something that can be reused by other libraries (a.k.a. packages). There is no Manifest.toml in these cases. You probably would want to register such packages so others can simply install them from within Julia. You do not activate packages, you add them.

TLDR: There is a difference between “an environment in which I execute some scripts and potentially share with others who want to execute the scripts in a reproducible fashion” and “a reusable library that other libraries can depend on”, but they both are structured around a Project.toml file. You do not need anything more than include for environments (but you can use local modules if you want). You have to use modules for structuring a library.

15 Likes
4

For relatively simple packages, which are little more than a module (which is what you get with Pkg.generate in the only file inside the src directory), the most noticeable differences between “Method 1” and “Method 2” may be those that you have already noticed, but there are further advantages of importing a package over just including its main module.

A remarkable additional advantage can be noticed if you are also using Revise.jl: in that case, importing the package instead of including the file that defines the module has the additional advantage of tracking the changes of the package. Let’s say that you change the default contents of HelloWorld.jl:

module HelloWorld

greet() = print("Hello World!")

end # module HelloWorld

To this:

module HelloWorld

greet() = print("Nice to meet you World!")

end # module HelloWorld

The new behavior of HelloWorld.greet() will apply immediately after saving the modified file, even if the change is done after import HelloWorld. That doesn’t happen if you just include include("HelloWorld.jl"). (That would happen if you use the function includet from Revise, though.)

You may have also noticed that when you import HelloWorld for the first time a message of “Precompiling HelloWorld” appears briefly: the impact of this is not very noticeable for tiny toy package examples, but it is very important for packages with larger codes.

1 Like
5

I use PkgSkeleton.jl which is similar but different to PkgTemplates.jl

6

@Krastanov - You sussed out that I was really after an education on the process for myself and for other beginners to Julia. I will eventually use PkgTemplates.jl as nsajko suggests. I was less interested in actually creating the package and more interested in understanding how Julia creates environments for the process. I couldn’t grasp how the environments were involved. When you compared it to Python’s venv, it clicked because I’ve used that in my Linux terminal. I knew that the .toml file played a vital role - your explanation helped pull it together for me.

Brilliant and thorough answer - thanks. I’m sure other beginners and anyone craving a bit of ‘under the hood’ knowledge of Julia will benefit from this post.

2 Likes
7

Thanks - I’m aware of PkgTemplates.jl and its power. I will get to it as it I begin to create actual, useful packages. This post was about seeking a little low-level knowledge on the process.

8

@heliosdrm - This was very helpful. I’ve started to include Revise.jl for more easily moving in and out of the REPL when editing the module, but still learning how it works. Also, I was wondering about that ‘quick’ precompiling message which I wouldn’t always see, depending on my method. You also answered my basic question between using ‘include’ and ‘import’. I really wanted to understand that and you cleared it up - thanks.

2 Likes
10

@tbeason - I will definitely look into PkgSkeleton.jl - thanks for that.

11

I’m aware, that’s exactly what I was aiming for.

12

Edited - 1-23-24 (Apologies for multiple edits - but Discourse doesn’t allow deletion, so I wanted to make right here)

Note:
This is one very simple method - without using PkgTemplates of PkgSkeleton
Verified on WSL (Arch Linux) / Win 10 and on BunsenLabs Boron Linux
Posting here in case it’s of use to others:

Create pkg in ~/.julia
begin in home/ben/.julia
switch to REPL
julia> switch to pkg
pkg> generate HelloWorld
cd HelloWorld
shell> tree HelloWorld (just a check)
navigate - .julia/HelloWorld/src/Helloworld.jl
use micro to edit HelloWorld.jl as follows:

Content of src / HelloWorld.jl is:
module HelloWorld
greet() = print(“Hello World!”)
end # module

Navigate out of src and back to .julia
switch from shell to julia to pkg
pkg > activate ./HelloWorld
switch to julia -
julia> import HelloWorld
julia> HelloWorld.greet()
Hello World!

1 Like
13

Personally I don’t store any of my own packages under .julia/. Keeping .julia/ decoupled from my personal stuff allows me to be carefree about deleting the .julia/ directory or playing with it otherwise.

Instead of activate-ing your package, just dev it. Then you’ll be able to import it without activating it. I never use activate, it’s a rarely needed command.

14

Dev it into what environment? If you mean the default environment it’s not a particularly good idea to fill it with random packages and no more so with packages in development.

The only time I dev packages is when I need to test changes in a dependency or co-develop dependent packages, and in that case I dev into the top package.

I frequently use activate --temp when I need a throw-away environment but rarely for anything else. When I develop a single package I always run julia --project though.

1 Like
15

@nsajko - Right - but for both WSL on Win 10 and my Linux Mint PC - regardless of where I begin the process - using PkgTemplates, then generate() - creates a dev folder in .julia (it isn’t there until the package is created). That new dev folder contains the new package. I work with many different programming languages and platforms - I need to learn the defaults in order to keep things straight in my mind. For both WSL and actual Linux PC - .julia/dev is where Julia creates my new package.

16

A few notes.

The ~/.julia/dev directory can be relocated via the environment variable JULIA_PKG_DEVDIR.

I usually use activate on a package environment when developing that package as a unit. This allows me to add dependencies to the package easily via Pkg.add or Pkg.develop.

I use develop when doing integration testing. In this case I create a new environment and then dev multiple packages into it.

Simplest Package

Only use this for very quick coding competition style programming, rapid prototyping, or in place of scripting. The main advantage over scripting is precompilation.

julia> write(
    "SimplestPackage.jl",
    "module SimplePackage end"
)
24

julia> push!(LOAD_PATH, pwd())
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/mkitti/aoc2023"

julia> using SimplePackage
[ Info: Precompiling SimplestPackage [top-level]

I personally only use this with Advent of Code.

Minimal Package Structure

You probably should use Pkg.generate instead kd this. This is meant for educational purposes only about what the minimal standard package structure needs to be.

julia> mkpath("HelloWorld/src")
"HelloWorld/src"

julia> write(
    "HelloWorld/Project.toml",
    """name = "HelloWorld"\n"""
)
20

julia> write(
    "HelloWorld/src/HelloWorld.jl",
    "module HelloWorld end"
)
21

julia> import Pkg

julia> Pkg.activate("HelloWorld")
  Activating project at `~/HelloWorld`

julia> import HelloWorld
[ Info: Precompiling HelloWorld [06179f83-a236-5011-a36f-f11c85487ca2]

However, you probably want a UUID.

julia> using UUIDs

julia> write("HelloWorld/Project.toml",
       """
       name = "HelloWorld"
       uuid = "$(uuid4())"
       """)

Final thoughts

I would only use include within a package.

1 Like
17

Thanks for the effort to explain all the options - seems everyone uses a different method. Multiple environments and packages to create packages. Honestly, it seems so much more straightforward in Rust.

1 Like
18

Yes. And that is fine. The end product is the same. Doesn’t really matter how you get there.

1 Like
19

That also means it is sufficiently complicated as to require people to seek out alternative methods to overcome limitations of the other approaches.

4 Likes
20

I would not store my stuff in the Julia depot (.julia).
If nothing else, it is inconvenient to locate this folder in the editor.
I use Documents as the root of projects. But, any old directory will do.

I want to make sure I never accidentally delete my files. The
folder .julia can be rebuilt easily if it only contains “common”
files. And, I am not shy about deleting the depot when the :imp: rides me. It can
happen a few times per month. ,-)

3 Likes
21

[quote=“benslinux, post:12, topic:108273”]
Create pkg in ~/.julia/dev

Point taken. This method I documented was created using Arch Linux WSL (no Documents directory in Arch). But even if I create a directory called say ‘Julia_Pkgs’ when I begin the process with: using Packages, then generate() - the new pkg is placed in my Arch WSL .julia /dev directory - (dev is created when pkg is generated). I figured it must just be how WSL manages the environment, but my Linux Mint Desktop does the same thing. Since I’m focused learning the pkg creation process and I know how to navigate Linux, I don’t fight this default and just move the package later.

Obviously, you seasoned Julia users know multiple ways of creating pkgs. I meant this post to maybe be of some help to beginners like myself as one way to complete the process. I probably should have posted this in ‘New to Julia’ - I’d do so now, but don’t know how to move all these threads.