Workflow for developing application in new Pkg manager


#1

I’m happy to see that the documentation in the 0.7 manual for the new package manager is greatly expanded, but it’s still a bit thin in describing how to develop applications (as opposed to packages). I’d like to develop my application using the package manager, but I’m not sure how.

Following the instructions in the manual about application development, I executed activate . in Pkg mode of the REPL in a new, empty directory, followed by add for packages needed by my application. What’s next? Should I create a subdirectory called src and place my .jl files in it? I did that. For example, suppose that the entry point to my application is a no-argument function run() in a module called app in a file called app.jl in the src subdirectory. How do I execute my application so that (1) the loader knows to look for source files in that directory and (2) the correct packages that were added with the add command are available when the application starts?

Also, is Revise.jl the recommended way to keep the REPL up-to-date with edits made to the source files during development?


#2

That’s pretty much it.

If it is the active project, and run() is in the main module, it could be a simple as

import MyApplication # once only when using Revise.jl
MyApplication.run()

Yes.


#3

Thanks for the suggestions! But it’s still not working for me (0.7.0-release). I created a source file c:/Users/vavasis/Documents/TestApp/src/TestApp.jl as follows:

module TestApp

function run()
    println("TestApp has run")
    nothing
end

end

Here’s what happens in Julia afterwards:

              _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0 (2018-08-08 06:46 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> using Revise
[ Info: Precompiling Revise [295af30f-e4ad-537b-8983-00126c2a3abe]

julia> cd("c:/Users/vavasis/Documents/TestApp")

(v0.7) pkg> activate .

(TestApp) pkg>

julia> import TestApp
ERROR: ArgumentError: Package TestApp not found in current path:
- Run `Pkg.add("TestApp")` to install the TestApp package.

Stacktrace:
 [1] require(::Module, ::Symbol) at .\loading.jl:817

#4

I don’t have a good suggestion, but what does pkg"status" say after pkg"activate"?


#5

I have a trace below showing the output of st. There is one other feature to note about this trace that I don’t understand. After the activate command, the system variable LOAD_PATH is unaffected, but the unexported function Base.load_path() has changed.

              _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.0 (2018-08-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> LOAD_PATH
3-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"

julia> Base.load_path()
2-element Array{String,1}:
 "C:\\Users\\jtest\\.julia\\environments\\v1.0\\Project.toml"
 "C:\\Users\\jtest\\AppData\\Local\\Julia-1.0.0\\share\\julia\\stdlib\\v1.0"

julia> cd("c:/Users/jtest/Documents/TestApp")

(v1.0) pkg> activate .

(TestApp) pkg> add Revise

 Resolving package versions...
  Updating `Project.toml`

[snip]
(TestApp) pkg> st
    Status `Project.toml`
  [295af30f] Revise v0.6.4

(TestApp) pkg>

julia> import TestApp
ERROR: ArgumentError: Package TestApp not found in current path:
- Run `Pkg.add("TestApp")` to install the TestApp package.

Stacktrace:
 [1] require(::Module, ::Symbol) at .\loading.jl:817

julia> LOAD_PATH
3-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"

julia> Base.load_path()
3-element Array{String,1}:
 "c:\\Users\\jtest\\Documents\\TestApp\\Project.toml"
 "C:\\Users\\jtest\\.julia\\environments\\v1.0\\Project.toml"
 "C:\\Users\\jtest\\AppData\\Local\\Julia-1.0.0\\share\\julia\\stdlib\\v1.0"

julia>

#6

How about

cd c:/Users/vavasis/Documents
]dev ./TestApp

?


#7
julia> cd("c:/Users/jtest/Documents/")

(v1.0) pkg> dev ./TestApp
ERROR: expected a `name` entry in project file at c:\Users\jtest\Documents\TestApp\Project.toml

#8

OK, I got this to work! The missing step was to cobble together my own Project.toml file. I copied the example out of the Julia manual, changing the name from HelloWorld to TestApp, changing a random digit in the UUID (so as not to collide with a possibly already existing UUID), and putting my own email address. Then I executed activate ., and now import TestApp works as expected. The next step is to try this with my real application.


#9

If I understand it correctly there shouldn’t be a need to modify the Project.toml file manually,] generate is doing what you are describing above.


#10

According to the docs, generate is for generating a new package, whereas I am trying to develop an application project. These are distinct according to the docs:

Projects vs. Packages vs. Applications:

  1. Project is an umbrella term: packages and applications are kinds of projects.
  2. Packages should have UUIDs, applications can have a UUIDs but don’t need them.
  3. Applications can provide global configuration, whereas packages cannot.

#11

True, but what is the actual difference between the end result of what you did and what ] generate TestApp gives? It seems like the same thing to me. Is this just a mistake in the wording of the docs?


#12

I think this matter is probably best resolved by opening an issue with respect to the documentation. So I just did this:

(EDIT: moved the issue to Pkg.jl instead of julia)


#13

I just want to pipe in to say I am also confused/unsatisfied with the documentation on how to use to “project environments” and Pkg3 for application development.

My use case is both simple and typical. I want to release code along with a scientific paper, and I want that code to have a easy way for the user to install the project dependencies. In python it is simple (and hacky):

  1. I create a virtual environment and install all dependencies in it (git ignored).
  2. I create/include a “requirements.txt” file via pip freeze in the repository
  3. The user clones my code, creates a virtual environment, and points pip to the requirements file, installing the dependencies. (I can even give them a shell script that does all this for them).
  4. My source code can now be run in the correct python environment by either:
    a) activating the virtual environment before calling the python scripts, or
    b) specifying the python binary which should be used, e.g., venv/bin/python src/mycode.py

Both 4a and 4b are useful: a) when the user wants to run my code in a REPL, add new packages, edit things, etc., and b) when I want to include a shell script (e.g. run.sh) that calls python (and the dependencies I specified).

Can someone clarify how I do this in Julia 0.7.0 / 1.0.0? Preferably going over the Julia equivalent of each step I mentioned? I seem to have found a cumbersome way of achieving 4a, but 4b is still eluding me.


#14

Okay I think I figured it out:

  1. In the project directory (e.g. MyApp ) run ] activate . , then add any dependencies via (MyApp) pkg> add SomeDep .
  2. Include the Project.toml and optionally the Manifest.toml in the repository (these live in the root of the project dir).
  3. The user clones my code, then either runs ] activate . and (MyApp) pkg> instantiate , OR julia --project -e "import Pkg; Pkg.instantiate()" (where the latter can be put into a setup script).
  4. a & b) the user / shell script simply adds the --project command line option when calling Julia.

Is this the best practice for my use case?

If so, what I was missing (and what is missing in the docs) is the --project option. Note it doesn’t even show up with julia --help . (I’m on Version 0.7.0 (2018-08-08 06:46 UTC) x86_64-apple-darwin14.5.0)


#15

AFAIK simply

pkg> add repo-url

should take care of everything.


#16

Hmm, will that work…? I’m not sure. That seems like the best way to proceed if the project is structured as a Julia package.

My use case falls under what the Pkg docs consider an application. I’m releasing a repository that simply showcases my experiments, scripts etc. I’m not releasing reusable code (in the sense of a package), but rather I’m showing how I’ve used a bunch of existing packages to create the experimental results I’ve published. Giving the user a platform to continue experimenting.


#17

Are there any docs about the --project flag? It’s the first time that I see it.


#18

The only documentation I found is here. But it’s only mentioned in passing. Basically seems like a shortcut to setting the JULIA_PROJECT environment variable to be the current directory.