Shared environments?

The thing is that there are two types of "need"s. One is the actual dependencies of the package/script, other is the eventual necessity to use some tool (Plots, for instance) to check or test something.

1 Like

yeah, I’ve been following those possibilities, and while they are a reasonable path when we are talking about a more involved package development workflow and unit testing, they do not solve the problem for more pedestrian scripts or even for the eventual use of a package or set of packages for checking something that should not persist in the development workflow in any way.

Something like

from @plots using: Plots
from @optim using: Optim, Ipopt

would be nice to have.

1 Like

For that I usually have a different folder with a local environment (lets call it scripts), that has the the package dev’ed and those additional dependencies that shouldn’t be part of the project itself.
So it looks like this

project/
├─ src/
├─ test/
├─ scripts/
│  ├─ Project.toml
│  ├─ Manifest.toml
├─ Project.toml
1 Like

Indeed, that is something that was suggested in a recent thread. Yet, that sort of implies that one would not be developing the package in the package environment. I see that is why a shared environment was suggested for that exact use. All possible, surely, but another layer of complication (and not composable either, since one still needs one of those environments for each project - sharing those would end causing the same dependency problems we may have with @v1.7).

Thats actually a good thing and while one may need to get used to the habit, its not much of an overhead.

I keep getting that answer :slight_smile: , but I have the impression that this is a misconception about a complete set of kind of people that will use Julia. As a package developer I see the many advantages of that, but:

  1. I don’t think it is a good thing that I have to manage environments just to sporadically plot some data which I happen to have to see while developing a package. I just want to use Plots once.

  2. Most of the users of my packages will never develop packages. They will follow a tutorial, and get their results. Having the possibility of these users having issues with dependencies of other projects, of asking them to get used to using environments is out of question.

And I think that 2 is quite transparent to everybody here. I don’t know of any package author that suggests how to deal with environments before installing his/her package. We don’t want our users to have these barriers, so we tell them to ] add MyPackage and using MyPackage and that is it. And it should safe to do that.

edit: I consider this thread solved, my delusional speculations on this are on this other thread.

2 Likes

I don’t think these are mutually exclusive. What people in this thread are asking (and what I think was the intention behind environments, other than seperation of concerns) was making that whole process of e.g. following a tutorial easier. If you don’t create new environments, you’ll have packages from old tutorials laying around, messing things up. Julia gives you the tool to solve that: use a new environment for that purpose.

This of course doesn’t stop you from having e.g. Plots in your main environment, if you’re using that all the time anyway. They stack after all and if you ]activate a different environment you’ll still be able to use Plots. That’s all people are advocating for here. Saying that this workflow is “out of the question” and asking other people to work around your preferred workflow instead (even if the workflow exists to solve real problems people have!) seems a little selfish, if I’m honest.

Only because the default Julia behavior is to start in an environment that we are advocating that should not be used (that’s why I started the other thread).

Sorry if that wording felt unappropriated. It was not my intention. To be sincere, I am well aware that any suggestion I give can be dismissed for technical reasons that are way beyond my understanding, and I am ok with that. My motivation for the other thread was simply that I thought that maybe it was something that someone with the skills could read and think about it, perhaps (I hope) agree that there is something not quite ideal in the current state, and give a thought about it. It is likely that what I propose there is not a solution to anything, but something of it may be worth rescuing.

In particular, I think that the idea that Julia should start in a temporary environment with some particularly behavior that avoided these cluttering problems is something that deserved some thought. I cannot know in advance if that possibility is already exhausted.

The out of question was about telling the users of my package that they should do all that. I’m sorry, but I am the one who knows the users of the package. I do think that environments are a powerful feature of Julia. But being forced to use them (even more than one, with careful stacking considerations) is not user friendly. I think this is transparent: There is not a single package in the whole Julia ecosystem that guides the new user through the workings of environments before telling them just to use the package and have fun.

People are expected to add the packages and start following the code, like here, or here, or here, to give a few examples of important packages (examples of which I know no exception). What happens in practice is that the users naturally start bloating the base environment to find out later on that this environment thing exists. It shouldn’t be that way and, IMHO, not because the environments should be presented to the users earlier, but by making that workflow safe(r).

3 Likes

Would a flag for julia to start in a temporary environment work?
Something like julia --temp ?
I don’t know if that is possible, but that was a thing I was looking for in the past.

I would like to have that, would save me a few keystrokes every time. But I do not think that the current behavior of temporary environments is completely ideal, because one still needs to add the packages explicitly to them, even if the packages are available in the local machine (implying possible delays and a lot of on-screen noise). Also I think that starting in a temporary environment (with specificities) should become the default behavior, because it is for more sporadic and casual users of the ecosystem that this makes most of a difference avoiding headaches.

In the interactive REPL one does not need to add the packages manually. Typing using is enough.
Would you like to have this behavior with automatic installs also when you run a file like julia file.jl?

(@v1.7) pkg> activate --temp
  Activating new project at `/tmp/jl_0ZUPqJ`

julia> using Unitful
 │ Package Unitful not found, but a package named Unitful is available from a registry. 
 │ Install package?
 │   (jl_0ZUPqJ) pkg> add Unitful 
 â”” (y/n) [y]: y
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `/tmp/jl_0ZUPqJ/Project.toml`
  [1986cc42] + Unitful v1.11.0
    Updating `/tmp/jl_0ZUPqJ/Manifest.toml`
  [187b0558] + ConstructionBase v1.3.0
  [1986cc42] + Unitful v1.11.0
  [56f22d72] + Artifacts
  [ade2ca70] + Dates
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [4536629a] + OpenBLAS_jll
  [8e850b90] + libblastrampoline_jll

julia> 1u"eV"  |> u"J"
1.602176634e-19 J

To me at least it’s not about never using the main environment (I use it all the time when testing a MWE from this forum, for example) but rather to only put stuff in there you know you’re using all the time. Maybe that should be mentioned more in tutorials :thinking:

I agree insofar that the “teaching people about environments” part is… iffy right now for people not used to the benefits of them. Maybe this can be combined with the above and with pkg> activate @myenv to amend those tutorials, to mention something along these lines:

In julia it’s customary to use environments to seperate different projects. For this tutorial, we’re also going to use an environment, to keep anything we’re messing around with/trying out here separate from our existing projects and to not interfere with them. We can create an environment just for this tutorial by typing the following into the REPL:
] activate @ODEtutorial
This also has the advantage of being able to return to this tutorial at a later date, and resume where you left off, by activating the same @ODEtutorial again. If you’re interested, you can find out more here (insert link about detailed explanation here) about environments in julia.

Would that help alleviate the problems while also (gently) introducing the concepts?

1 Like

Exactly, that is part of it, with some additions: that using should install the package locally available only. If the user never added the package, I think it is fine to error.

And, in the REPL, the output of the command should be cleaner, it should “almost” feel like a using from the shared @v1.7 environment. Maybe with one line of warning.

I think that is a great way to introduce the environments as a feature, and important feature.

I also tested the MWEs in the base environment, until it became a mess. From then on, I start a new temporary environment every time for that. And I would love to be able to save those at the end.

The more I use Julia, the more I feel that the base environment is like a root account, which I should be very careful in using, where changes can cause widespread confusion in all my development workflows. I feel thus that it should be hidden, not exposed, by default.

So something like automatically install packages when locally available.
Maybe this can be achieved with Pkg.offline()

julia> import Pkg

julia> Pkg.offline()

julia> using Unitful
 │ Package Unitful not found, but a package named Unitful is available from a registry. 
 │ Install package?
 │   (jl_jMsi1H) [offline] pkg> add Unitful 
 â”” (y/n) [y]: 
   Resolving package versions...
    Updating `/tmp/jl_jMsi1H/Project.toml`
  [1986cc42] + Unitful v1.11.0
    Updating `/tmp/jl_jMsi1H/Manifest.toml`
  [187b0558] + ConstructionBase v1.3.0
  [1986cc42] + Unitful v1.11.0
  [56f22d72] + Artifacts
  [ade2ca70] + Dates
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [4536629a] + OpenBLAS_jll
  [8e850b90] + libblastrampoline_jll

julia> using Javis
ERROR: ArgumentError: Package Javis not found in current path:
- Run `import Pkg; Pkg.add("Javis")` to install the Jarvis package.


1 Like

Yes, the tools are there, this is what I mention here: Speculations about the default environment (or a new draft environment) - #3 by lmiq

Would be mostly about deciding what the default behavior should be.

1 Like

I am not sure how one would force the automatic installation when not using the REPL.

For a first try one can use the startup.jl file to configure the default behavior.

startup.jl

import Pkg
Pkg.offline()
Pkg.activate(temp=true)

macro using_and_add(pkg)
    quote
        #TODO check if installed
        Pkg.add($(string(pkg)))
        using $pkg
    end 
end

With this, one would have to use @using_and_add Package instead of using Package, but it also works if only a file is executed.

1 Like

So exactly. What would be the downsides of behaving like that?

With that, if the package is not installed anywhere, we can tell the user to install it.

My idea was to have that behavior activated by a draft environment. For instance, the user would do:

% julia --draft

and the environment would be a temporary environment with those options turned on, but where instead of the new macro, the using command would behave like that. With that, scripts would not be environment-specific.

With the additional feature of having a save_environment("my_new_env") feature, with the proper information that then the environment is bounded to the versions of the packages in use at that time (with the advantages that that may bring).

(then I think that this would be a nicer default behavior than the current one, but that’s another story).

One could write that:

A draft environment is fresh environment designed for experimenting and using Julia scripts. It tries to be responsive by not downloading new package versions if local installed versions are available. Adding new packages or updating previous packages must be an explicit action, and require a working internet connection. A draft can be saved into a full featured reproducible environment, in which the package versions used are annotated, with save_env("@new_env"). Draft environments are a good alternative to run package tutorials, plotting, and other tasks that require perhaps large packages to be loaded, but do not involve a heavy involved development workflow.

1 Like

To save the project this might work:

function save_environment(target) 
   isdir(target) && cp(Base.active_project(),joinpath(target,"Project.toml"))
end
1 Like

With the exception of the command line flag, could this be implemented in a package?