How to ship the exact environment together with a julia script

I did come across with the dependency hell for a Pyton script I wrote 4-5 years ago whose output was finally to get published.
I need to run however my script under a slightly changed scenario and when I try to blindly press run on each cell of a Python notebook… BOOM !! I come to the problem of dependencies that changed their API and I have now to spend time to look at my old code in detail, look how to update it etc, all in order to change a single parameter of a function.

Now, to be fair, I think also in Python there must be ways to solve this problem, but I now switched to Julia :slight_smile: :slight_smile: :-).

Which is the proper way to “ship” a project (that can be a simple script) with the exact version of its dependencies ? ]activate . on the folder , ].add the various dependencies and then ship the folder with the script and the two files Project.toml and Manifest.toml ?

What will it happens when I will try to run the script on a machine with more up to date packages ? Will it try to update the dependant libraries automatically?

And what if the machine has a newer version of Julia? I did notice that both Project.toml and Manifest.toml do not contain any information on the Julia base version.

1 Like

Yes, this is the way to do it. You need to be observant that you don’t depend on anything in your global environment though, preferably by removing it from your load path.

No, it will only update versions of dependencies if you run Pkg.update, so don’t do that. You may need to Pkg.instantiate() though to make all the packages available in the right versions and it’s a good idea to start your script with that.

Yes, this is a known weakness. If you can instantiate on a newer version of Julia with the same major version you should be good to go but otherwise you need to run an older version of Julia. It’s a good idea to document which version of Julia was used originally.
Edit: This doesn’t sound right. Updating to a higher minor version should be fine but going to an older version can certainly be problematic, so it’s at least a good idea to document the used version for that reason.

1 Like

How do I do that (be sure that if I use some package that’s has been included in the local Project/Manifest file) ?

See How to install packages which are needed only for development for a related discussion. In contrast to that case you most likely want to keep @stdlib though.

I see. It would have been nice if there was a way to “force” it from Julia itself rather than using env variables, something like Pkg.envpaths(["."])

You can do it from within Julia, you just manipulate the global variable LOAD_PATH. See e.g. Creating empty isolated environment

1 Like

Haha, seems I got lost in my searches. That was what I had intended to link to.

1 Like

I think I finally got it.
for example on a project where I need DataFrames:

Only once:

cd(@__DIR__)
using Pkg
Pkg.activate(".")
Pkg.add("DataFrames")
# .. add the other package needed...

In myProject.jl

# This script works on julia 1.5
cd(@__DIR__)
using Pkg
push!(empty!(LOAD_PATH), ".")
Pkg.activate(".")
Pkg.instantiate()

using DataFrames # Ok, in local {Project|Manifest}.toml as it has been specifically added
a = DataFrame(a =[1,2,3], b=["aaa","bbb","ccc"])
using Flux # In global, but not in local toml files, it will (correctly) rise an error

Ship:
⤑ the folder with myProject.jl, Project.toml and Manifest.toml

Looks about right to me!

You can also just use the package REPL mode (type ] to enter the mode, then add DataFrames etc), instead of a separate script.

You can do this in one line, which maybe Juno would like better? push!(empty!(LOAD_PATH), ".")

3 Likes

correct, edited my post above…

You can (and should) include a [compat] entry for julia, too.

2 Likes

To add on to what @Tamas_Papp wrote, you can find the documentation about compatibility in the docs of Pkg (linked from the documentation of julia).

2 Likes

What is the effect of adding Julia in the compat section ?

I did add

[compat]
julia = "2.6.0"

or even

[compat]
julia = "= 2.6.0"

but the script still runs without giving any warning… (no, I don’t have Julia 2.6 installed :upside_down_face: :upside_down_face:)

On master, you should get a warning along the lines of

[ Warning: julia version requirement for project not satisfied

when you instantiate. I imagine this will be dealt with more strictly when 2.0 is released.

ok, thank you for the info (I don’t have the warning with Julia 1.5.0)