Pkg behavior that confused me

Hello all

I spent a long time chasing something very confusing to me in Pkg.

when I run our code on the command line as we always do, it seems to run

julia -e 'push!(LOAD_PATH,"Zinstrument/");using Zinstrument;ProcessDatabase()'

but runs with a old version of the code

when I do the same in the repl

push!(LOAD_PATH,"Zinstrument")
4-element Vector{String}:
 "@"
 "@v#.#"
 "@stdlib"
 "Zinstrument"

julia> using Zinstrument
[ Info: Precompiling Zinstrument [d6e76910-d7cc-43bf-969a-b976b4bc5f70]
WARNING: using Zinstrument.Zinstrument in module Main conflicts with an existing identifier.
ERROR: importing Zinstrument into Main conflicts with an existing identifier

The repl ERROR behavior seems correct, but is the command line also correct?

sorry for the confusion

best, Jack

Maybe it gets picked up from one of the earlier environments. Try pushfirst! instead maybe.

2 Likes

Thanks @kristoffer.carlsson

yes, can be solved, but it is concerning to us that one can end up running different code depending on whether one uses the repl or command line in exactly the same way.

and this took us hours to diagnose as all was correct in the development (repl) and not in production runs (command line), on julia 1.8.

all the best, jack

Is the REPL the plain Julia REPL started from a terminal. Or is it inside VS Code or Juno? Those IDEs might automatically activate the current project enviroment e.g. the equivalent of julia --project=.

1 Like

I do believe changing the LOAD_PATH is not considered good practice, and it is not suggested anywhere in the documentation (correct me if I am wrong). So I would recommend steering away from it.

2 Likes

in this specific case, directly from the terminal.

Thanks. So a weird corner case we should not have found. Fair enough. We will search and eliminate.

I forget what brought us to that solution, but reason was because we never understood how Pkg worked and all our code lives in a shared drive and run on different julia versions and op systems. We have moved away from using Pkg to just modules but this was a leftover.

thanks a lot for all the clarifications

best, jack

1 Like

Not a good idea. You should always use project specific environments, and for these you need Pkg…

And if there is something you do not understand just ask…

2 Likes

I highly recommend using Pkg.develop to load code from a local path. You are creating an ad hoc environment stack.

Now there seems to be several issues here;

  1. What is the current active environment of the REPL versus the script? Do you use the --project option in either case or do you attempt to Pkg.activate anything? Is the environment variable JULIA_PROJECT define? What is the output of pkg"status" in the REPL? What is the output of InteractiveUtils.versioninfo() in the REPL?

  2. There seems to be a module called Zinstrument that contains another module called Zinstrument. This seems odd. Could you tell us more about how code is structured. What are the relative paths of files called Zinstrument.jl? How do you use include, using, and import?

2 Likes

We don’t use --project. JULIA_PROJECT is not defined

julia> Pkg.status()
Status ~/.julia/environments/v1.8/Project.toml
[fbe9abb3] AWS v1.76.1
[336ed68f] CSV v0.10.4
[a93c6f00] DataFrames v1.3.4
[31c24e10] Distributions v0.25.68
[59287772] Formatting v0.4.2
[c27321d9] Glob v1.3.0
[682c06a0] JSON v0.21.3
[bdcacae8] LoopVectorization v0.12.124
[e1d29d7a] Missings v1.0.2
[d9ec5142] NamedTupleTools v0.14.1
[429524aa] Optim v1.7.2
[d96e819e] Parameters v0.12.3
[626c502c] Parquet v0.8.4
[92933f4c] ProgressMeter v1.7.2
[6f49c342] RCall v0.13.13
[295af30f] Revise v3.4.0
[276daf66] SpecialFunctions v2.1.7
⌃ [9e3dc215] TimeSeries v0.14.0
Info Packages marked with ⌃ have new versions available

InteractiveUtils.versioninfo()
Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
OS: macOS (arm64-apple-darwin21.3.0)
CPU: 8 × Apple M1
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
Threads: 1 on 4 virtual cores

it lives is folder called Zinstrument
all we do is
push!(LOAD_PATH,“Zinstrument”)
using Zinstrument

Zinstrument: ✗ tree
.
├── Zinstrument
│ ├── Manifest.toml
│ ├── Project.toml
│ └── src
│ ├── Forecast.jl
│ ├── Functions.jl
│ ├── MakeZinstrumentFiles.jl
│ └── Zinstrument.jl

Zinstrument.jl looks like

module Zinstrument

using Random 
using Distributions 
using DataFrames 
using JSON 
using SpecialFunctions 
using Optim 
using Statistics 
using TimeSeries 
using CSV 
using Parquet
using NamedTupleTools
#using SHA

#using AWS
include("Functions.jl")
include("Forecast.jl")
include("MakeZinstrumentFiles.jl")
end 

Thanks @ufechner7

We have shared drive on a zfs with frequent snapshots, all backed up to 2 zfs remote systems. We used to use git, but now the team prefers the simplicity of zsh snapshots, and the added benefit of all data, code, binaries, etc. can be found as it existed at any past date via the simplicity of the .zfs folder. so no git. (yes, git branches would make sense, but team refused). The zfs and rest is managed by IT support not us.

we followed the instructions here, 5. Creating Packages · Pkg.jl and that is why we need to use push!(LOAD_PATH. It says to cd into the folder, but push!(LOAD_PATH seems to do the same. Except we ended up with the inconsistency between command line and repl (and old code running on command line)

That page says to use templates, GitHub - invenia/PkgTemplates.jl: Create new Julia packages, the easy way , and we found the quote on that page " Using PkgTemplates is straightforward." to be interesting. Perhaps simple if code lives on githup, but not else. For us, PkgTemplates.jl is very complicated and hard to understand. 5. Creating Packages · Pkg.jl is simpler.

All we want is

  • not to use git
  • something like the simplicity of R local libraries
  • if a package is changed on one system, and tested to work there, it will automatically run on any remote system regardless or operating system or julia version.
  • The ability to have “using MyPackage” instead of having to manage paths.
  • No change to the environment of the remote system, except to download julia and its packages.
  • documentation that someone with a PhD in physical sciences and not computer science will understand. Team does science not coding, except when necessary, and then they will do as little as they get away with. They like simple recipes.

@ufechner7 I know we have special needs and outside of what Julia normally does. Besides packages, Julia is a wonderful language and our code is simpler, faster to run and code, and more reliable than when we used R+Rcpp before. And we have become annoying Julia evangelist in our org.

and thank you all for making such a wonderful language

all the best, Jack

Did you try to run julia with the option

--startup-file=no

?
A startup file might explain the difference between repl and script use…

Otherwise your usage scenario is very special, it will be difficult to get support for this from the open source community…

1 Like

It is very brave to try to do serious SW development in a team without using git. I know git is not very user friendly, but I think it is worth the effort to learn it if you program more than one hour per week, even if you are a scientist who works on his own…

Hi

we have no startup files.

sorry for all the trouble, just was confused by different Julia behavior when run in repl and command line. and wanted to know why. but now I know what not to do.

And I appreciate we had a weird case, packages without github.

and in any case, we can make it all run without packages, modules work fine.

all the best, Jack

I know, I have used git a lot and know how it works, but when one never makes branches, the advantages of git over alternatives are not so clear. we snapshot our zfs every few minutes. and when we have binary files, and data, need something besides git. Still we use git.

we have several packages, and other code to run things. My understanding of julia packages is that each needs its own git repository, ideally on githup. So does not play nice with all our packages being in same git repository.

and even if one lives by git, there is no reason to keep packages in separate repositories.

So if there was a way to easily use Julia packages without git (I know it can be done, but its not easy), would much prefer that.

Can you share your “lessons learnt” so that others can avoid to make the same mistake?

FWIW, there is no need to use git at all for developing packages locally. Pkg has support to directly download a package from an URL using git, but you can just as well download it normally and use Pkg.develop to get it into your environment.

The git stuff in Pkg is a feature but it isn’t really required for many things.

2 Likes

if it is just my code, I happily use git and julia packages on githup.

for a group of people who are not computer scientists its different (in my world). They complain about all that that is complex and does not bring immediate benefits.

I deal with a team that uses Dropbox history to manage their code. And that is prohibited by our security policy. That got overridden. (I run git on cron without them knowing every 30 min. )

most are happy to keep 100s of source files from all sorts of languages in same folder. see no benefit in multiple folders. most common view in the youngest collegues who use search and not folders.

my collegues love zfs snapshots. automatic, simple, handled by IT.

So when it comes to Julia. They told me point blank. "No packages. No git. unless you pay someone to do it for us. " except the language was not that polite. Modules are a hard sell, but now accepted, sort of. Sellable benefit: no namespace collusions.

so my lesson learned is: lowest common denominator. All help comes from stackoverflow. Don’t expose anybody to compexity that does not bring immediate benefits.

1 Like

Yes, I know it possible and am reading 5. Creating Packages · Pkg.jl carefully

thanks for the comments

jack