I spent a long time chasing something very confusing to me in Pkg.
when I run our code on the command line as we always do, it seems to run
julia -e 'push!(LOAD_PATH,"Zinstrument/");using Zinstrument;ProcessDatabase()'
but runs with a old version of the code
when I do the same in the repl
push!(LOAD_PATH,"Zinstrument")
4-element Vector{String}:
"@"
"@v#.#"
"@stdlib"
"Zinstrument"
julia> using Zinstrument
[ Info: Precompiling Zinstrument [d6e76910-d7cc-43bf-969a-b976b4bc5f70]
WARNING: using Zinstrument.Zinstrument in module Main conflicts with an existing identifier.
ERROR: importing Zinstrument into Main conflicts with an existing identifier
The repl ERROR behavior seems correct, but is the command line also correct?
yes, can be solved, but it is concerning to us that one can end up running different code depending on whether one uses the repl or command line in exactly the same way.
and this took us hours to diagnose as all was correct in the development (repl) and not in production runs (command line), on julia 1.8.
Is the REPL the plain Julia REPL started from a terminal. Or is it inside VS Code or Juno? Those IDEs might automatically activate the current project enviroment e.g. the equivalent of julia --project=.
I do believe changing the LOAD_PATH is not considered good practice, and it is not suggested anywhere in the documentation (correct me if I am wrong). So I would recommend steering away from it.
Thanks. So a weird corner case we should not have found. Fair enough. We will search and eliminate.
I forget what brought us to that solution, but reason was because we never understood how Pkg worked and all our code lives in a shared drive and run on different julia versions and op systems. We have moved away from using Pkg to just modules but this was a leftover.
I highly recommend using Pkg.develop to load code from a local path. You are creating an ad hoc environment stack.
Now there seems to be several issues here;
What is the current active environment of the REPL versus the script? Do you use the --project option in either case or do you attempt to Pkg.activate anything? Is the environment variable JULIA_PROJECT define? What is the output of pkg"status" in the REPL? What is the output of InteractiveUtils.versioninfo() in the REPL?
There seems to be a module called Zinstrument that contains another module called Zinstrument. This seems odd. Could you tell us more about how code is structured. What are the relative paths of files called Zinstrument.jl? How do you use include, using, and import?
module Zinstrument
using Random
using Distributions
using DataFrames
using JSON
using SpecialFunctions
using Optim
using Statistics
using TimeSeries
using CSV
using Parquet
using NamedTupleTools
#using SHA
#using AWS
include("Functions.jl")
include("Forecast.jl")
include("MakeZinstrumentFiles.jl")
end
We have shared drive on a zfs with frequent snapshots, all backed up to 2 zfs remote systems. We used to use git, but now the team prefers the simplicity of zsh snapshots, and the added benefit of all data, code, binaries, etc. can be found as it existed at any past date via the simplicity of the .zfs folder. so no git. (yes, git branches would make sense, but team refused). The zfs and rest is managed by IT support not us.
we followed the instructions here, 5. Creating Packages · Pkg.jl and that is why we need to use push!(LOAD_PATH. It says to cd into the folder, but push!(LOAD_PATH seems to do the same. Except we ended up with the inconsistency between command line and repl (and old code running on command line)
something like the simplicity of R local libraries
if a package is changed on one system, and tested to work there, it will automatically run on any remote system regardless or operating system or julia version.
The ability to have “using MyPackage” instead of having to manage paths.
No change to the environment of the remote system, except to download julia and its packages.
documentation that someone with a PhD in physical sciences and not computer science will understand. Team does science not coding, except when necessary, and then they will do as little as they get away with. They like simple recipes.
@ufechner7 I know we have special needs and outside of what Julia normally does. Besides packages, Julia is a wonderful language and our code is simpler, faster to run and code, and more reliable than when we used R+Rcpp before. And we have become annoying Julia evangelist in our org.
and thank you all for making such a wonderful language
It is very brave to try to do serious SW development in a team without using git. I know git is not very user friendly, but I think it is worth the effort to learn it if you program more than one hour per week, even if you are a scientist who works on his own…
sorry for all the trouble, just was confused by different Julia behavior when run in repl and command line. and wanted to know why. but now I know what not to do.
And I appreciate we had a weird case, packages without github.
and in any case, we can make it all run without packages, modules work fine.
I know, I have used git a lot and know how it works, but when one never makes branches, the advantages of git over alternatives are not so clear. we snapshot our zfs every few minutes. and when we have binary files, and data, need something besides git. Still we use git.
we have several packages, and other code to run things. My understanding of julia packages is that each needs its own git repository, ideally on githup. So does not play nice with all our packages being in same git repository.
and even if one lives by git, there is no reason to keep packages in separate repositories.
So if there was a way to easily use Julia packages without git (I know it can be done, but its not easy), would much prefer that.
FWIW, there is no need to use git at all for developing packages locally. Pkg has support to directly download a package from an URL using git, but you can just as well download it normally and use Pkg.develop to get it into your environment.
The git stuff in Pkg is a feature but it isn’t really required for many things.
if it is just my code, I happily use git and julia packages on githup.
for a group of people who are not computer scientists its different (in my world). They complain about all that that is complex and does not bring immediate benefits.
I deal with a team that uses Dropbox history to manage their code. And that is prohibited by our security policy. That got overridden. (I run git on cron without them knowing every 30 min. )
most are happy to keep 100s of source files from all sorts of languages in same folder. see no benefit in multiple folders. most common view in the youngest collegues who use search and not folders.
my collegues love zfs snapshots. automatic, simple, handled by IT.
So when it comes to Julia. They told me point blank. "No packages. No git. unless you pay someone to do it for us. " except the language was not that polite. Modules are a hard sell, but now accepted, sort of. Sellable benefit: no namespace collusions.
so my lesson learned is: lowest common denominator. All help comes from stackoverflow. Don’t expose anybody to compexity that does not bring immediate benefits.