# Best practice for packages on shared drives

**URL:** <https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374>\
**Category:** General Usage\
**Created:** [June 26, 2022, 10:31pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374 "2022-06-26T22:31:37Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 26, 2022, 10:31pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/1 "2022-06-26T22:31:37Z")

</div>

Hi

We have julia running on both linux and mac, and on intel and M1, and the julia versions might not the same. We use synced folders which contains all the code for our in-house packages, the applications that use the packages and data and output. The .julia folder is specific to the system. This code runs unattended, and we might not log onto some computers for months, and hopefully eventually years.

And it all works so long as we only run a package on one system only. But not if we run it on many, apparently our in-house package dependencies all get messed up. So we end up having to install the julia packages again and resolve, which somehow ends working after some time, but then it messes up the other setup that used to run fine.

My team members tell me that its the fault of julia package system and all would be good if we stopped using packages, adding if only julia’s package system for in-house packages was like that in R. I can see that would work, but stop using in-house packages seems not the best practice.

So I come to you to ask what is the best practice to allow our in-house packages to run on many systems on shared drives.

I am sure there information somewhere in the package documentation on this, but I havent been able to either find it or understand it.

all the best, Jack

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [June 26, 2022, 11:03pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/2 "2022-06-26T23:03:41Z")

</div>

I would use a different JULIA\_DEPOT\_PATH for each platform:  
[https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA\_DEPOT\_PATH](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_DEPOT_PATH)

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 27, 2022, 7:08am UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/3 "2022-06-27T07:08:51Z")

</div>

Thanks @mkitti

We will look at JULIA\_DEPOT\_PATH

But I am coming to the conclusion that packages are meant to solve something more complicated than a small team of users that are scientists and not professional programmers managing code on heterogeneous platforms. So if JULIA\_DEPOT\_PATH does not help, will move off packages.

best, jack

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 27, 2022, 7:51am UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/4 "2022-06-27T07:51:53Z")

</div>

If your shared folders contain Manifest.toml files, they are fundamentally incompatible between julia versions. They might sometimes work, but officially Manifests are julia version-specific.  
This cannot really be any different in general. Suppose that there is dependency PkgA with PkgA@0.1 only compatible with Julia 1.6, and PkgA@0.2 only compatible with Julia 1.7. Then there would be no possible configuration that works with both Julia 1.6 and 1.7.

Why do you need to use a shared folder at all for in-house packages? It makes sense to treat them similar to other packages and just specify required versions in your project/application dependencies.

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 27, 2022, 8:11am UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/5 "2022-06-27T08:11:03Z")

</div>

Thanks @aplavin

That is indeed what we have come to understand. And is seems very sensible indeed.

> [@aplavin](#):
>
> Why do you need to use a shared folder at all for in-house packages?

This came about because we did not understand julia packages. Its very simple if on the same system, and we didn’t think it though. I am now looking at custom registries, but that seems yet another layer of complexity.

If Julia offers the same as R: Keep package source in shared drive, then inside them: library(devtools); document(); install(); every time code changes, and then library(mylibrary) where code is used, it would be fantastic. Maybe its very simple indeed to achieve that, but I haven’t discovered how.

best, jack

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 27, 2022, 8:38am UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/6 "2022-06-27T08:38:52Z")

</div>

I’m not familiar with R packages story, but from this description seems like they are installed globally, for the whole R installation on a computer. Julia Pkg allows independent environments, with completely independent package sets and their versions. This is crucial for reproducibility, eg when after a year you want to rerun some analysis and get the exact same results.

A custom registry (using LocalRegistry.jl) is easy to set up and, most imporantly, use: just `using LocalRegistry; register()` to register a new package version. Unlike the General registry, it’s instant: the new version is immediately available to all users of the registry.

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 27, 2022, 12:35pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/7 "2022-06-27T12:35:24Z")

</div>

Thanks @aplavin

we will try LocalRegistry.jl, thanks for the tip

100% agree on reproducibility. We use docker images for the milestone analysis for that reason. Can see why this is even a better way.

best, jack

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [June 28, 2022, 5:35am UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/8 "2022-06-28T05:35:44Z")

</div>

I think you should strongly consider using packages. The reason is that a significant part of the Julia precompilation cache is dependent on packages and their UUIDs. Effectively the package is the unit of precompilation for Julia. This compilation cache lives within `.julia/compiled`. Other configuration items live within `.julia` as well. Some platform specific code lives within `.julia/artifacts`.

Package source does not need to live within the Julia depot or `.julia/dev` specifically. A proper [Project.toml and Manifest.toml](https://pkgdocs.julialang.org/v1/toml-files/) can refer to packages anywhere on the system by using [`Pkg.add`](https://pkgdocs.julialang.org/v1/managing-packages/#Adding-packages). Given a Project.toml and Manifest.toml you should be able to reproduce a Julia environment. A local registry is also a good idea, but I think it would be mainly a convenience.

The general documentation on the Julia package manager can be found in the Pkg.jl manual.

[https://pkgdocs.julialang.org/v1/](https://pkgdocs.julialang.org/v1/)

A description of the contents of the Julia depot can be found with the documentation for the Julia variable `DEPOT_PATH`.

[https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT\_PATH](https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT_PATH)

The relationship between code loading, packages, and environments is described below.

[https://docs.julialang.org/en/v1/manual/code-loading/#code-loading](https://docs.julialang.org/en/v1/manual/code-loading/#code-loading)

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 28, 2022, 9:23am UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/9 "2022-06-28T09:23:03Z")

</div>

Thanks @mkitti

I fully agree with all you said, but lost the battle with my colleagues on this yesterday. I hope its OK if I tell you why.

It has to do with our lack of understanding of Julia packages, but also how we do versions and backup of code. All code (and data and analysis) lives on several ZFS with frequent snapshots to our remote disk and [rsync.net](http://rsync.net). We have one git repo for each part of our code. No need to use githup or gitlab or some git server, all is local. This also means we never push/pull. All we do is commit. We are fully aware of the pros and cons of this setup but it works for us. And a big advantage is simplicity.

Also, we could not care less about precompilation speed. Our code runs from crontab and can take hours or days. Startup speed is irrelevant.

So the julia page on packages is [5. Creating Packages · Pkg.jl](https://pkgdocs.julialang.org/v1/creating-packages/) We had made packages using what it calls " the minimal `pkg> generate` functionality" which is simple and works well, but its recommendation “We recommend that you use PkgTemplates” seems to require githup/gitlab or some repository with push/pull, it does not work with a local git repository that does nor need push/pull.

LocalRegistry.jl is not compatible with our setup either, it needs git push/pull

So, the reason my team rejected using packages is because of how they depend on git being used in a way that is both complicated and incompatible with our very simple setup.

It is quite possible we are wrong, but I was told in no uncertain terms by my colleagues yesterday that they had no interest in spending hours understanding the package system to solve our problem when using only modules work.

best, Jack

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [June 28, 2022, 2:22pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/10 "2022-06-28T14:22:41Z")

</div>

I’m not sure what else to say other than good luck. My concern is not about your colleague’s understanding of proper code organization. My concern is there may be a fundamental misunderstanding of how Julia works.

How are you loading modules? Are you invoking `using SomePackage` or `import AnotherPackage`? or… are you just doing `include("somepackage.jl")`?

> [@JackStrauss](#):
>
> We had made packages using what it calls " the minimal `pkg> generate` functionality" which is simple and works well

OK, so you are actually using packages. That at least adds `uuid` and `version` fields to your Project.toml for each package.

# PkgTemplates

> [@JackStrauss](#):
>
> “We recommend that you use PkgTemplates” seems to require githup/gitlab or some repository with push/pull, it does not work with a local git repository that does nor need push/pull.

This is false. The entire configuration is entirely customizable. You can create a template exactly once and reuse it.

```julia-auto
julia> using PkgTemplates

julia> template = Template(interactive=true)
Template keywords to customize:
[press: d=done, a=all, n=none]
   [] user
   [] authors
   [] dir
   [] host
   [] julia
 > [X] plugins

```

You can activate or deactivate git, GitHub, or GitLab.

```julia
Select plugins:
[press: d=done, a=all, n=none]
   [X] CompatHelper
   [X] ProjectFile
   [X] SrcDir
   [] Git
   [X] License
   [X] Readme
   [X] Tests
 > [X] TagBot
   [] AppVeyor
   [] BlueStyleBadge
   [] CirrusCI
   [] Citation
   [] Codecov
   [] ColPracBadge
   [] Coveralls
   [] Develop
   [] Documenter
   [] DroneCI
   [] GitHubActions
   [] GitLabCI
   [] PkgEvalBadge
   [] RegisterAction
   [] TravisCI
CompatHelper keywords to customize:
[press: d=done, a=all, n=none]
 > [] cron
   [] destination
   [] file
ProjectFile keywords to customize:
[press: d=done, a=all, n=none]
 > [] version
   [] None
SrcDir keywords to customize:
[press: d=done, a=all, n=none]
 > [] destination
   [] file
License keywords to customize:
[press: d=done, a=all, n=none]
 > [] destination
   [] name
   [] path
Readme keywords to customize:
[press: d=done, a=all, n=none]
 > [] badge_off
   [] badge_order
   [] destination
   [] file
   [] inline_badges
Tests keywords to customize:
[press: d=done, a=all, n=none]
 > [] file
   [] project
TagBot keywords to customize:
[press: d=done, a=all, n=none]
 > [] branches
   [] changelog
   [] changelog_ignore
   [] destination
   [] dispatch
   [] dispatch_delay
   [] file
   [] gpg
   [] gpg_password
   [] registry
   [] ssh
   [] ssh_password
   [] token
   [] trigger

```

# Pkg.develop

If seems that you are using `git` at some point, so that should make things easier. However, you can use packages without using `git` at all by using [`Pkg.develop` or `]dev`](https://pkgdocs.julialang.org/v1/managing-packages/#developing) with a local path.

```julia
(SomeJuliaPkg) pkg> generate YetAnotherJuliaPackage
  Generating project YetAnotherJuliaPackage:
    YetAnotherJuliaPackage/Project.toml
    YetAnotherJuliaPackage/src/YetAnotherJuliaPackage.jl

(SomeJuliaPkg) pkg> dev ./YetAnotherJuliaPackage
   Resolving package versions...
    Updating `~/SomeJuliaPkg/Project.toml`
  [ff33dcb1] + YetAnotherJuliaPackage v0.1.0 `YetAnotherJuliaPackage`
    Updating `~/SomeJuliaPkg/Manifest.toml`
  [ff33dcb1] + YetAnotherJuliaPackage v0.1.0 `YetAnotherJuliaPackage`

julia> using YetAnotherJuliaPackage
[Info: Precompiling YetAnotherJuliaPackage [ff33dcb1-3271-4990-a66c-63e68f98716f]

```

For example, I actually recommend using the `develop` mode rather than the `add` mode in your case so that the state of the code on disk is what will be loaded when invoking `using` or `import`.

In a subsequent julia session, notice that no precompilation occurs:

```julia
julia> using YetAnotherJuliaPackage

```

If before loading the package, a source file has changed, then precompilation will occur if you are using `Pkg.develop` or `]dev`:

```julia-auto
julia> Base.Filesystem.touch("YetAnotherJuliaPackage/src/YetAnotherJuliaPackage.jl")
"YetAnotherJuliaPackage/src/YetAnotherJuliaPackage.jl"

julia> using YetAnotherJuliaPackage
[Info: Precompiling YetAnotherJuliaPackage [ff33dcb1-3271-4990-a66c-63e68f98716f]

```

Also after this precompilation has been done, there will be a `.ji` file in `.julia/compiled/v1.7/YetAnotherJuliaPackage` containing type inferred code cached during the precompilation step. I believe the Julia devs were smart enough to make the `.ji` files platform independent. Yet there are still some configuration issues between platforms that I would still recommend using a distinct `JULIA_DEPOT_PATH` environment variable for each platform.

```julia-auto
julia> readdir(joinpath(DEPOT_PATH[1], "compiled", "v1.7", "YetAnotherJuliaPackage"))
1-element Vector{String}:
 "92CAK_9xKaS.ji"

```

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 28, 2022, 2:58pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/11 "2022-06-28T14:58:16Z")

</div>

Thanks @mkitti for the patience, much appreciated.

> [@mkitti](#):
>
> I’m not sure what else to say other than good luck. My concern is not about your colleague’s understanding of proper code organization. My concern is there may be a fundamental misunderstanding of how Julia works.
> 
> How are you loading modules? Are you invoking `using SomePackage` or `import AnotherPackage`? or… are you just doing `include("somepackage.jl")`?

[/quote]

You are spot on correct here, we are all scientists not programmers and I would not be surprised if we had a fundamental misunderstanding of how Julia works. While Julia is a much better language than what we used before, we found the package documentation hard to navigate, or at least hard to see the benefit.

We used for packages (using the Pkg.develop)  
Pkg.activate(“code/package/”)  
using package

Works well on a single machine. The problem we had here was how to deal with packages and code being on a shared folder across heterogenous systems. My question at the start of this discussion was how to solve that problem.

and with modules  
push!(LOAD\_PATH,“code/package/”)  
using package

And the last just works. zero issues, dead simple.

best, jack

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [June 28, 2022, 3:03pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/12 "2022-06-28T15:03:57Z")

</div>

> [@JackStrauss](#):
>
> and with modules  
> push!(LOAD\_PATH,“code/package/”)  
> using package

Try `]dev code/package`. This will persist the path in thr Manifest.toml so you can skip the `LOAD_PATH` manipulation everytime.

---

<div class="post-metadata">

**Author:** ![JackStrauss](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@JackStrauss](https://discourse.julialang.org/u/JackStrauss)\
**Post date:** [June 28, 2022, 3:12pm UTC](https://discourse.julialang.org/t/best-practice-for-packages-on-shared-drives/83374/13 "2022-06-28T15:12:10Z")

</div>

Thanks @mkitti

Will do.

Thanks for the patience. And having read thought all the fantastic comments above, I’ve learned a lot! and will make a determined efforts to understand the details of packages, and hopefully put into use in our project.

best, jack