Request for advice on workflow in distributed team of scientists

General Usage
1

Hello
I am sorry for the stupid question. I work in a group of scientists in many places. All our work is organised in dropbox, using mac, linux and windows. We do science, code is a necessity, and minimise complexity. This means git is too complicated. We use R because of RStudio, and python does not work because of endless problems with breaking upgrades, venv complexity, and .venv will break in this dropbox environment.

in steps Julia, better language and faster. 2 downsides, no RStudio (vscode is a very poor substitute), but that can be overcome, and precompile speed. When we load out common code, it takes a minute to precompile. colleagues not happy.

Got that solved for one person by a precompiled local package in the dropbox but then oops, it fails in this heterogeneous multi os dropbox environment, endless package problems and eventually having to delete .julia.

So question is, are we missing a trick with packages that only live inside dropbox and people have linux, mac, windows? githup is a non-starter.

Or can we precompile with modules only?

Sorry for the very basic and stupid question, we did read the documentation and google, but we are not computer scientists.

best Davide

3 Likes
2

More insightful people are likely going to comment after me, but having experienced similar struggles to you, I’ve found my peace with it for at least these two reasons:

  1. Julia is an attempt at middle ground as a compiled interactive languages. This means that, like lower level languages, you get some compile time, but the performance that comes with that. But also, like interactive languages, you get the high level syntax and interactivity. This trade-off middle ground experience isn’t too much of a surprise to me when I think of things that way.

  2. All Julia developers recognise your plight, especially the core developers, and in the continuing, persistent, and patient spirit of their initial proclamation on Why They Made Julia, they are also working on tools to improve both the compilation and interactive experiences. In my slowly progressive learning over the 7+ish years I’ve been learning Julia, I keep learning new things about it, and the developments over the years have been exciting.

In the context of your work, it sounds like you and your colleagues may appreciate enterprise help via JuliaHub.com where they’ll guide you through best practices and help manage anything that can go wrong. People here on Discourse are generally happy to help when available as well, so it’s awesome that you asked here instead of calling it quits - it’s not a stupid question, if it is then I ask them all the time.

Regarding suggestions specific to your challenges, I know of things like PrecompileTools.jl and having a LocalRegistry.jl that may help you. I don’t have much experience with Julia in the workplace shared among colleagues so hopefully someone better in the know will respond after me.

4 Likes
3

I would say, do not put source code on Dropbox. This causes a lot of problems and is not worth the trouble. But then again, their might be workarounds I am not aware of.

6 Likes
4

Thanks @kapple, good to hear we are not alone. Your middle ground comment strongly resonates. More difficult to use than R, less than c. We really like Julia and could see it be THE language in our work. For some collaborators these are trivial issues, but for others, insurmountable. And the lowest common denominator wins.

Great idea on JuliaHub and professional help, did not know it existed, I will see if one of us has the funding.

Thanks again for your insights.

best Davide

1 Like
5

Thanks @ufechner7. Fully agree on dropbox. But, when my Uni tried to ban dropbox and put all on OneDrive, the rebellion was so strong they had to back down. In my world, research runs on dropbox.

We have discussed alternatives to dropbox (I was put on the Uni committee), and every single one has such strong downsides as to prevent adoption.

If someone says githup, well, it put a complexity layer between researchers and their work. Some just refuse to touch it. I personally use it for projects with students, but its impossible for joint research outside of that.

So, given all the imperfections, its dropbox, and we are looking for using Julia in our workflow. And it has to meet the lowest common denominator objection.

I think that is why R and Rstudio continue to thrive. Its the lowest common denominator.

Thanks again

Davide

2 Likes
6

I will not help by providing an answer to the question, but I think that I can help by advising to change the title a bit. First, there are not stupid questions here, and yours is not an exception – in fact, it is very reasonable. Second, if you make the title a bit more descriptive, you may increase your chances of attracting those who can help. I suggest changing the title to something like “Request for advice on workflow in distributed team of scientists” or something like that. Even this would help passers-by recognise what is inside. Good luck.

4 Likes
7

Wow @zdenek_hurak that makes perfect sense. Took your advice and renamed it!

Thanks
Davide

1 Like
8

Regarding R-like syntax and functionalities, check out Tidier.jl

Regarding the minute-long compilation, what version(s) of Julia are you running, and what packages are typically in your modules?

Yes compilation can take a minute or so, but from then on Julia jormally doesn’t precompile things unless it changes. So the subsequent loads shouldn’t take as long.

It’s typical in a Julia programming session to keep your REPL session open for as long as possible using Revise.jl, which precompiles called code if it changes. Do you have that in your workflows? You get everyone to add it to their default Julia environment, and add it to their .julia/config/startup.jl.

An additional suggestion is to work on Pluto.jl notebooks. They are standalone notebooks comparable to Jupyter notebooks, and you can share them in dropbox without issues. The core developers of Pluto recently released functionality of developing modules/packages in the Pluto.jl development environment (which is just a browser). I haven’t tried it, but it might be worth looking into for you.

If you have trouble understanding any of the above, let us know. I’m happy to elaborate on them.

3 Likes
9

Thanks @kapple

Yes, Tidier.jl is nice, it and the tidy generates a lot of fun after work debates in the pub. I’m on the dataframe (and data.table in R) side.

Yes, all the outside packages are precompiled. We use revise.jl.

Actually, I am one of the people who does NOT usually keep the REPL open. If only running stuff (99% of the time), prefer to call julia (and python and R) on the command line like

julia --threads=auto -e 'include("process.jl")'

Some colleagues have a desktop icon to run that command, I usually have it in a shell script with other stuff.

I, personally, don’t get along with local configuration like .julia/config/startup.jl. will inevitably end up using a computer I have not set up. So never customise. better for my nerves.

We use Quarto for final reports, guess similar to pluto.jl,

all the best,

Davide

1 Like
10

Okay awesome, that’s the sinple stuff I know ticked off the list.

I relate to that :grin::sweat_smile:

1 Like
11

Using non-git packages will get a lot easier in Julia 1.11 with the new [sources] section in Project.toml. For now I have the following setup:

Workspace/
├── Package1/
│   ├── Project.toml
│   └── src/
│       └── Package1.jl
├── Package2/
│   ├── Project.toml
│   └── src/
│       └── Package2.jl
└── ScriptsAndTests/
    ├── install-deps.jl
    ├── Project.toml
    ├── Manifest.toml
    ├── runtests.jl
    ├── script1.jl
    └── script2.jl

install-deps.jl is

#!/bin/env julia
using Pkg: Pkg, PackageSpec
Pkg.activate(@__DIR__)
root = relpath(dirname(@__DIR__))
Pkg.develop([
    PackageSpec(path=root*"/Package1"),
    PackageSpec(path=root*"/Package2"),
])
Pkg.instantiate()

This ensures the Manifest.toml has the relative paths to Package1 and Package2. Because the paths are relative, this should work fine with Dropbox.

5 Likes
12

Wow @nhz2, thanks for telling me, will stick to modules until 1.11.

It is truly wonderful how Julia is constantly getting better. The developers deserve so much.

all the best,
Davide

1 Like
13

Hi, I did some writeup on this:

https://j-fu.github.io/marginalia/julia/project-workflow/

Also, as already mentioned, having your own package registry managed with LocalRegistry.jl helps to leverage the Julia package manager for code distribution via git.

4 Likes
14

Hi @j-fu

That is very nice, super clear, thanks. Sharing with my group.

best, D