I have a collegue that works with Python and Snakemake , telling me how nice is this tool for scientific workflow management, but I haven’t understood much about it, and above all if what it provides is really needed much more in a python world rather than in a Julia one (for example, reproducibility…

As I understand DrWatson is a set of tools for setting up and easily do some commons operations in a reproducible way, but it’s not a workflow system. In snakemake, nextflow, etc or the dagger example I posted, one of the goal is to build a graph that capture the dependencies between the inputs file…

In the DrWatson paper we do discuss alternatives in other languages, but unfortunately snakemake isn’t there. Maybe it is a recent development? Would love to hear a summary from someone that have used it.

Not so new… [image] I would love too :slight_smile: :slight_smile: :slight_smile:

I was checking it out now. The framework focuses on reproducible (and cross-platform?) data analysis pipelines. DrWatson isn’t really about data management / analysis which may be the reason we didn’t compare in the paper? I don’t remember it was many years ago!

I believe @tecosaur 's DataToolkit.jl would be the Julia competitor in this space!

Funnily enough there was a post 2h before this thread about basically the same topic by @jonathanBieler [image] Dagger + Dates = snakemake? General Usage I have a folder data with samples (csv files), I want to compute some statistics for each file and then make a summary.…

[image] mrufsvold: I believe @tecosaur 's DataToolkit.jl would be the Julia competitor in this space! Thanks for the shoutout! :heart_eyes: [image] jonathanBieler: DataToolkit seems to be doing some of that, but it seems more tailored to managing/ingesting the raw data that being a full …

[image] jonathanBieler: one of the goal is to build a graph that capture the dependencies between the inputs files and the results you want to compute, and to execute that workflow automatically in parallel, with the ability to resume, or update only the part that needs to be updated. Just fo…

I think everyone interested in any of this might also be interested in showyourwork , which uses snakemake, but does a lot more. @MilesCranmer had a very nice series of posts starting here discussing this, comparing to other tools (including the excellent DrWatson, which I particularly love!), and e…

Snakemake with julia (and similarities with Dr Watson or..)

General Usage

abraemer February 20, 2025, 6:28pm 6

Funnily enough there was a post 2h before this thread about basically the same topic by @jonathanBieler

Topic		Replies	Views
Workfow for data analysis project: avoid re-running with many inputs, intermediate steps, and models New to Julia workflow	8	1039	October 8, 2020
Dagger + Dates = snakemake? General Usage data , speculative , design	3	431	February 25, 2025
[RFC] Mr Phelps - a distributed workflow orchestrator Package Announcements	42	6194	August 4, 2021
Is there a Julia package similar to the Python's "ShowYourWork"? VS Code	17	2466	May 1, 2023
Suggestions for best practice in scaling analysis Julia at Scale	3	486	June 16, 2024

Snakemake with julia (and similarities with Dr Watson or..)

Related topics