# \[ANN\]: MetidaFlows.jl - experimental workflow engine project for Julia

**URL:** <https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298>\
**Category:** Package Announcements\
**Created:** [May 27, 2026, 4:11pm UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298 "2026-05-27T16:11:57Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [May 27, 2026, 4:11pm UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/1 "2026-05-27T16:11:57Z")

</div>

Hi everyone!

I’d like to share an early-stage project called **MetidaFlows.jl**.

The package is currently in active development and should be considered **experimental**.  
The goal right now is to explore workflow-engine design patterns in pure Julia and collect feedback from the community.

## What is MetidaFlows.jl?

MetidaFlows.jl is a lightweight graph-based workflow engine prototype for building:

- data-processing pipelines
- DAG-based computations
- agent/event-driven workflows
- typed node graphs with validation

It is designed to stay minimal, explicit, and fully Julia-native (no external runtime).

## Current functionality

The package already implements:

- typed input/output ports
- graph-based workflow model
- connection validation with type checking
- DAG topological scheduler (`DAW` )
- queue-based scheduler (`ABW` , experimental)
- execution invalidation propagation
- node execution state tracking
- validation hooks (settings / result / structure)
- incremental graph modification
- simple serialization helpers

## Execution model

Nodes are defined via multiple dispatch:

```julia-auto
function MetidaFlows.execute_unsafe!(node::DataNode{MyNode})
    ...
end

```

## Workflow modes

### DAW — Data Analysis Workflow

Deterministic execution using topological sorting of a DAG.

### ABW — Agent-Based Workflow

Queue-based execution model for dynamic/reactive workflows (still experimental).

## Real working example

This example is taken directly from the test suite and actually runs:

```julia-auto
using MetidaFlows
using CSV, DataFrames

struct CSVNode <: AbstractNodeType end
struct DataFrameNode <: AbstractNodeType end

csv_spec = NodeSpec(
    "Load CSV",
    PortSpec[],
    [PortSpec("CSV File", CSV.File, :csv)],
    [:file]
)

df_spec = NodeSpec(
    "DataFrame",
    [PortSpec("CSV File", CSV.File, :csv)],
    [PortSpec("DataFrame", DataFrame, :dataframe)]
)

function MetidaFlows.execute_unsafe!(node::DataNode{CSVNode})
    csv = CSV.File(node.settings[:file])
    setdata!(node, :csv, csv)
    return [:csv]
end

function MetidaFlows.execute_unsafe!(node::DataNode{DataFrameNode})
    csv = getinputdata(node, :csv)
    setdata!(node, :dataframe, DataFrame(csv))
    return [:dataframe]
end

workflow = Workflow(0)

id1 = add_node!(workflow, DataNode(CSVNode, csv_spec))
id2 = add_node!(workflow, DataNode(DataFrameNode, df_spec))

add_connection!(workflow, id1, :csv, id2, :csv)

setsettings!(workflow, id1, Dict(:file => "data.csv"))

scheduler!(workflow)

df = getdata(workflow, id2, :dataframe)

```

## Current status

This is **not a production-ready package**.

Expect breaking changes while the architecture evolves. The main focus right now is:

- stabilizing execution semantics
- improving scheduler correctness
- refining invalidation model
- improving test coverage

## Roadmap ideas

- caching & checkpointing
- audit/logging system
- better serialization/export formats

## Feedback welcome

I’d especially appreciate feedback from people working with:

- DAG systems
- ETL pipelines
- scientific computing workflows
- node-based editors
- agent-based systems

GitHub:  
[MetidaFlows.jl](https://github.com/PharmCat/MetidaFlows.jl)

Documentation:  
[Documentation link](https://pharmcat.github.io/MetidaFlows.jl/dev/)

---

<div class="post-metadata">

**Author:** ![tp2750](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tp2750/32/207806_2.png) [@tp2750](https://discourse.julialang.org/u/tp2750)\
**Post date:** [May 30, 2026, 7:34pm UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/2 "2026-05-30T19:34:30Z")

</div>

Looks cool! Thanks for sharing.  
Are you planning dashboard functionality like [Apache Airflow](https://airflow.apache.org/)?

---

<div class="post-metadata">

**Author:** ![kahliburke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kahliburke/32/208472_2.png) [@kahliburke](https://discourse.julialang.org/u/kahliburke)\
**Post date:** [June 1, 2026, 8:06am UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/3 "2026-06-01T08:06:50Z")

</div>

This looks on first glance a bit like Apache Hamilton?

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [June 1, 2026, 2:02pm UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/4 "2026-06-01T14:02:56Z")

</div>

..or like [Snakemake](https://snakemake.github.io/) in Python ??

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [June 15, 2026, 1:07am UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/5 "2026-06-15T01:07:05Z")

</div>

Hi! Thank you all for the feedback and interest in the project. I’m really happy to see that it has sparked some discussion. The package was originally developed for my own work, but I thought it could also be useful to the Julia community.

> [@tp2750](#):
>
> Are you planning dashboard functionality like [Apache Airflow](https://airflow.apache.org/)?

At the moment, the package is still at a very early stage. A visual dashboard or UI is definitely an interesting direction, but it’s not a primary goal yet.

> [@kahliburke](#):
>
> This looks on first glance a bit like Apache Hamilton?

Yes, Apache Hamilton follows a similar idea of building data workflows from interconnected computational units.

> [@sylvaticus](#):
>
> ..or like [Snakemake](https://snakemake.github.io/) in Python ??

Yes, there is some overlap in the overall goal of reproducible and structured data analysis.

The project is still evolving, so feedback are suggestions are very welcome.

---

<div class="post-metadata">

**Author:** ![davide.crucitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davide.crucitti/32/218224_2.png) [@davide.crucitti](https://discourse.julialang.org/u/davide.crucitti)\
**Post date:** [June 15, 2026, 10:04am UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/6 "2026-06-15T10:04:06Z")

</div>

I don’t know whether you know [row](https://github.com/glotzerlab/row), it is very useful in HPC since it does not depend on a service running: you submit it and it executes all available actions. This is better on an HPC as you can schedule a cron or something like that rather than trying to seup a long running job which may stop at any time.  
Moreover, it does not touch the actual files, allowing a job to manage its own files autonomously. For example SnakeMake will delete results if the calculation did not complete, this is not always desirable. You can set a calculation to restart from a checkpoint and would not want to lose days of computation.

Overall, this is an approach I prefer when you have to conduct long running and compute intensive calculations. While other managers are better when handling many short running calculations.

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [July 29, 2026, 10:26pm UTC](https://discourse.julialang.org/t/ann-metidaflows-jl-experimental-workflow-engine-project-for-julia/137298/7 "2026-07-29T22:26:55Z")

</div>

Add documentation, cycles and “multi-port” support. Bugfixes. Version 0.1.0 ready for registry.