# Planing on building an ETL Framework

**URL:** <https://discourse.julialang.org/t/planing-on-building-an-etl-framework/124747>\
**Category:** Data\
**Tags:** data, etl\
**Created:** [January 13, 2025, 11:01pm UTC](https://discourse.julialang.org/t/planing-on-building-an-etl-framework/124747 "2025-01-13T23:01:31Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![p-gw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/p-gw/32/210518_2.png) [@p-gw](https://discourse.julialang.org/u/p-gw)\
**Post date:** [January 14, 2025, 1:01pm UTC](https://discourse.julialang.org/t/planing-on-building-an-etl-framework/124747/2 "2025-01-14T13:01:41Z")

</div>

I tried something similar a while ago when I needed an automation solution for work. I started to build very basic custom pipelines reading data from databases and parquet files (DuckDB.jl), cleaning and aggregating data (DataFrames.jl) and saving the results to parquet files.

I can’t find the threads now, but I remember there were some initial tries to program a framework like this. However, I think they ended up abandoned.

While I found julia quite suitable for the task, in my project I discovered myself basically reinventing [dlt](https://github.com/dlt-hub/dlt) and [dbt](https://www.getdbt.com/), so I just ended up using these existing solutions instead.

---

_[View the full topic](https://discourse.julialang.org/t/planing-on-building-an-etl-framework/124747)._