My apologies for the generic question. Unfortunately after searching around I haven’t been able to find a direct answer. I would like to know the communities opinion on using Julia for Data Engineering tasks. You know pipelines, streaming data into a warehouse, etc. I’ve built many pipelines and…

I don’t have experience making an actual data pipeline in Julia (since I work on a larger team I usually play safe and go to languages such as Python and Scala), but from what I’ve seen from Julia right now it kinda isn’t unless you need to do some very customized and math oriented logic where Julia…

In the abstract, I think Julia is an excellent language for this use. It’s integration with the shell and excellent reproducibility of environments, to say nothing of the code for actually manipulating data, make it ideal. But unfortunately, I have to agree with @fborda that the libraries aren’t th…

I too have the same feeling but it’s unclear which areas exactly are lacking. It would be nice if people doing data engineering in other languages/ecosystems can write up a list here and prioritize it so the community can tackle them in a more coordinated fashion. I can volunteer to put together an…

The problem is that even when a tool is available, it’s too barebones. For example, if you want to use Kafka, there is RDKafka.jl , which technically cover the bare minimum (the ability to read and write to kafka), but for a proper data pipeline you need a lot more than the minimum. You need ways to …

[image] fborda: The problem is that even when a tool is available, it’s too barebones. For example, if you want to use Kafka, there is RDKafka.jl , which technically cover the bare minimum (the ability to read and write to kafka), but for a proper data pipeline you need a lot more than the minim…

@tk3369 I finally got around to exploring this more and wrote up some thoughts here: Data Engineering in Julia. Work with massive datasets to design… | by Logan Kilpatrick | Towards Data Science TLDR; Julia can do a lot and is a natural choice for DE, there are still sharp edges but it doesn’t appe…

Oh @logankilpatrick ! One thing that I’d also suggest you highlight is a great project that has been a game changer for me called FunSQL.jl . Basically you can write a query in a Julia Domain Specific Language, use all the niceties of Julia, and then convert your query to nearly any type of SQL fla…

That’s a great writeup! Thanks, i really enjoyed reading it. When it comes to data engineering i feel that Kedro Kedro: A New Tool For Data Science | by Jo Stichbury | Towards Data Science is really something to aspire to. Seems like a data engineers dream. I would love to see something like this m…

At my last job, I had a very successful project that built a distributed data pipeline. The entire process is written in Julia and runs on Kubernetes. I certainly think that Julia is the right tool for data engineering jobs.

Is Julia a good choice for Data Engineering?

General Usage

kevbonham May 4, 2020, 11:58am 5

This thread aggregated a bunch of things.

Topic		Replies	Views
Recent experience with Julia as the main data science driver General Usage	18	3733	August 8, 2021
What's the current (spring 2024) canonical approach to data science in Julia? General Usage dataframes	34	4729	April 8, 2024
Why is Julia so great? New to Julia	77	11742	April 16, 2023
JuliaDB, dataframes: Speculations over the future of data packages Data	24	7590	August 21, 2020
What's the latest and greatest in data in Julia Data	29	2476	August 15, 2024

Is Julia a good choice for Data Engineering?

Related topics