Comparing Julia and Python for Data Science Projects

Python is widely used for data analysis and machine learning, while Julia is designed with a strong focus on numerical computing and performance.

For projects involving large datasets, statistical modelling, or scientific computing, choosing the right language can depend on performance requirements, available libraries, and team experience.

Question: For data science and machine learning workflows, when have you found Julia more suitable than Python? Are there particular packages or use cases where Julia offers a clear advantage?

The advantages Julia has are in extensibility/composition and the ability to write high performance custom numerical routines directly in the language. If what you are doing is already covered by the package ecosystem in Python, it’s going to be more time efficient to stick with Python. If you are going to implement custom algorithms or arbitrarily compose different packages together for things like automatic differentiation, Julia wins hands down. I like how I can express things in Julia and then use them in many different contexts; I don’t ask the question “should I implement this in PyTorch, JAX, or NumPy/Numba?”. I just write the code once and re-use it everywhere.

The type of person who will appreciate Julia over Python is typically not the sort of person who just wants to load everything off the shelf and call it a day. There are many excellent packages in the ecosystem, but the sheer breadth of the Python ecosystem isn’t something Julia is ever going to match. In Julia it’s less any particular package that is the advantage (although there are quite a few that are superior to their Python counterparts), but rather that the language itself enables you to express your vision through the superior extensibility, composition, and code re-use. You can also take your Julia code and efficiently execute it across a large number of computing paradigms and accelerators.

These are just a few of the reasons I prefer to use Julia for all of my ML work.

That basically sums it up for me. With very few exceptions (maybe some of my JuMP.jl projects), all of my data science projects could be done in Python. It wouldn’t be difficult to argue that most of my data science projects probably should be done in Python, or R (since they’re used extensively across my employer). However, I am fortunate enough to have the freedom to choose, and I’m the kind of guy that has always enjoyed rolling up my sleeves, taking things apart to see how they work, and challenging myself to build things from scratch. When I cook, I buy raw ingredients and make the dish the hard way. If I want a birdhouse for my backyard, I build it. When I need a new computer, I determine the ideal specs for my needs and buy all the components individually and then assemble it myself. I roast my own coffee, (used to) brew my own beer, etc. - you get the picture. Julia was just the natural choice for me because it’s the language that best allowed me to take that same philosophical approach and apply it to programming tasks.

I actually love this response. So does that mean you code your own data cleaning/ preprocessing? Do you also code your own ML algorithms like the typical XGBoost or random forest?

I have always strived to adhere strictly to this rule: never use a technique or method that you don’t understand. So I generally use packages just like everyone else, but what I like so much about Julia is that I think it’s the best language for learning new things. The fact that most library code is written in Julia is a big part of that, but just the ergonomics and syntax of the language make it great for learning and implementing new algorithms and methods yourself. I have been lucky enough to be able to take the time to implement a lot of algorithms myself, but it’s usually just an academic exercise to make sure I understand exactly how it works and what it’s doing. Once I feel that I have a good grasp of the technique, I’m more than happy to rely on a good package!

Just as an example, I asked Claude to put together an elementary gradient boosting algorithm, and it gave me:

using Statistics

# Best single split on x, minimizing squared error of residuals r
function fit_stump(x, r)
    best = (sse = Inf, t = 0.0, left = 0.0, right = 0.0)
    for t in unique(x)
        L, R = r[x .<= t], r[x .> t]
        (isempty(L) || isempty(R)) && continue
        sse = sum(abs2, L .- mean(L)) + sum(abs2, R .- mean(R))
        sse < best.sse && (best = (; sse, t, left = mean(L), right = mean(R)))
    end
    return best
end

predict(s, x) = ifelse.(x .<= s.t, s.left, s.right)

# Squared loss: the negative gradient is just the residual
function boost(x, y; rounds = 100, η = 0.1)
    f0 = mean(y)
    F = fill(f0, length(y))
    stumps = []
    for _ in 1:rounds
        s = fit_stump(x, y .- F)
        F .+= η .* predict(s, x)
        push!(stumps, s)
    end
    return xnew -> f0 .+ η .* sum(predict(s, xnew) for s in stumps)
end

x = range(0, 2π, length = 200)
y = sin.(x) .+ 0.1 .* randn(200)
model = boost(x, y)
mean(abs2, y .- model(x))   # training MSE, close to the noise level

If I was studying this, I’d normally fire up a Pluto notebook, add in some sliders, etc., and spend a couple of hours looking at how the pieces fit together, how changing the inputs changes the outputs, etc. Then I might go over to EvoTrees.jl and browse their code for another couple of hours until I felt confident I could explain to you (at least to a reasonable degree) how the algorithm works. Only then would I actually use it in practice.

I think it’s fair to say that if you wanted to adopt this approach in Python or R, it’s a lot more difficult and you would frequently have to roll up your sleeves and dig into some C++ to get to the interesting parts of the package code…