Where does Julia provide the biggest benefits over other ML frameworks for research?

Specific Domains Machine Learning
34

My personal list of issues with ML in PyTorch looks like this:

  1. In 30% of cases code just works.
  2. In other 20% of cases I forget to call zero_grad(), item(), detach() or to(device). I wish there were a more high-level API like in Keras (I’m looking at skorch right now), but such issues are usually resolved within 10-15 minutes.
  3. 10% of issues are caused by poor choice of optimizer and learning rate. Sometimes it takes half an hour to make my loss to start going down.
  4. 20% are caused by numeric issues like vanishing gradient, division by nearly zero, etc.
  5. In the other 20% of cases it’s just my mistake.

I never had to debug building blocks of PyTorch, only concrete models created using it. And, honestly, PyTorch is much better in this regard since it has a lot more implemented (and sometimes pretrained) models than any framework in Julia.

But training a good ML model is just a part of the workflow. Usually my job includes:

  1. Data analysis. Normally I use SQL + Pandas (Python), but Pandas only helps with a sample since whole dataset doesn’t fit into memory.
  2. Data preparation for ML. Usually I have to use SQL or Spark with Scala (PySpark has poorer support on big clusters like Amazon EMR), but some people point to Python’s Dask.
  3. Other data utilities. For example, today I implemented custom KD tree for data anonymization purposes. I used Python since in Java/Scala it would be much harder to do, but performance is quite poor.
  4. Exposing trained models to production via HTTP. Normally we use Java, Scala, Go or at best Python frameworks over C servers, but never pure Python servers.

We have to use this zoo of technologies since none of them alone supports all our needs and it’s unlikely they ever will. Julia at its current state also doesn’t solve many problems, but at least - and this is the main reason I continue to invest into it - Julia has a chance to become a one-stop solution. And the best thing is that contributing to these tools is much, much easier in Julia than in Scala or C backend of Python libraries.

16 Likes