Why is Python, not Julia, still used for most state-of-the-art AI research?

[Palli]

@ToucheSir

Distributed and multi-gpu training: this has unfortunately become a must for certain streams of research in CV, NLP and deep RL. CliMA is the only public project I know of that has distributed GPU support, but it’s not DL and the only way to do something similar for DL right now is to implement your own framework from scratch on top of MPI

Hmm, yes, multi-GPU was a blind spot to me, I see it done as far back as 2016, but not specific to ANNs/DL:

Multiple-GPU Parallelism on the HPC with Julia - Stochastic Lifestyle

How to use multiple GPUs correctly?

Hi all, I am trying to change my GPU code to use multiple GPUs due to the memory limit of a single GPU. I found an example (Multiple-GPU Parallelism on the HPC with Julia | juliabloggers.com) and the basic idea is split the whole data into different parts. store each part of the data using CuArrays in different GPU cards. launch the kernels asynchronously in different GPUs with local data. So I have done a test with the following code using CuArrays, CUDAnative using BenchmarkTools N = 2 …

@ToucheSir

or resuscitate NCCL.jl

Is it for sure dead? This one or at JuliaGPU (both updated recently)? GitHub - vchuravy/NCCL.jl: A Julia wrapper for the NVIDIA Collective Communications Library.

There’s interesting work being done to scale NNs down not just up (as with GPT-3), both for NLP and computer vision. Still, GPT-3 is huge (ALBERTA I mentioned much smaller), so multi-GPU seems needed for sure (at least for good NLP now).

I’m curious, if the network itself doesn’t need to be that big (say fits on one memory), but the problem is the dataset/training, what happens if you spit it 2 or N ways and train independently, can you in general (or say for images only) combine two such trained networks? Isn’t that what people call minibatching? I could see it maybe not working for NLP.

And can you simply use:

GitHub - horovod/horovod: Distributed training framework for TensorFlow,...

Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet

E.g. Julia has by now (for a long time, while it’s not the post popular framework, for Julia or otherwise) official support for MXNet, and as I posted, there’s a PyTourch wrapper, while the Tensorflow one is a bit outdated.