Automatic Speech Recognition in Julia

[jbrea]

Does anyone have (or work on) an implementation of modern automatic speech recognition tools like conformers or wav2vec2.0 in one of Julia’s deep learning frameworks Flux/Knet/Avalon? Ideally pretrained on LibriSpeech . I couldn’t find anything here or on github.

[Palli]

I don’t find anything either in Julia. Why do you need it implemented (fully) in Julia? Isn’t it good enough to call say PyTorch from Julia and use whatever is available (or Avalon.jl then helpful)? I’m not up-to-speed on moving models from Python to Julia, i.e. just the parameters, weights and biases, shouldn’t that be possible, and wasn’t there even a standard for it ONNX? Might likely just work for certain types of networks, e.g. I believe it’s an older standard than Transformers, so those excluded?

What I did find however brand-new from 31 March 2022:

Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired...

Comprehensive experiments on the LibriSpeech corpus show that the proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training, and also outperforms significantly the state-of-the-art wav2vec 2.0 and HuBERT on finetuning subsets of 10h and 100h

“Wav2Vec 2.0” was state-of-the-art in 2020, according to its paper 2020 paper, is it still so, even though this other Feb 2022, states so (or is it just an evaluation/survay paper, and they tend to repeat claims?):

If someone DOES want to reimplement something in Julia, I at least would want them to find the state-of-the-art and use that…

Wav2Vec2.0 on the Edge: Performance Evaluation

Might be a helpful thread:

Speech-based Emotion Recognition

Hello, I wasn’t able to find any package for Speech-based Mood/Emotion recognition to use in a project I’m working on, so I need some help setting up something myself. I know that LSTM’s are in use for this, I’ve seen WaveNet adapted for almost every other audio problem at this point and I have even seen some of the slightly outdated Spatio-Temporal Box Filters. I guess what I really want to ask is, what would be a good place to start from? Not necessarily the best, most accurate or even fast…

SincNet was also intriguing when I noticed it (might be outdated, or not, hadn’t heard of SpeechBrain):

GitHub - mravanelli/SincNet: SincNet is a neural architecture for efficiently...

SincNet is implemented in the SpeechBrain (https://speechbrain.github.io/) project as well.

Sinc as a Neural Networks Activation Function - Sefik Ilkin Serengil

sinc (and sin) looked intriguing for periodic functions, but may actually be outdated. SIREN is if I recall newer and better, and even something more recent, even better (applications I saw however for computer vision).

I hadn’t heard of conformers (thanks for the tip), only transformers, which it’s a variant of, but might also be too old:

GitHub - sooftware/conformer: [Unofficial] PyTorch implementation of...

[jbrea]

Thanks!

@Palli

Why do you need it implemented (fully) in Julia?

I was thinking about backproping through h(\mathrm{asr}(g_\theta(x))) where h and g_\theta are functions written in julia, \mathrm{asr} is some automatic speech recognition network and \theta are the parameters to be adjusted. But you’re right, maybe I should just pass the gradient of h to pytorch, compute the pytorch gradient of \mathrm{asr} with respect to its input and pass the result to g_\theta. Having everything in julia just feels a bit easier .