# Announcing Whisper.jl

**URL:** <https://discourse.julialang.org/t/announcing-whisper-jl/96614>\
**Category:** Package Announcements\
**Tags:** machine-learning, audio, speech-recognition\
**Created:** [March 26, 2023, 12:05am UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614 "2023-03-26T00:05:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![avik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avik/32/17_2.png) [@avik](https://discourse.julialang.org/u/avik)\
**Post date:** [March 26, 2023, 12:05am UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/1 "2023-03-26T00:05:34Z")

</div>

I’m very pleased to announce the availability of [Whisper.jl](https://github.com/aviks/Whisper.jl), a Julia package to perform speech recognition. It uses the Whisper model developed by OpenAI, and runs the inference on the CPU using [Georgi Gerganov](https://github.com/ggerganov)’s [whisper.cpp](https://github.com/ggerganov/whisper.cpp). That C library is provided as a jll, and the model weights are downloaded on demand.

Currently, a `transcribe` function is exposed that takes raw audio data, and produces a text transcription. Suggestions, or contributions, for other kinds of interfaces will be gratefully received. The whisper.cpp low level functions are, of course, already available. But a high level Julia streaming interface does need to be added – hopefully soon.

The package is [awaiting registration](https://github.com/JuliaRegistries/General/pull/80328) in the General Registry. Please try it out and let me know how it goes.

Regards

Avik

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [March 26, 2023, 5:00am UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/2 "2023-03-26T05:00:34Z")

</div>

Cool! If you need it tested on a Scottish accent let me know…

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [March 26, 2023, 5:36pm UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/3 "2023-03-26T17:36:38Z")

</div>

Great! I see for:

> **[GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak...](https://github.com/openai/whisper)**
>
> Robust Speech Recognition via Large-Scale Weak Supervision - GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision

> There are five model sizes, four with English-only versions.

What languages/model do you support (`large-v2` model?), e.g. my native Icelandic? I see best word-error rate (WER) 3.2% for Spanish, then Italian, then English at 4.2%, and way down Icelandic at 38.2%, and Nepali last.

State-of-the-art is however now:

> **[Conformer-1: a robust speech recognition model](https://www.assemblyai.com/blog/conformer-1/)**
>
> We’ve trained a Conformer speech recognition model on 650K hours of audio data. The new model, Conformer-1, approaches human-level performance for speech recognition, reaching a new state-of-the-art on real-world audio data.

Makes “43% fewer errors on noisy data on average”, and 3.3% WER (for English presumably on non-noisy, “human transcriptionists” get 4%; Conformer-1 gets 9.9% on noisy data):

[![](https://global.discourse-cdn.com/julialang/original/3X/9/a/9a816e8dee921ddd0338c4e648e81d981fa4c0fc.jpeg "How Well Can GPT-4 See? And the 5 Upgrades That Are Next") ](https://www.youtube.com/watch?v=FceQxb96GO8&t=295)

---

<div class="post-metadata">

**Author:** ![zacharyneveu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zacharyneveu/32/34771_2.png) [@zacharyneveu](https://discourse.julialang.org/u/zacharyneveu)\
**Post date:** [April 6, 2023, 7:01pm UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/4 "2023-04-06T19:01:15Z")

</div>

Very cool! Any plans to bring the model inference code into Julia natively? Reading through the whisper.cpp code it’s mainly just in two C++ source files and most of it is doing stuff that would be easier/cleaner in Julia (i.e. there are currently separate functions for different input data types which could be moved to multiple dispatch).

---

<div class="post-metadata">

**Author:** ![avik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avik/32/17_2.png) [@avik](https://discourse.julialang.org/u/avik)\
**Post date:** [April 6, 2023, 7:24pm UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/5 "2023-04-06T19:24:29Z")

</div>

That would indeed be very cool, and I think is quite feasible. However, that is beyond the limits of my time and skills currently. I hope someone does this, and I’ll be happy to retire the current ccall based codebase.

---

<div class="post-metadata">

**Author:** ![CiPa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cipa/32/38253_2.png) [@CiPa](https://discourse.julialang.org/u/CiPa)\
**Post date:** [May 23, 2023, 3:57pm UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/6 "2023-05-23T15:57:52Z")

</div>

Hi @avik! Sharing a Genie app we built with Whisper.jl. Have a look!  
 ![whisper genie app - genie cloud](https://global.discourse-cdn.com/julialang/original/3X/3/c/3c436e3d3e21ce2b045ad70ac89c74173a3ba744.gif)

The code is available here: [https://github.com/GenieFramework/GenieFrameworkDemos/tree/main/Whisper](https://github.com/GenieFramework/GenieFrameworkDemos/tree/main/Whisper)

---

<div class="post-metadata">

**Author:** ![avik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avik/32/17_2.png) [@avik](https://discourse.julialang.org/u/avik)\
**Post date:** [May 23, 2023, 5:17pm UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/7 "2023-05-23T17:17:01Z")

</div>

Cool! Thanks for sharing.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [May 23, 2023, 6:10pm UTC](https://discourse.julialang.org/t/announcing-whisper-jl/96614/8 "2023-05-23T18:10:57Z")

</div>

Just gave it a try and it works quite well.  
I was surprised that the result was automatically translated into an english result from german reading. So I tried to figure out how to change some parameters perhaps to get a german text, but failed to find a solution. It’s not so easy by just looking into the C interface. Some hints or explanations on this would be helpfull. Not too deep but a bit to find a starting point.