Hey everyone,
I’ve been exploring signal processing in Julia and working on some basic DSP and communications tools. As I look into expanding toward audio signal processing, Music Information Retrieval (MIR), and feature extraction, I wanted to check in with the community regarding the current state of the ecosystem and where the most meaningful gaps lie.
Looking at the existing landscape, we have solid foundational I/O and streaming types in the JuliaAudio organization (SampledSignals.jl, PortAudio.jl, LibSndFile.jl) alongside DSP.jl. However, higher-level toolkits specifically modern native counterparts to Python’s librosa / torchaudio or C++'s Essentia for feature extraction (MFCCs, CQT/VQT, chromagrams, pitch/onset tracking) and real-time DSP nodes seem either fragmented or unmaintained.
Before committing to a larger open-source project in this space, I’d love to gather thoughts from folks working with audio in Julia:
Current Workflows: What packages or custom setups are you currently using for audio feature extraction, MIR, or real-time audio manipulation in Julia? Do you find yourself wrapping C++/Python libraries via CxxWrap or PythonCall?
Key Friction Points: What specific transforms, psychoacoustic models, or audio block abstractions do you consistently find missing or awkward to implement in pure Julia?
Julia’s Unique Advantage: Where does Julia offer a clear competitive edge over established C++ and Python tools in the audio domain (e.g., zero-allocation real-time pipelines without GC pauses, differentiable DSP via AD engines, or native GPU feature extraction)?
Strategic Focus: From a practical standpoint, do you feel there is sufficient demand in the community to justify building a dedicated, unified audio DSP/feature-extraction framework in Julia? If so, which direction would yield the highest return on effort (e.g., offline MIR & ML feature pipelines, real-time live audio processing graphs, or differentiable/GPU audio primitives)?
Looking forward to hearing your thoughts, current workarounds, and ideas!
Thanks
-JB
Thanks for this initiative! I’ll stop on the third sentence and mention that it would be great to have better support for different file formats. Support for MP3 exists but is not maintained and not part of the General registry. And I couldn’t find support for .m4a (is there?).
I guess, both m4a and wav can be used interchageably having the same loseless formats, you can convert it’s a trivial method but I have to check it out myself!
Hey!
Thanks for raising this topic. I am not a professional audio developer nor a researcher in this field. However, I have maintained a keen interest in audio related area as a hobby for several years. I’m writing from a user perspective, like reproduction of audio waves.
I would like to ask, is it possible to play a waveform (WAV) file at standard or high-resolution sampling frequencies, with a specific focus on ultra-low latency and jitter?
I’m wondering how far Julia can go in the realm of low-latency, low-jitter music reproduction. Which packages and operating systems would you recommend? Are there any tricks to streamline the process? I’m not that much into DSP as for now, my focus is to reproduce audio data with minimal interference.
P.S. In addition to SampledSignals.jl, PortAudio.jl, LibSndFile.jl and DSP.jl, I think it’s worth adding WaveFlow.jl (developed by @Gesee) to the list.
Thanks for the callout on WaveFlow.jl by @Gesee! That is a great addition to the list, especially for buffer management and streaming.
To answer your question directly: Yes, ultra-low latency, low-jitter, and bit-perfect high-resolution audio reproduction is fully achievable in Julia. Because Julia compiles to native code via LLVM and calls C libraries directly via ccall with zero wrapper overhead, it can feed PCM buffers to audio hardware with the exact same efficiency as a C or C++ application.
Here is how you can approach low-latency, low-jitter playback in Julia:
1. OS & Driver Recommendations
-
Linux (Best for Jitter Control): A distribution running a
PREEMPT_RTor low-latency kernel with JACK or PipeWire-JACK allows you to drop buffer sizes down to 32–64 samples (1–2 ms latency) with virtually zero jitter. -
Windows: Standard DirectSound introduces OS-mixer latency and forced resampling. You should use WASAPI Exclusive Mode or ASIO (via
PortAudio.jl) to bypass the Windows mixer entirely and stream directly to the hardware. -
macOS: Apple’s native CoreAudio engine handles low-latency, low-jitter streams exceptionally well out of the box.
2. Optimization Strategies in Julia
To ensure the stream never drops frames or suffers micro-stutters:
-
Pre-load to RAM: Load the entire uncompressed WAV file into contiguous memory (
Matrix{Float32}) before triggering playback to prevent disk I/O from blocking the audio stream. -
Zero-Allocation Callbacks: Ensure the ring buffer callback function allocates zero heap memory (
@allocated my_callback= 0) so Julia’s Garbage Collector never pauses the audio thread. -
Thread Pinning: Use
ThreadPinning.jlto pin the audio playback thread to an isolated CPU core and elevate its OS scheduling priority to real-time. -
Bit-Perfect DAC Matching: Ensure
PortAudio.jlsets the physical DAC clock to match the WAV file’s native sampling rate (e.g., 24-bit / 192 kHz) to avoid lossy OS-level resampling.
Regarding Multi-rate Signal Processing: If your DAC matches the file’s native sample rate (e.g., 96 kHz file \\to 96 kHz DAC), it is single-rate streaming. However, if you plan to upsample standard 44.1 kHz audio to high-res rates (e.g., 192 kHz or 384 kHz) before the DAC, that becomes a multi-rate signal processing pipeline requiring polyphase interpolation filters.
I’d be curious to hear more about your setup and your experience with WaveFlow.jl!
I am not an expert, but just want to mention Sound.jl by @JeffFessler (author of Linear Algebra for Data Science, Machine Learning, and Signal Processing) and the JuliaMusic organization if you are not aware of them.
It would be great to have better DSP tools in Julia for sure.
My experience is that it is still quite challenging to work with the current tools - at least on Linux.
@blah_blah.jl for Real Time DSP, the Faust language added Julia support some years ago. We are using it internally at GRAME for ML kind of research.
That’s awesome, I had no idea Faust added a Julia backend! Having Faust’s signal-flow compiler output native Julia code sounds like a slick setup, especially when paired with Julia’s ML and AD tools (Lux.jl, Enzyme.jl).
I’d love to hear more about how you’re combining Faust and ML at GRAME! Are there specific areas in Faust.jl or the architecture templates where the community could jump in and help test or contribute?
Basically testing evolutionary algorithms on different kind of DSP programs.
Concerning AD, we have our own plan, trying to add AD as new primitives in the Faust language. This is part of this experimental AI helped rewrite of the Faust compiler.
This is quite new so we’ll see how this evolves.
Yea I guess so, here to learn and evolve aren’t we all?
I like it a lot. I think @Gesee is a kind of a maverick. As for the setup, I’m not running anything fancy right now. Just a laptop plus Pioneer SE-A1000s. They are similar to the K701s, of course not as rounded, but I’d say with a bit more air around the strings.
Big props to @Gesee WaveFlow.jl really is a neat project!
Also, love the Pioneer SE-A1000 mention. “More air around the strings” is classic audiophile description, and honestly, a solid pair of open-back cans plugged straight into a laptop is often the sweetest, hassle-free setup. Thanks for sharing!
Oh, no, no, no, my friend, I did not write this. Furthermore, I do not consider myself an audiophile. However, I do enjoy listening to music. :- )
I made this using KaimonSlate.jl, I’m just kind of messing around here to show off various modules and a few sonic directions this could go, but it’s pretty capable already and I’m still adding more.
(NOTE: This video contains audio with a large dynamic range so be aware that there are some loud and potentially alarming sounds. Also there are some sections which might be considered flashing or stroboscopic, for anyone with sensitivity to such images)
All audio is generated and processed in Julia, realtime audio streamed to the browser using Kaimon/Slate’s amazingly performant communication fabric based on zeroMQ (and I should also mention excellent websocket support from Http.jl v2.x, thank you @quinnj).
Amazing stuff Kahli.
Haha, fair point! Honestly, “just enjoying the music” beats getting wrapped up in audiophile debates any day. 
Good to know WaveFlow.jl is treating your setup well!
This is seriously impressive! Real-time audio generation running in pure Julia and streaming straight to the browser via ZeroMQ and HTTP.jl is such an awesome showcase of what’s possible in the ecosystem.
Thanks for sharing the demo it’s super cool to see KaimonSlate.jl pushing live audio streaming like this!
@blah_blah.jl The benchmark image I pasted is showing this framework sending over 800 MBytes/second of binary data over two hops, from the notebook worker Julia process through the Slate “hub” Julia process and then delivered through a websocket to the browser.
This is enough throughput to theoretically stream around twenty different 8k (7680 × 4320) 60fps video streams (using mp4 or some similar codec) simultaneously.
30k frames (144 KBytes) each are sent at a rate of 5.5k frames/sec with latencies of p50/p95/p99 = 12/34/47 ms … if you use bigger frames and don’t send quite as much data, you can get those latencies averaging sub millisecond and p99 of <10ms.
It could likely be optimized further but I think I’m running up against Julia’s uv scheduling system internals, but I’m not complaining! 
Your points around GC issues and memory allocation are important. The core audio generation and DSP pipeline is using this approach in this project. If you’re interested in exploring it I can share the notebook with you. (shameless tactic to get you to install KaimonSlate.jl 
)
Those benchmark numbers are wild—800 MB/s over websockets with sub-10ms latency is huge!
5.5k frames/sec with sub-millisecond average latency over WebSockets is a massive testament to how fast Julia’s data transport can be when tuned well.
I’d definitely love to take you up on that offer and check out the code/notebook to see how you structured the zero-allocation DSP pipeline!
Full disclosure: I’m a die-hard .jl script / REPL fan (I have some mild PTSD from Jupyter state and module-reloading headaches in the past haha), but I’d love to check out the notebook or pull out the code to study the memory layout. Please feel free to share it!