State of Audio DSP & Feature Extraction in Julia: Gaps, Priorities, and Future Directions

This is simply astonishing… I wouldn’t know where to start…

Again, I don’t consider myself an audiophile, however, after reading your posts I have to admit that you sound a bit like one. What’s your setup, if I may ask? Judging by the profile photo, my guess is Mark Burson and John Delmo / Joseph and John Grado sound signatures, or something like that?

To be honest, I tried it once, shortly after it was released, but I was running Julia inside Windows Subsystem for Linux, and I recall encountering some problems. However, I wasn’t kidding when I said that @Gesee is a kind of maverick. Before WaveFlow.jl, I was interested in his other package related to entity component systems. Very cool.

My profile picture is actually just a collage of Tony Stark / Iron Man, not boutique audio pioneers! While Tony probably has a holographic, Arc-reactor-powered DAC in his Malibu lab, my real-world setup is embarrassingly simple: just standard headphones plugged into my machine while I tinker with math, DSP scripts, and code.

I don’t own any high-end discrete amps or exotic open-backs I’m purely here for the software, the signal processing, and the fun of building things in Julia!

Ah, WSL audio will definitely do that! Running real-time audio through WSL (especially with the WSLg/PulseAudio virtualization layer) is notoriously tricky compared to bare-metal Linux or native Windows with WASAPI. Getting direct hardware access and low latency through that bridge is always an uphill battle.

And 100% agreed on @Gesee building both high-performance audio tools and ECS architectures in Julia is seriously impressive work!

I guess this new info changes the picture a bit because the modifications to the transport layer and your OS tuning proposals, including the schedulers, will still be noticeable, but I’m afraid they won’t be as dramatically pronounced.

Haha, sorry to disappoint on the exotic gear front!
You make a very fair point without dedicated discrete DACs and amps, aggressive kernel scheduler tweaks or transport-layer micro-optimizations aren’t going to produce a night-and-day audible difference.
From an engineering perspective, though, I’ve started experimenting with keeping the signal processing and buffer management as CPU-agnostic and portable as possible. My main goal is simply eliminating buffer underruns (xruns) and ensuring zero-allocation inner loops, so the code runs rock-solid across standard machines without needing hyper-specific OS hacks. Still very much experimenting, but having a ton of fun digging into it!

It wasn’t a disappointment by any means. Out of respect for your musical taste, I mentioned those designers. I wasn’t sure about John Grado, but I decided to mention Mark Burson in case you going on a trip into electrostatic territories. With the kind of music you mentioned, at those speeds, and above all, the level of current produced, we might all start moving along with the bases of our chairs.

yeah, had to go fix up MFCC.jl myself to make it usable enough for my purposes, as it was written for an older version of julia. Lots of librosa functionality that one usually takes for granted, are hard to find here.

Quite amazing work on real-time audio, but idk, Julia feels like it’s not made for real-time, so that involves swimming upstream…I think non-rt processing is a more natural fit for the language, what with GC and dynamic dispatch often being so handy while hot loops can still be type-stable and fast.

So if this is a wishlist for fairy godmothers, yep: it would be great if someone collated / unified the various feature extraction packages, perhaps with package extensions? Doing what MusicProcessing.jl (which as mentioned is still very preliminary) seems to be aiming at. Maybe by figuring out a painless librosa translation layer, if existing packages don’t have the same API / assumptions, because compatibility is very nice to have :slight_smile:

Haha, love that mental image! Shaking the bases of our chairs with high-current electrostatics definitely sounds like the dream listening setup. :grinning_face_with_smiling_eyes:
Truly appreciate the kind words and the awesome designer recommendations if I ever take the plunge into the high-end hardware rabbit hole down the line, I know exactly who to look up!

Totally with you on that! Real-time audio in Julia definitely comes with its hurdles wrestling with GC pauses and low-level thread scheduling can certainly feel like swimming upstream. That said, if we design the core inner loops to be zero-allocation and strictly type-stable, we get the best of both worlds: primitives optimized enough for real-time if someone needs it, but an absolute breeze for high-throughput, non-RT feature extraction, we get blazing-fast batch feature extraction for ML, while keeping the routines efficient enough to slot into real-time buffers if someone needs it.

Your wishlist is spot on a unified toolkit with a predictable librosa-style translation layer and modern package extensions would save a ton of people from reinventing the wheel. Did you push your MFCC.jl fixes to a public branch or fork? I’d love to check out what you patched. If you’re interested, I’d definitely be down to help organize a repo/scaffold and start consolidating these features!

yep, the patches were included in a release like 3 years ago and MFCC.jl is currently in a usable state, although with no new features / development.

Ah gotcha, good to know MFCC.jl is at least functional on current versions!

Seems like the consensus is that the ecosystem has decent low-level primitives, but a modern, unified toolkit for feature extraction and analysis is wide open for development. Thanks for the context and pointers!
Hope I got it in the right meaning

With all due respect, I don’t know why, however, I’m constantly getting a feeling you are trying to rewrite my own words. I am just simply trying my best to keep up with the latest developments regarding the K701, HP100SE, and LCD-MX4 mixing tables, which isn’t always that easy.

Anyway, I support your view about the importance of “ultra-low latency, low-jitter, and bit-perfect high-resolution audio reproduction”. In addition, I believe a well-engineered transport layer is crucial not only for audio reproduction and reviewing final recordings, but also for minimizing specific forms of distortion and jitter at the early stages of the recording path.

Appreciate the perspective! Definitely agree that a clean transport layer and low jitter matter a ton on the recording and monitoring side. Thanks for sharing the insights! Pardon me I guess I genuinely misunderstood the context

No need to be sorry, really! :- )

Glad to know!

I hope to see more of your packages published on your specific subject matter. Not necessarily just about the frequency domain or electromagnetic interference. However, please kindly be informed that gravitational wave distortion of spacetime is slightly outside my scope of interest. :- )

Off-topic, but:

IMHO it appears that’s because blah_blah.jl is using LLMs to respond, which tend to repeat your own words back to you and often also lack brevity. While I find LLM-speak highly annoying and very very grating, putting off OSS contributors who may be using LLMs to translate because they’re not native English speakers also doesn’t feel great, and there is the slightest possibility that LLM usage just infects people and causes them to write like that, so maybe it’s better to ignore it most of the time…

Nevertheless, I bet most people are more comfortable reading idiosyncratic Google translated responses and slightly ungrammatical language than they are with the uncanny feeling that they’re responding to a clanker :neutral_face:

I’m sorry, of course, you can’t drive electrostatics with designs by Mr. Mark Burson and Mr. John Delmo. It was a broader parallel related to my colleague DB’s profile photo, which, at the time of writing, seems to include direct references to AC/DC band. Even though these designs offer a really decent amount of current, it might certainly not be enough. I’ll try to explain myself directly to Mr. Mark and Mr. John the next time I have a chance to be in touch with them. Nevertheless, even without electrostatics, I believe the combo I proposed in that post would offer a tremendous amount of joy and fun for this particular music material.

As for LLMs, I use them often. For me, it’s better to read correct English on the forum. I’m not a native English speaker, however, I’ve noticed that the models don’t change much of my original writing. When they do try to do it, I tell them not to. Personally, I find writing correct English to be a form of respect toward my interlocutors. I don’t mind using LLMs in this space at all.

I don’t know why I got that feeling. For me, it wasn’t annoying. It was just strange. I took part in this discussion by asking questions that I believe were directly related to the topic of this thread.

Well @blah_blah.jl @j_u, I’m kind of happy you liked WavesFlow.jl
I can for sure continue maintaining it since peoples find it cool :slightly_smiling_face:

Great to see you Gesee, and thanks for the WaveFlow!