# Julia for audio live coding?

**URL:** <https://discourse.julialang.org/t/julia-for-audio-live-coding/101593>\
**Category:** General Usage\
**Tags:** audio\
**Created:** [July 13, 2023, 10:08pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593 "2023-07-13T22:08:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 13, 2023, 10:08pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/1 "2023-07-13T22:08:35Z")

</div>

Hi all, I’m new to Julia and trying to gauge feasibility with regards to audio live coding.  
What I would like to do essentially is to use the REPL or the “Run code” function in vscode to create audio callbacks that generate sound using synthesis techniques, that stay running in the background while I write more code.  
Upon evaluating new code I would on the fly replace the callback function with another one, with no pause or glitching ( assuming I make sure that the audio stream has no discontinuities etc ).

I was thinking of doing something like using PortAudio.jl and setting up a background thread with an audio streaming function. The streaming function would have a reference to another function inside it and I would update that reference with a new one when I want to change the sound generation.

Replacing the audio gen function with a new one would not need to be instantaneous - I would in fact want to schedule it ahead of time for the next beat or the next bar, which should give time to JIT compile it before replacing the reference.  
In practice I would replace the graph in advance and have some kind of fade in to avoid pops and such that the new sound starts at the right time  
I might also need some kind some kind of thread synchronisation to do this safely and would probably also want to disable the GC.

Before I set myself on a goose chase, does this all sound doable without major hacking and hair pulling ?  
This is all very similar to what gibber does in Javascript ( [gibber](https://gibber.cc/) ) for those who know it, but being able to use all the power of Julia would be incredible.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [July 14, 2023, 2:02am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/3 "2023-07-14T02:02:18Z")

</div>

> [@JTriggerFish](#):
>
> use the REPL or the “Run code” function in vscode

Neither of these are live coding. In short, you evaluate the exact lines you typed when you press Enter in a REPL, but a live coding environment automatically evaluates your edits and reevaluates other affected parts of your source code (though it may not be equivalent to rerunning the whole thing, depends on implementation). I suggest the paper “Supporting exploratory data analysis with live programming” by Robert DeLine and Danyel Fisher for a better comparison of REPLs and live programming. Jupyter, Revise.jl, and especially Pluto.jl, get closer to live programming.

> [@JTriggerFish](#):
>
> like using PortAudio.jl

The documentation is scant and there aren’t Julia examples or tutorials for many of the things people would do with PortAudio. I myself had to dig into source code to tell another user how they can play stereo audio from 2 vectors. The only thing in the docs about callbacks is a `call_back` argument that only supports `C_NULL`, and there is an open issue about supporting callbacks. Risking undue criticism, I don’t think you’ll have an easy time using this package.

> [@JTriggerFish](#):
>
> The streaming function would have a reference to another function inside it and I would update that reference with a new one when I want to change the sound generation.

Really iffy on whether this is feasible, since there’s a whole C side of PortAudio. In pure Julia, this can also be problematic depending on what you mean by “reference to another function” and how you try to update that while streaming. On a broad level, you need to make sure that the stream doesn’t bake in the function at the start, that you can communicate the function at every frame, whether you changed it or not. For a simple example, `x = 1; loopprint(x)` would run `loopprint(1)` even if you change `x = 2` later because the `loopprint` only got to access `x` once at the call site. For a more relevant example, a `Task` runs as functions existed when `schedule`d (running in a world age, formally speaking), so even if you managed to edit methods before the `Task` runs, it won’t reflect the changes.

Being able to change functions at runtime means the compiler can’t infer the return type at some level, so you could help out with [type annotations of the return values](https://docs.julialang.org/en/v1/manual/performance-tips/#Annotate-values-taken-from-untyped-locations) or FunctionWrappers.jl. I’m not sure if these can fully remove the type instability’s overhead, you’d have to look out for the associated allocations.

> [@JTriggerFish](#):
>
> Replacing the audio gen function with a new one would not need to be instantaneous - I would in fact want to schedule it ahead of time for the next beat or the next bar, which should give time to JIT compile it before replacing the reference.

Generally JIT compilation is triggered upon the first call, so you’ll have to figure out how you can make sure something is precompiled before you throw it in the stream. Precompilation has been changed in a recent version to be more complete for packages, specifically caching the endpoint native code rather than intermediate code, so I’d check if that more complete precompilation can be used interactively.

> [@JTriggerFish](#):
>
> would probably also want to disable the GC.

If you like JIT compilation, you’d want to keep the GC to handle the garbage. I know you’re trying to avoid stop-the-world jitters, but there’s no known ideal solution in any language. Incremental GCs try to split the work into subsets of frames, but there are still complaints of dropped frames since it overall does the same work + overhead and can thus fall behind. Background GCs put a lot of the work in a background thread, but there are still pauses. AFAIK Julia’s GC does not address soft realtime latency like that, but an upcoming version’s GC can run on multiple threads to save time.

> [@JTriggerFish](#):
>
> Before I set myself on a goose chase, does this all sound doable without major hacking and hair pulling ?

Any development will have hair pulling. Even if the pieces are there, putting together a puzzle is hard and has many setbacks.

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [July 14, 2023, 2:36am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/4 "2023-07-14T02:36:57Z")

</div>

I wonder if this could be split into a dynamic process that does all the fancy frontend stuff and a static process that just takes serialized instructions and does non-allocating audio generation?

Like, Clojure’s [Overtone](https://github.com/overtone/overtone) generating instructions, sending them to C++ [SuperCollider](https://github.com/supercollider/supercollider). But I wonder if actually both could be in separate Julia processes.

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 14, 2023, 7:31am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/5 "2023-07-14T07:31:35Z")

</div>

Thank you very much for the detailed answers !  
Some interesting points to ponder.  
With regards to live coding I see what you are saying but I think it might be splitting hairs a bit, the two known implementations of the concept which are Extempore and gibber, employ an approach similar to what I described.

And indeed an alternative approach like supercollider where you precompile nodes ( UGens ) ahead of time and you construct a graph with them on the fly solves a lot of these problems.  
But it’s also a lot less flexible and I prefer the ‘all in one ‘ approach.

With regards to the GC: I can live with having a lot of garbage and memory leaks as long it stays contained ( ie not multiple gigs ) given that a ‘live session’ does not need to run for more than an hour. My intuition is that it would be ok but I could be wrong.

As to hair pulling, yes I write mathematical software for a good part of my day job so I’m quite familiar with the idea 🙂  
It’s more a question of : is it doomed from the start or it has a chance to work.  
Not being able to force the compilation ( or most of it ) ahead of execution and the thread safety problem do sound like they could be show stoppers.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [July 14, 2023, 8:16am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/7 "2023-07-14T08:16:47Z")

</div>

I have no experience working with audio, real-time or otherwise, but I would imagine the biggest factor is how large your buffer is. If you can accept a 1+ second buffer, then that should be enough to cover just about any GC pause.

I would expect that live code changes you make shouldn’t trigger too much heavy compilation. You’re likely making parameter or small logic changes, not bringing in new large packages (which hopefully you can load/precompile ahead of time).

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [July 14, 2023, 9:32am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/8 "2023-07-14T09:32:48Z")

</div>

I’ve played around with synthesizing using PortAudio before. It definitely involved hair-pulling at the time, getting a ring buffer to work without a C shim library and other things. I think something about interacting with threads from C has changed in Julia since then, possibly making the callback logic easier. At the time, I had to take a lot of care never ever to allocate anything, or I’d get an immediate segfault. Which was not nice for development experience 😉 As for live updates, I think when Julia compiles that completely stops everything across all threads, so you are likely to get glitches that way. Not sure if it can be completely avoided if the C thread does its own thing and continues playing from the buffer while Julia halts.

It was still fun though, even though I tried to take all the dirty shortcuts I could, I did manage to get a polyphonic synth playable via Midi if I remember correctly. You could have a look here [GitHub - jkrumbiegel/PortAudioSynth.jl](https://github.com/jkrumbiegel/PortAudioSynth.jl)

---

<div class="post-metadata">

**Author:** ![cormullion](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cormullion/32/49131_2.png) [@cormullion](https://discourse.julialang.org/u/cormullion)\
**Post date:** [July 14, 2023, 9:36am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/9 "2023-07-14T09:36:18Z")

</div>

If you write a Julia equivalent of Sonic Pi, I’ll be fascinated to see it!

I tried controlling Sonic Pi from Julia (a Pluto notebook), and - although it was easy - I didn’t find it an experiment worth pursuing - might just as well write the Ruby directly…

![Screen Recording 2023-07-14 at 10.15.23](https://global.discourse-cdn.com/julialang/original/3X/e/f/ef024180a46649ae5b6041ff87ff9ffe1a05bc4d.mp4)

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 14, 2023, 10:41am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/10 "2023-07-14T10:41:14Z")

</div>

This is very cool ! But I agree there is limited upside in making yet another supercollider client, there’s already many good ones.

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [July 14, 2023, 11:50am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/11 "2023-07-14T11:50:04Z")

</div>

> Not sure if it can be completely avoided if the C thread does its own thing and continues playing from the buffer while Julia halts.

I think you could just have a separate process that plays the audio from a shared memory buffer.

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 14, 2023, 2:38pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/12 "2023-07-14T14:38:02Z")

</div>

This is a good point actually, that would be a good way to airgap any issue with portaudio or c code from the rest of the execution.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [July 14, 2023, 8:39pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/13 "2023-07-14T20:39:06Z")

</div>

> [@jules](#):
>
> I think when Julia compiles that completely stops everything across all threads

Does it? If compilation or anything else like GC halts all threads, it probably should be written down in multithreading docs.

> [@jules](#):
>
> PortAudioSynth.jl

Looking at the code, it looks like you put a fixed callback in a lower-level wrapper of a PortAudio Stream, and all it really does is pull samples via a `Ref{AudioBuffer}` userData, the processing is elsewhere in a `Timer`. How feasible do you think it would be for a separate thread to write to the userData with arbitrary function calls when prompted by PortAudio’s asynchronous callbacks, like how robsmith11 and JTriggerFish are thinking? The prompting is important because otherwise it’s just a `write` that can get into a race condition with a C thread, and PortAudio.jl already seems to have implemented PortAudio’s blocking `write`.

Now that I’ve read more of the docs, is it possible to get away with a conditional `write` loop?

> If you want to avoid blocking you can query the amount of available read or write space using Pa\_GetStreamReadAvailable() or Pa\_GetStreamWriteAvailable() and use the returned values to limit the amount of data you read or write.

If the buffers are too full then we could opt not to `write` that frame and perhaps adjust how many samples to process or how much time waited before writing processed data? Really speculating here, though.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [July 15, 2023, 5:24am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/14 "2023-07-15T05:24:58Z")

</div>

In my testing at the time, I couldn’t get Julia to do anything “prompted” by the C thread. There was only this option to wait on an async condition that gets notified from C, but this was too slow and dropped events as well. That’s why I said I think nowadays there are different options that might be worth trying out.

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 21, 2023, 10:35pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/15 "2023-07-21T22:35:44Z")

</div>

Here’s my current attempt, I had some early success with SDL’s audio queing model which possibly allievates some of the issues mentioned above.  
The current version errors at runtime, I need to spend more some time debugging it but in case anyone is interested or has comments ( I still don’t really know what am doing when coding in Julia ):

> <https://github.com/JTriggerFish/InteractiveSynthPOC.jl/blob/main/src/InteractiveSynthPOC.jl>

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [July 21, 2023, 11:18pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/16 "2023-07-21T23:18:55Z")

</div>

If you’re changing everything about the `SineWave`, it might be simpler to have it be immutable and just replace it with a new one. [Accessors.jl](https://juliaobjects.github.io/Accessors.jl/stable/) should make that easy.

Also

```julia
        if index > length(sine_waves) || index < 1
            println("Invalid index.")
        else
            wave = sine_waves[index]

```

seems a little unusual. Are you trying to avoid getting an exception here? See `checkbounds` and `@inbounds`.

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 21, 2023, 11:42pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/17 "2023-07-21T23:42:36Z")

</div>

Thanks, that looks quite useful, though I really want to avoid any allocations so I’ll have to check what it does behind the scenes.  
And yes I want to avoid exceptions too though there’s probably better ways to do this in the language indeed.

---

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [July 22, 2023, 1:10am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/18 "2023-07-22T01:10:39Z")

</div>

Dude this is sick

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [July 22, 2023, 2:17am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/19 "2023-07-22T02:17:19Z")

</div>

Here’s one without allocations.

```julia
using BangBang
struct S
  x::Float64
end
julia> let
       ss = [S(1)]
       @time @set!! ss[1].x = 99
       end
  0.000001 seconds
1-element Vector{S}:
 S(99.0)

```

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 22, 2023, 9:18pm UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/20 "2023-07-22T21:18:06Z")

</div>

I got it sort of working.

Observations:

- I can’t get the buffer size much lower than 1024 without all sorts of crackling. My hunch is that the thread locking is quite slow, but my threading implementation may also be quite suboptimal - suggestions welcome here
- It can play in the order of 200-300 sines implemented in a very naive and inefficient way, which is not too bad. I don’t want to optimize the memory layout and access because ultimately I want to evalute a whole compute graph for each sample so there will be very little cache coherency. However any other performance tips welcome again. I have been quite careless with types and casts, I don’t know how much it matters in Julia
- The very basic code interpreter based on readline() and eval() works. I can type “supersaw(110)” to instantiate a couple of naive saw waves and not notice any audio glitch.  
However, if a syntax error throws an exception, the first one will glitch the audio. After that following errors are fine. I’m guessing the JIT compiler does some work the first time, not sure how to avoid that.

All in all it seems somewhat promising but there are many questions left.  
Also, is there a simple way to do a more REPL-like input, with completion , syntax highlighting etc ?

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [July 23, 2023, 2:53am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/21 "2023-07-23T02:53:26Z")

</div>

> [@JTriggerFish](#):
>
> Also, is there a simple way to do a more REPL-like input, with completion , syntax highlighting etc ?

> **[GitHub - MasonProtter/ReplMaker.jl: Simple API for building repl modes in Julia](https://github.com/MasonProtter/ReplMaker.jl)**
>
> Simple API for building repl modes in Julia. Contribute to MasonProtter/ReplMaker.jl development by creating an account on GitHub.

> **[GitHub - KristofferC/OhMyREPL.jl: Syntax highlighting and other enhancements...](https://github.com/KristofferC/OhMyREPL.jl)**
>
> Syntax highlighting and other enhancements for the Julia REPL - GitHub - KristofferC/OhMyREPL.jl: Syntax highlighting and other enhancements for the Julia REPL

> [@JTriggerFish](#):
>
> I have been quite careless with types and casts, I don’t know how much it matters in Julia

Type stability is very important to performance. You can use `JET.@report_opt` to identify these problems.

---

<div class="post-metadata">

**Author:** ![JTriggerFish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jtriggerfish/32/51516_2.png) [@JTriggerFish](https://discourse.julialang.org/u/JTriggerFish)\
**Post date:** [July 23, 2023, 4:30am UTC](https://discourse.julialang.org/t/julia-for-audio-live-coding/101593/22 "2023-07-23T04:30:20Z")

</div>

Thanks a lot! The types are constant but there may be a few casts here and there, good to know in any case and I will check.

EDIT: @jar1 turned out to be mostly fine but it helped fixed a couple of interesting things, and indirectly made me realise that non mutable structs were stack allocated so replacing them with a new one is indeed the best solution as you suggested.

Seems like it can play about 1000 naive sines at a buffer size of 512 now.  
I also spent a lot of time debugging some annoying crackles until I realised that SDL2 doesn’t seem to like a sampling rate other than 44.1khz at all, there must be some bug in the sample rate conversion.