I have some signal processing code (pitch detection from an audio signal) and I would like to write a very simple application that displays the result graphically in a window, continuously updated in real time, maybe controlled by a few sliders/buttons for some parameters.
I am absolutely inexperienced in programming GUIs. I would like to ask for recommendations on a GUI framework to use on Linux with current Julia.
I have little to no knowledge of how signals and event loops should be implemented for something like this. The rough idea is that
I want to do a blocking read from an audio source and do the processing; this is the bulk of the CPU usage and should probably live in its own thread (doing this fast is the main motivation for using Julia),
the above should deposit results in a shared container between threads,
another thread should do the display and control.
But maybe I need a way that the processing thread can tell the display thread that it has new results?
Pointers to code that does something like this are welcome.
This example works fine on macOS. It shows a real-time view of the mic audio signal in the frequency domain. To detect the pitch you will have to find the peaks and transform them into note information.
For the GUI part, I’d recommend QML. This examples shows, how to embed GR graphics in a QML window.
Unfortunately this application is in a private repository, which is for several reasons. For instance the code is pretty ugly, monolithic, and tight to a specific application. My (middle term) vision would be that we build some high-level UI widgets for data visualization and put them into a public repository. For instance I have a signal viewer, where one can display a signal in time and frequency domain and zoom in and out.
If you are willing to give Gtk.jl a try feel free to ask questions. When it comes to plotting I can also send you some code snippets. Best would be that you make a drawing, how your UI should look like.
Regarding the concrete implementation: I think it is sufficient that you use one BG thread that does the processing. This BG thread can update the UI using g_idle_add. No need for an additional thread.
This is how the latest version of PortAudio.jl works. I’m about to merge and release the latest branch. It uses Julia’s multi threading support to do the wait in a separate thread so that it doesn’t block the Julia event loop. So you can call the read within an @async task and everything will play together nicely.
This part should take very minimal CPU because it spends most of its time waiting.
For simple stuff with some sliders etc. I’ve found Makie sufficient, see their doc page on interaction. http://makie.juliaplots.org/dev/interaction.html
Everything in a Makie plot is an observable, which makes tying behaviors to gui events and gui components to signals very easy.
I think I am working on something similar, but I am generating samples and doing write, not read. I was surprised how smooth it works. My local PortAudio.jl is a bit tweaked so probably you won’t be able to run it till everything is merged and linux problems will be solved.
I am also planning to add some UI, I was thinking about using Electron.jl for that but I am also interested about different options.
I’m curious if you created the layout using Glade or the directly with the code. i’m finding it very painful to get glade to do what I want.
also, are you using a plot package for the graphing and just pointing them at a cairo window, or are you doing the graphics (and plotting) using your own code ?
Just thought I mention that similar discussions on slack between Makie and image-processing channels have discussed how we can do a more flexible version of ImageView.jl and maybe build on some ideas of TInker.jl. Not as robust as GTK (yet), but if you need 3D modeling at some point it might be hard to do in GTK.
Tamas, this is an aside that has nothing to do with your question, but several weeks ago I went down some patent rabbit holes which ended with US patent 2,286,030, and I suspect you might find it interesting at least from an historical point of view.
I think they still make a version of those, but I have only seen them on videos.
I am experimenting with modern algorithms, like
@article{de2002yin,
title={YIN, a fundamental frequency estimator for speech and music},
author={De Cheveign{\'e}, Alain and Kawahara, Hideki},
journal={The Journal of the Acoustical Society of America},
volume=111,
number=4,
pages={1917--1930},
year=2002,
publisher={Acoustical Society of America}
}
@inproceedings{mcleod2005smarter,
title={A smarter way to find pitch.},
author={McLeod, Philip and Wyvill, Geoff},
booktitle={ICMC},
volume=5,
pages={138--141},
year=2005
}
combined with some heuristics.
This is just a fun project, modern CPUs are incredibly powerful and can do a lot of analysis in real time. My main interest lies in the following areas:
compensating for the effect that plucking the string changes the pitch, so the actual pitch is a bit uneven for short notes. I want to visualize and analyze this.
saving and reproducing a slightly “tweaked” microtuning (adjusting the chromatic scale) that I happened to like because it sounds good.
I will make the code public as it matures. Stay tuned
I got to see one in person when a technician was working on my son’s sax, and was fascinated by how much better the experience is than with every tuner app I’ve seen. No lag at all. This was a solid state unit, probably from around 1970. The patent is cool because they describe how you can see that overtones are not necessarily integer multiples of the fundamental by seeing the different rings appear to slide at different rates. Relies on AM modulation of the neon indicators. This isn’t a real project for me, just something I found interesting. I’ll have to look at your papers a bit later.
@Tamas_Papp If you’re looking at pitch tracking I’ve seen recent papers seem to use pYIN rather than the classic regular YIN. I’m not super familiar with either one, but pYIN might be worth checking out if you haven’t yet.
Thanks. It looks like I have a lot of reading to do.
My reading of the literature is that a lot of modern algorithms are focused on general pitch detection (eg voice), but musical instrument spectra have a specific structure that can be exploited for further accuracy.
