Is there a way we can play a video and its sound currently in Julia?
The excellent VideoIO.jl package doesn’t play sound.
As far as I know there isn’t currently a way. There’s been some discussion in the FileIO package about handling container types, and that would probably be my preferred way to handle this. When you open a .avi, .mov, .mkv, etc file, FileIO could recognize it as a container that has encoded audio and video streams within, and then dispatch those streams to their appropriate decoders.
Given that VideoIO already has ffmpeg bindings, it might not be too hard to plug into their audio decoders, but I don’t know anything about the ffmpeg API and how hard it is to get the audio.
Cool. Sounds like a good way of doing it. Synchronization of the two streams though is important.
Right now I plan to “solve” this by spawning mpv and using its command interface to control and communicate with that specific instance.
Any update on extracting audio from video files?
Wow talk about reviving a thread from the past!!
The answer seems to still be somewhat “no”, at least when it comes to 100% Julia native code.
However, here is a notebook which demonstrates some solutions using all Julia code:
@Jake it’s been long on the feature request list for VideoIO so here’s a go at adding it. Please try it out and comment on the PR how you get on audio: add openaudio, loadaudio and extract_audio - Pull Request #466 - JuliaIO/VideoIO.jl - GitHub
I think I should be able to install it with ] add VideoIO#466 but this is not working. What am I missing?
Edit: never mind, this works add https://github.com/JuliaIO/VideoIO.jl/pull/466
This works, thank you for the enhancement.
julia> using VideoIO
julia> file = "20260908_162227.mp4"
"20260908_162227.mp4"
julia> samples, fs = VideoIO.loadaudio(file)
(Float32[0.0 0.0; 0.0 0.0; … ; 0.006339062 0.59741265; 0.051741634 0.6391921], 12000)
julia> using Sound
julia> sound(samples, fs)
3822592
Note that using MATLAB’s help for Sound.jl was really helpful, as this package is very similar to the MATLAB commands.
Yes and no, I could do it with:
julia> video_path = ".vscode/extensions/openai.chatgpt-26.715.31925-linux-x64/webview/assets/appshot-demo-DcV9m9GT.mp4";
julia> # wait=false runs it in the background so your Julia REPL doesn't freeze!
p = run(`mpv --vo=x11 --hwdec=no --autofit=1280x720 $video_path`, wait=false);
julia> # You can stop or kill the playback programmatically from Julia at any time:
# kill(p)
[You might not actually need some options, like that -vo=x11 (or --hwdec=no) for me without it I got a segfault “Aborted (core dumped)”); I was just using it for my broken Linux setup (apparently EGL and GLX is broken for me). I’m not sure what this does on non-Linux, hopefully ignored.]
needing first setup (this might be different across different Linux, even if you have apt-get):
$ sudo apt install mpv libmpv-dev
Note the file I found and used above DOES have audio, just not much, so at first I though I also had audio issues… trying to debug pulse, for audio, that was never broken…
See (–ao=help and):
$ mpv --vo=help
Apparently --vo=sixel might work for you, for me I could see the video in the terminal with (but not from within Julia):
mpv --vo=tct .vscode/extensions/openai.chatgpt-26.715.31925-linux-x64/webview/assets/appshot-demo-DcV9m9GT.mp4
That will decode and play the audio while not creating a video window, which is also why it avoided my current (Linux) GLX graphics problem:
ffplay -nodisp -autoexit video.mp4
Something like this might also work for you, please confirm and on which platform (probably works on all non-Linux, and Linux without my Nvidia userspace/kernel driver mismatch problems, that disabled GLX, not EGL):
using FFplay_jll, FFMPEG_jll # the latter just for some workarounds I was trying for paths/dependencies
julia> env = Dict(
"DISPLAY" => get(ENV, "DISPLAY", ""),
"XAUTHORITY" => get(ENV, "XAUTHORITY", ""),
"PULSE_SERVER" => get(ENV, "PULSE_SERVER", ""),
"LD_LIBRARY_PATH" => joinpath(FFMPEG_jll.artifact_dir, "lib"),
);
run(setenv(`$(FFplay_jll.ffplay()) -autoexit "vscode/extensions/openai.chatgpt-26.715.31925-linux-x64/webview/assets/appshot-demo-DcV9m9GT.mp4"`));
# Maybe just: run(setenv(`$(FFplay_jll.ffplay()) -autoexit ".vscode/extensions/openai.chatgpt-26.715.31925-linux-x64/webview/assets/appshot-demo-DcV9m9GT.mp4"`,
"LD_LIBRARY_PATH" => joinpath(FFMPEG_jll.artifact_dir, "lib")))
Make sure it first works for you from the shell (the problems I had were not Julia problems, nor even arguably ffplay problems, which is cross-platform):
Since I get:
$ glxinfo -B
name of display: :0
X Error of failed request: BadValue (integer parameter out of range for operation)
Major opcode of failed request: 152 (GLX)
Minor opcode of failed request: 24 (X_GLXCreateNewContext)
Value in failed request: 0x0
Serial number of failed request: 69
Current serial number in output stream: 70
It also ruled out for me (since it uses GLX, not EGL, and I spent a lot of time how to look into fixing my GLX install…):
ffplay -vn -autoexit \
.vscode/extensions/openai.chatgpt-26.715.31925-linux-x64/webview/assets/appshot-demo-DcV9m9GT.mp4
Showing videos (with or without audio) is a can of worms. You can decode video separately with Julia or the audio, but what does it mean to get both (or only video), and to the screen? It implies a GUI application (or popup window from other process, just invoked by Julia as above). And that GUI application needs to work with Linux, EGL or GLX or whatever you have and can use, and ideally this application is cross-platform to e.g. Windows.
Out of the scope how to do a GUI (without or) with web tech. It might be simplest that way with something like Electron.jl or: