Julia for live control of USB3.0 camera and DAC/ADC boards, with GUI?

I’ve prototyped a system in LabView that has the functionality I want, but I’m coming up against speed issues and would like to avoid developing further in a commercially licensed language.

The current labview system controls a USB3.0 camera, and records in raw AVI, while controlling a series of DAC/ADC boards in parallel. And it does it with a functional GUI interface. Basically it does what LabView does well… but I’m pushing it to its speed limits and dropping frames, and also want to move away from commercial software platforms for this system.

Would you say Julia is ready for this kind of thing? I mainly use it through IJulia, so real-time video-based GUIs seem pretty far from my experience. Are there any examples out there? What GUI packages are appropriate?

@ianshmean It would be great to help with this.
VideoIO.jl comes to mind
https://github.com/kmsquire/VideoIO.jl
I was going to say this is not being updated - but the latest changes were seven days ago, so it is looking better. Would VideoIO.jl read the images from your camera?
I would think that a USB 3.0 device is probably pretty fast - can you just read an image from the device by opening and reading from a file?

I Cant say anything useful about GUI packages.

The camera I’m trying to use is this: https://www.ptgrey.com/grasshopper3-32-mp-mono-usb3-vision-sony-pregius-imx252

To read from it will probably require using ActiveX or directshow through the FlyCapture SDK
https://www.ptgrey.com/flycapture-sdk

I suspect that it’s a very long shot to expect Julia to handle this at Julia speeds, because it would likely require Python interfacing, to get access to the camer

Perhaps you could use the flycapture C API from Julia? Calling C code from Julia is extremely efficient (essentially zero overhead) and quite easy to do: https://docs.julialang.org/en/stable/manual/calling-c-and-fortran-code/index.html

1 Like

Not that anyone is interested, but I do have a FireWire camera somewhere, complete with cables and a PCI board.
Completely useless these days, but at one point I was interested in FireWire for high speed networking.

I don not have the skills but Genicam looks to be the thing GenICam - Wikipedia
Maybe there will be a Julia implementation one day.

you could try calling this from Julia?

Sorry for the off-topic (in the sense of “non-Julia”) but there are open source fast alternatives for this tasks (acquiring videos, real time processing and communicating with boards). Most people at my institute use bonsai: Bonsai — Kampff Lab (it’s based on C#, Microsoft Reactive framework and OpenCV as far as I understand). I’m not sure how is the C# interoperability with Julia.

1 Like

It might also be interesting to look into Julia bindings for libuvc, which is cross-platform and BSD-licensed: GitHub - libuvc/libuvc: a cross-platform library for USB video devices

Hi,

Can you use ffmpeg to get video from the camera? Some googling suggests people have tried with other Point Grey cameras before, although I didn’t look enough to know if they succeeded.

Anyway, my reason for asking is that VideoIO wraps ffmpeg, so if you can use ffmpeg to get video directly, then there’s a reasonable chance you can get VideoIO to read from it. (Note that, at the moment, VideoIO requires an older version of ffmpeg (<= 3.0) to work).

Cheers,

Kevin

EDIT: I just realised misunderstood what they were sharing… I thought it was the code for how they interact with ffmpeg… but it’s just the ffmpeg source code… but at least it shows you can use ffmpeg!

Interesting idea. PointGrey actually provide the ffmpeg source code used in their SDKs.

They require you to log in with a free account to access the file, so perhaps I shouldn’t provide the full download link here, but this is the text found through https://www.ptgrey.com/support/downloads:


FFMPEG source code used in FlyCapture2 — 11/21/2017 - 13.660MB
The FlyCapture2 and Spinnaker software uses code of FFmpeg http://ffmpeg.org licensed under the LGPL v2.1 (GNU Lesser General Public License v2.1 - GNU Project - Free Software Foundation). The ffmpeg code can be found online at GitHub - FFmpeg/FFmpeg at 5156578d1f486163d5b83f1d63246cd23d107933 or as attached.


It would be so great to have PointGrey camera support in VideoIO!

Note that PointGrey/FLIR are migrating to their Spinnaker SDK for new cameras - so this might be a better target for any significant effort. The library is cross platform and there are C bindings available.

Regarding your original question I’m sure this is viable. I’ve previously used Julia to do some real time digital holography - grabbing frames from a Hamamatsu ORCA over FireWire (using a hacked fork of DC1394.jl) doing some FFTs and linear algebra, then displaying it all in real time on a Cairo canvas.

(The trickiest bit was making the rendering fast enough… I’m not sure how you’d manage that in, e.g., a notebook environment)

Great to hear! And you’re right, the Spinnaker SDK would be the better target.
If I get anywhere I’ll share progress…

Also, you’re right that the speed of the rendering live video in a GUI is a big challenge… especially if you want to serve it to a browser-based web app… you have to figure out a way to transcode down the raw result to a preview-appropriate resolution and framerate on the fly.

Is there anything like this out there in Julia? Perhaps for IJulia? Again, it’s one of those things that LabView does very well…

There was this topic Fastest way to blit images to a window - mentioning an implementation of a Vimba wrapper in Julia.

Any news regarding wrapping Spinnaker? I am looking into wrapping Basler’s Pylon API and providing a generic Cameras (ImageAcquisition) package that should provide an AbstractCamera type.

I’ve wrapped the Spinnaker SDK and provided a high level interface: GitHub - samuelpowell/Spinnaker.jl: A Julia interface to the FLIR/PointGrey Spinnaker SDK

Certainly work in progress, but makes it very easy to set camera parameters and grab images from, e.g., Blackfly S, using Julia.

5 Likes

Fantastic! I can confirm this works with the Grasshopper 3 USB3.0 camera on Windows 10
I’ll continue to test and file any issues that arise.

Really awesome to have this functionality, thanks @samuelpowell

1 Like

Great work!

Here is our current state of a (very minimal) high-level camera interface: GitHub - IHPSystems/Cameras.jl: A Julia interface for cameras

It could be great if we could implement a common interface. See [WIP] Cameras.jl

A desirable outcome if viable. I’ll keep an eye on your package and see how it develops.

1 Like

OK. Feel free to comment/contribute. Have you considered naming property setters with a set_-prefix? E.g. set_gain! rather than gain! ?

Feels more Julian to me to consider these as mutating the state of the camera, and hence gain!(cam, setting) in my implementation (and gain(cam) to retrieve it).

Hi ianshmean,
Sorry for out of topic question.
I am also using FLIR Grasshopper USB3 camera, It works fine with Spinnaker software provided on thr site.
I am trying to build a python code using Tkinter for GUI to capture live streaming of this camera. unfortunately nothing is working out…
I am coding Jetson nano developer kit for this. Its Ubuntu 18.04.
Have been struggling from more than 2months. Please if you could help