real-time camera acquisition and processing, howto

This is the same issue that is covered by my stackoverflow post. I have searched to best of my ability online, especially this discourse and stackoverflow.

I am new to Julia. I want to use the REPL to learn, and then demo its capability.

I am on Ubuntu 16.04, installed the Julia 1.03 LTS build (downloaded binaries), and installed all packages from the package handler system from the REPL interface. Only two packages added so far: VideoIO and ImageViewer.

I want to show real-time processing of a camera feed. I am using a usb camera (video4linux2).
First impression of running VideioIO.viewCamera(): immediate crash. The interface to ImageViewer seems broken. I fixed that by making the following:

in “myViewCam.jl”:

module myViewCam
export myView

import VideoIO, ImageView;

function myView() 
        camera = VideoIO.opencamera();
        buf = VideoIO.read(camera);
        guidict = ImageView.imshow(buf);
        while !eof(camera)
            VideoIO.read!(camera, buf);
            ImageView.imshow(guidict["gui"]["canvas"], buf);
            sleep(0.00001);
        end
    end
end

which I run by:

include("myViewCam.jl");
myViewCam.myView();

This will display the videostream in a window, with initially 3 or 4 second lag. The lag evens out over time, and lands steadily on about 0.5 seconds. If I increase the sleep duration in the loop, the CPU can go lower useage, say 90% or so without frame-loss, but it then does not reduce the lag time over time.

1 Like

Is there a concrete question here? What sort of error/crash do you get when running just VideoIO.viewCamera()?

1 Like

I think the question is how to avoid the lag, so you can display the video in real time. I think either @ssfrr or @ianshmean were working with real-time video capture at some point?

1 Like

Hi, I’m not actually answering anything but I just want to say that I used to use a tiny interface to OpenCV capture C API I created and am now making it public at https://gist.github.com/cdsousa/5c707f767cd766bb9df79f837b90a1a9. It may help (or not).

1 Like

@Sukera, I am not sure I can edit the post, the GUI tells me it will discard my post. The question is: How do I achieve low lag, and low cpu camera acquisition, so I can do real-time video processing in Julia?

1 Like

@Sukera, the behaviour of the viewcam() (I spelled it wrong before) is as follows. It pops up a single frame of the camera in a window, and then crashes with the following error message:

ERROR: MethodError: no method matching imshow(::Pair{String,Any}, ::PermutedDimsArray{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2,(2, 1),(2, 1),Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2}}; flipx=true, interactive=false)

The code in original viewcam attempts to retrieve a canvas from an ImageView call but it doesnt work. I am guessing it is due to an outdated interface to ImageView, dunno really. I just hacked something together that seems right in the code i posted. Seeing its my first adventure into Julia, I am sure I am doing all things wrong.

1 Like

@mkborregaard and @cdsousa, thanks for helping. I have been thinking about the solution of trying to hook this up with OpenCV (common ground for me). I have heard that Julias strenghts include easy and efficient combination with c++ libs. Holding off with that for a little while longer, to see if this approach works out first.

1 Like

Indeed! This is the Makie approach I arrived at, with dummy data generated by rand!() and an efficient downsampling approach for lower resolution previews.

Also there’s a preview scalefactor speed testing example below which demonstrates that on my setup downsampling is only beneficial at lower than a ~0.5 preview scale factor

using Random #For generating fake live video data
using Makie, ImageTransformations, Interpolations, Colors, FixedPointNumbers

nframes = 500
previewscalefactor = 0.2

img = rand(UInt8,1536,2048) #Raw image
scene = Scene(resolution = (size(img,2),size(img,1)),colormap=Reverse(:Greys)) #Reverse colormap to match normal greyscale image representation 

preview_size = (round(Int,size(img,1)*previewscalefactor), round(Int,size(img,2)*previewscalefactor))
#Lower resolution preview, with efficient downsampling
buff = zeros(N0f8, preview_size[1],preview_size[2])
itp = interpolate!(reinterpret(N0f8, img), BSpline(Linear()))
#Set up Makie scene for preview
hmap = heatmap!(scene, buff, show_axis = false, scale_plot = false)[end]
display(scene)

#Test dummy image capture with preview
t = @elapsed for i = 1:nframes
    rand!(img)     # aquire new data into raw image array
    ImageTransformations.imresize!(buff, itp)
    hmap[1] = buff
    yield()
end
println("Video generation & live preview at ",previewscalefactor," scale: ",round(nframes/t,digits=1)," FPS")

Video generation & live preview at 0.2 scale: 286.0 FPS

Testing preview scale factor

using Random #For generating fake live video data
using Makie, ImageTransformations, Interpolations, Colors, FixedPointNumbers

nframes = 500
img = rand(UInt8,1536,2048) #Raw image

#First test dummy image capture only
t = @elapsed for i = 1:nframes
    rand!(img)     # aquire new data into raw image array
end
println("Video generation: ",round(nframes/t,digits=1)," FPS")

for previewscalefactor in [1,0.5,0.25,0.1]
    scene = Scene(resolution = (size(img,2),size(img,1)),colormap=Reverse(:Greys)) #Reverse colormap to match normal greyscale image representation 
    if previewscalefactor != 1
        #Lower resolution preview, with efficient downsampling
        preview_size = (round(Int,size(img,1)*previewscalefactor), round(Int,size(img,2)*previewscalefactor))
        buff = zeros(N0f8, preview_size[1],preview_size[2])
        itp = interpolate!(reinterpret(N0f8, img), BSpline(Linear()))
        #Set up Makie scene for preview
        hmap = heatmap!(scene, buff, show_axis = false, scale_plot = false)[end]
    else
        hmap = heatmap!(scene, img, show_axis = false, scale_plot = false)[end]
    end

    display(scene)

    #Test dummy image capture with preview
    t = @elapsed for i = 1:nframes
        rand!(img)     # aquire new data into raw image array
        if previewscalefactor !=1
            ImageTransformations.imresize!(buff, itp)
            hmap[1] = buff
        else
            hmap[1] = img
        end
        yield()
    end
    println("Video generation & live preview at ",previewscalefactor," scale: ",round(nframes/t,digits=1)," FPS")
end

Video generation: 2137.1 FPS
Video generation & live preview at 1.0 scale: 67.0 FPS
Video generation & live preview at 0.5 scale: 60.8 FPS
Video generation & live preview at 0.25 scale: 207.1 FPS
Video generation & live preview at 0.1 scale: 536.4 FPS

6 Likes

@Stefan_Karlsson What a marvellous thread. I have been working with installing VideoIO on and off for two years I think. Look at the documentation - it says there is a simple high level interface.
I may be out of order here, but I think your question is really saying - there is a general purpose utility for reading video, but it turns out to be slow. How then do we access hardware devices with Julia? I guess that becomes much more specific to the hardware and OS, and as @cdsousa says probably involves wrapping some C code.
But lets not give up - Julia is growing.

1 Like

I agree. Given it’s relevance here I thought I’d plug that we started up a #video slack channel in the julia slack a few weeks back, focused on building out VideoIO's functionality primarily.

Also, @samuelpowell’s spinnaker.jl package for PointGrey cameras is a nice example of camera interaction that’s currently being built out. I’ve even got that working right out of the box on the ARMv8 build of 1.0.3 on a raspberry pi style board (Rock64) https://github.com/samuelpowell/Spinnaker.jl

3 Likes

@ianshmean and @johnh, thanks for your kind words. Note that what I want are 2 things:

  1. low-latency camera acquisition and
  2. low drain on the hardware resources.

What I am hoping to achieve (before either it or I am too old :slight_smile: ), is to port my Matlab real-time optical flow toolbox into Julia, with a free license.

7 Likes

Regarding work on VideoIO, here is a wish list from me:

  1. low latency and low cpu load acquistion for video4linux devices (maybe same issue with directShow, havent tried windows)
  2. streaming video from IP cameras. Just motion jpeg over tcp/IP, http would be a good feature for starters.
  3. genicam, (usb3vision and GigE vision cameras)

if VideoIO was to support genicam devices (usb3vision and gigE vision cameras). It would really open up for a new crowd of engineers and researchers.

5 Likes

See also Cameras.jl - we should have a generic camera interface – similar to GenICam, but even more generic, to include consumer hardware, such that a GenICamCamera is a sub-type of an abstract Camera.

Low latency and low CPU load is obviously needed - even for consumer cameras: In order to achieve this, dynamic memory allocation must be avoided - one solution: ResourcePools.jl

In a short while, I will publish a Basler Pylon Camera-implementation achieving low-latency by statically allocating images - based on pylon_julia_wrapper, Cameras.jl and ResourcePools.jl.

Here’s a sample of acquiring images (using the raw wrapper) - in an asynchronous-friendly way – i.e. without blocking the calling thread unnecessarily: https://github.com/IHPSystems/pylon_julia_wrapper/blob/master/samples/grab_async.jl

6 Likes

For reference, similar to answer/comment in original Stack Overflow post – for anyone looking for a Video4Linux.jl wrapper: https://github.com/Affie/Video4Linux.jl

Using this wrapper in a multi-threaded way should be doable.

4 Likes

This shuts down any potential demos of how efficient Julia is for real-time video-processing. Still hopefully waiting.

1 Like

@stemann Can you please tell me where to find the a summary of latest capability of Julia Pylon wrap?

Also, I’m trying to move away from Matlab for machine vision applications and just graduated from Jane’s intro tutorial. I would like to know the current capability with regard to porting julia applications in compiled (executable) form. Can you please advise some resources or where to look for?

Thanks.

Cf. GitHub - IHPSystems/PylonCameras.jl