# real-time camera acquisition and processing, howto

**URL:** <https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398>\
**Category:** Performance\
**Created:** [February 3, 2019, 1:38pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398 "2019-02-03T13:38:22Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [February 3, 2019, 1:38pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/1 "2019-02-03T13:38:22Z")

</div>

This is the same issue that is covered by my [stackoverflow post](https://stackoverflow.com/questions/54469038/real-time-camera-input-to-julia-lang). I have searched to best of my ability online, especially this discourse and stackoverflow.

I am new to Julia. I want to use the REPL to learn, and then demo its capability.

I am on Ubuntu 16.04, installed the Julia 1.03 LTS build (downloaded binaries), and installed all packages from the package handler system from the REPL interface. Only two packages added so far: VideoIO and ImageViewer.

I want to show real-time processing of a camera feed. I am using a usb camera (video4linux2).  
First impression of running VideioIO.viewCamera(): immediate crash. The interface to ImageViewer seems broken. I fixed that by making the following:

in “myViewCam.jl”:

```julia
module myViewCam
export myView

import VideoIO, ImageView;

function myView() 
        camera = VideoIO.opencamera();
        buf = VideoIO.read(camera);
        guidict = ImageView.imshow(buf);
        while !eof(camera)
            VideoIO.read!(camera, buf);
            ImageView.imshow(guidict["gui"]["canvas"], buf);
            sleep(0.00001);
        end
    end
end

```

which I run by:

```julia
include("myViewCam.jl");
myViewCam.myView();

```

This will display the videostream in a window, with initially 3 or 4 second lag. The lag evens out over time, and lands steadily on about 0.5 seconds. If I increase the sleep duration in the loop, the CPU can go lower useage, say 90% or so without frame-loss, but it then does not reduce the lag time over time.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [February 3, 2019, 6:38pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/2 "2019-02-03T18:38:59Z")

</div>

Is there a concrete question here? What sort of error/crash do you get when running just `VideoIO.viewCamera()`?

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [February 3, 2019, 6:46pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/3 "2019-02-03T18:46:49Z")

</div>

I think the question is how to avoid the lag, so you can display the video in real time. I think either @ssfrr or @ianshmean were working with real-time video capture at some point?

---

<div class="post-metadata">

**Author:** ![cdsousa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdsousa/32/215553_2.png) [@cdsousa](https://discourse.julialang.org/u/cdsousa)\
**Post date:** [February 3, 2019, 9:41pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/4 "2019-02-03T21:41:21Z")

</div>

Hi, I’m not actually answering anything but I just want to say that I used to use a tiny interface to OpenCV capture C API I created and am now making it public at [https://gist.github.com/cdsousa/5c707f767cd766bb9df79f837b90a1a9](https://gist.github.com/cdsousa/5c707f767cd766bb9df79f837b90a1a9). It may help (or not).

---

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [February 4, 2019, 1:14pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/5 "2019-02-04T13:14:19Z")

</div>

@Sukera, I am not sure I can edit the post, the GUI tells me it will discard my post. The question is: **How do I achieve low lag, and low cpu camera acquisition, so I can do real-time video processing in Julia?**

---

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [February 4, 2019, 1:14pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/6 "2019-02-04T13:14:23Z")

</div>

@Sukera, the behaviour of the `viewcam()` (I spelled it wrong before) is as follows. It pops up a single frame of the camera in a window, and then crashes with the following error message:

> ERROR: MethodError: no method matching imshow(::Pair{String,Any}, ::PermutedDimsArray{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2,(2, 1),(2, 1),Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2}}; flipx=true, interactive=false)

The code in original viewcam attempts to retrieve a canvas from an ImageView call but it doesnt work. I am guessing it is due to an outdated interface to ImageView, dunno really. I just hacked something together that seems right in the code i posted. Seeing its my first adventure into Julia, I am sure I am doing all things wrong.

---

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [February 4, 2019, 1:14pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/7 "2019-02-04T13:14:25Z")

</div>

@mkborregaard and @cdsousa, thanks for helping. I have been thinking about the solution of trying to hook this up with OpenCV (common ground for me). I have heard that Julias strenghts include easy and efficient combination with c++ libs. Holding off with that for a little while longer, to see if this approach works out first.

---

<div class="post-metadata">

**Author:** ![ianshmean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ianshmean/32/216042_2.png) [@ianshmean](https://discourse.julialang.org/u/ianshmean)\
**Post date:** [February 4, 2019, 4:02pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/8 "2019-02-04T16:02:48Z")

</div>

> [@mkborregaard](#):
>
> I think the question is how to avoid the lag, so you can display the video in real time. I think either @ssfrr or @ianshmean were working with real-time video capture at some point?

Indeed! This is the `Makie` approach I arrived at, with dummy data generated by `rand!()` and an efficient downsampling approach for lower resolution previews.

Also there’s a preview scalefactor speed testing example below which demonstrates that on my setup downsampling is only beneficial at lower than a ~0.5 preview scale factor

```julia
using Random #For generating fake live video data
using Makie, ImageTransformations, Interpolations, Colors, FixedPointNumbers

nframes = 500
previewscalefactor = 0.2

img = rand(UInt8,1536,2048) #Raw image
scene = Scene(resolution = (size(img,2),size(img,1)),colormap=Reverse(:Greys)) #Reverse colormap to match normal greyscale image representation 

preview_size = (round(Int,size(img,1)*previewscalefactor), round(Int,size(img,2)*previewscalefactor))
#Lower resolution preview, with efficient downsampling
buff = zeros(N0f8, preview_size[1],preview_size[2])
itp = interpolate!(reinterpret(N0f8, img), BSpline(Linear()))
#Set up Makie scene for preview
hmap = heatmap!(scene, buff, show_axis = false, scale_plot = false)[end]
display(scene)

#Test dummy image capture with preview
t = @elapsed for i = 1:nframes
    rand!(img) # aquire new data into raw image array
    ImageTransformations.imresize!(buff, itp)
    hmap[1] = buff
    yield()
end
println("Video generation & live preview at ",previewscalefactor," scale: ",round(nframes/t,digits=1)," FPS")

```

Video generation & live preview at 0.2 scale: 286.0 FPS

**Testing preview scale factor**

```julia
using Random #For generating fake live video data
using Makie, ImageTransformations, Interpolations, Colors, FixedPointNumbers

nframes = 500
img = rand(UInt8,1536,2048) #Raw image

#First test dummy image capture only
t = @elapsed for i = 1:nframes
    rand!(img) # aquire new data into raw image array
end
println("Video generation: ",round(nframes/t,digits=1)," FPS")

for previewscalefactor in [1,0.5,0.25,0.1]
    scene = Scene(resolution = (size(img,2),size(img,1)),colormap=Reverse(:Greys)) #Reverse colormap to match normal greyscale image representation 
    if previewscalefactor != 1
        #Lower resolution preview, with efficient downsampling
        preview_size = (round(Int,size(img,1)*previewscalefactor), round(Int,size(img,2)*previewscalefactor))
        buff = zeros(N0f8, preview_size[1],preview_size[2])
        itp = interpolate!(reinterpret(N0f8, img), BSpline(Linear()))
        #Set up Makie scene for preview
        hmap = heatmap!(scene, buff, show_axis = false, scale_plot = false)[end]
    else
        hmap = heatmap!(scene, img, show_axis = false, scale_plot = false)[end]
    end

    display(scene)

    #Test dummy image capture with preview
    t = @elapsed for i = 1:nframes
        rand!(img) # aquire new data into raw image array
        if previewscalefactor !=1
            ImageTransformations.imresize!(buff, itp)
            hmap[1] = buff
        else
            hmap[1] = img
        end
        yield()
    end
    println("Video generation & live preview at ",previewscalefactor," scale: ",round(nframes/t,digits=1)," FPS")
end

```

Video generation: 2137.1 FPS  
Video generation & live preview at 1.0 scale: 67.0 FPS  
Video generation & live preview at 0.5 scale: 60.8 FPS  
Video generation & live preview at 0.25 scale: 207.1 FPS  
Video generation & live preview at 0.1 scale: 536.4 FPS

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [February 4, 2019, 4:36pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/9 "2019-02-04T16:36:27Z")

</div>

@Stefan_Karlsson What a marvellous thread. I have been working with installing VideoIO on and off for two years I think. Look at the documentation - it says there is a simple high level interface.  
I may be out of order here, but I think your question is really saying - there is a general purpose utility for reading video, but it turns out to be slow. How then do we access hardware devices with Julia? I guess that becomes much more specific to the hardware and OS, and as @cdsousa says probably involves wrapping some C code.  
But lets not give up - Julia is growing.

---

<div class="post-metadata">

**Author:** ![ianshmean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ianshmean/32/216042_2.png) [@ianshmean](https://discourse.julialang.org/u/ianshmean)\
**Post date:** [February 4, 2019, 4:41pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/10 "2019-02-04T16:41:01Z")

</div>

> [@johnh](#):
>
> What a marvellous thread

I agree. Given it’s relevance here I thought I’d plug that we started up a #video slack channel in the julia slack a few weeks back, focused on building out `VideoIO`’s functionality primarily.

Also, @samuelpowell’s `spinnaker.jl` package for PointGrey cameras is a nice example of camera interaction that’s currently being built out. I’ve even got that working right out of the box on the ARMv8 build of 1.0.3 on a raspberry pi style board (Rock64) [GitHub - samuelpowell/Spinnaker.jl: A Julia interface to the FLIR/PointGrey Spinnaker SDK](https://github.com/samuelpowell/Spinnaker.jl)

---

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [February 4, 2019, 8:28pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/11 "2019-02-04T20:28:55Z")

</div>

@ianshmean and @johnh, thanks for your kind words. Note that what I want are 2 things:

1. low-latency camera acquisition and
2. low drain on the hardware resources.

What I am hoping to achieve (before either it or I am too old 🙂 ), is to port my [Matlab real-time optical flow toolbox](https://se.mathworks.com/matlabcentral/fileexchange/44400-tutorial-and-toolbox-on-real-time-optical-flow?w.mathworks.com) into Julia, with a free license.

---

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [February 4, 2019, 8:43pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/12 "2019-02-04T20:43:24Z")

</div>

Regarding work on VideoIO, here is a wish list from me:

1. low latency and low cpu load acquistion for video4linux devices (maybe same issue with directShow, havent tried windows)
2. streaming video from IP cameras. Just motion jpeg over tcp/IP, http would be a good feature for starters.
3. genicam, (usb3vision and GigE vision cameras)

if VideoIO was to support genicam devices (usb3vision and gigE vision cameras). It would really open up for a new crowd of engineers and researchers.

---

<div class="post-metadata">

**Author:** ![stemann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stemann/32/4030_2.png) [@stemann](https://discourse.julialang.org/u/stemann)\
**Post date:** [March 19, 2019, 7:35pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/13 "2019-03-19T19:35:08Z")

</div>

See also [Cameras.jl](https://github.com/IHPSystems/Cameras.jl) - we should have a generic camera interface – similar to GenICam, but even more generic, to include consumer hardware, such that a `GenICamCamera` is a sub-type of an abstract `Camera`.

Low latency and low CPU load is obviously needed - even for consumer cameras: In order to achieve this, dynamic memory allocation must be avoided - one solution: [ResourcePools.jl](https://github.com/IHPSystems/ResourcePools.jl/tree/feature/pooled_resource)

In a short while, I will publish a Basler Pylon Camera-implementation achieving low-latency by statically allocating images - based on [pylon\_julia\_wrapper](https://github.com/IHPSystems/pylon_julia_wrapper), [Cameras.jl](https://github.com/IHPSystems/Cameras.jl) and [ResourcePools.jl](https://github.com/IHPSystems/ResourcePools.jl/tree/feature/pooled_resource).

Here’s a sample of acquiring images (using the raw wrapper) - in an asynchronous-friendly way – i.e. without blocking the calling thread unnecessarily: [pylon\_julia\_wrapper/grab\_async.jl at master · IHPSystems/pylon\_julia\_wrapper · GitHub](https://github.com/IHPSystems/pylon_julia_wrapper/blob/master/samples/grab_async.jl)

---

<div class="post-metadata">

**Author:** ![dehann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dehann/32/2003_2.png) [@dehann](https://discourse.julialang.org/u/dehann)\
**Post date:** [March 31, 2020, 8:21pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/14 "2020-03-31T20:21:48Z")

</div>

For reference, similar to answer/comment in original Stack Overflow post – for anyone looking for a Video4Linux.jl wrapper: [GitHub - Affie/Video4Linux.jl: Julia interface for video4linux](https://github.com/Affie/Video4Linux.jl)

Using this wrapper in a multi-threaded way should be doable.

---

<div class="post-metadata">

**Author:** ![Stefan\_Karlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefan_karlsson/32/18745_2.png) [@Stefan\_Karlsson](https://discourse.julialang.org/u/Stefan_Karlsson)\
**Post date:** [November 11, 2020, 8:56am UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/15 "2020-11-11T08:56:52Z")

</div>

This shuts down any potential demos of how efficient Julia is for real-time video-processing. Still hopefully waiting.

---

<div class="post-metadata">

**Author:** ![SCD](https://avatars.discourse-cdn.com/v4/letter/s/6f9a4e/32.png) [@SCD](https://discourse.julialang.org/u/SCD)\
**Post date:** [February 1, 2021, 1:44pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/16 "2021-02-01T13:44:56Z")

</div>

@stemann Can you please tell me where to find the a summary of latest capability of Julia Pylon wrap?

Also, I’m trying to move away from Matlab for machine vision applications and just graduated from Jane’s intro tutorial. I would like to know the current capability with regard to porting julia applications in compiled (executable) form. Can you please advise some resources or where to look for?

Thanks.

---

<div class="post-metadata">

**Author:** ![stemann](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stemann/32/4030_2.png) [@stemann](https://discourse.julialang.org/u/stemann)\
**Post date:** [May 7, 2021, 12:40pm UTC](https://discourse.julialang.org/t/real-time-camera-acquisition-and-processing-howto/20398/17 "2021-05-07T12:40:13Z")

</div>

Cf. [https://github.com/IHPSystems/PylonCameras.jl](https://github.com/IHPSystems/PylonCameras.jl)