# Stream help in Genie

**URL:** <https://discourse.julialang.org/t/stream-help-in-genie/111531>\
**Category:** General Usage\
**Tags:** question, genie\
**Created:** [March 12, 2024, 4:53pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531 "2024-03-12T16:53:28Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 12, 2024, 4:53pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/1 "2024-03-12T16:53:29Z")

</div>

Hi everyone!

For some background, I am trying to set up a little site using the Genie Framework where I can watch livestreams of my 3D printer as well as start timelapses, and be able to review and download the timelapse videos. I am using a Raspberry Pi Picamera2 python package to control the camera, and I can take timelapses with PyCall relatively easily.

Where I am getting confused is in [this example](https://github.com/raspberrypi/picamera2/blob/main/examples/mjpeg_server_2.py) they are writing the frame to the client in the http server over and over again to implement accessing a livestream as a simple image in the browser. Any ideas how I would begin to mirror this in Genie? Would I just be better off doing something like the [server sent events example](https://juliaweb.github.io/HTTP.jl/stable/examples/#Server-Sent-Events) from HTTP.jl? If so, how would I set that up with `route`?

Any help would be greatly appreciated, as well as any general direction I should go.

---

<div class="post-metadata">

**Author:** ![Pere](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pere/32/214250_2.png) [@Pere](https://discourse.julialang.org/u/Pere)\
**Post date:** [March 13, 2024, 8:33am UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/2 "2024-03-13T08:33:55Z")

</div>

We’ve had someone share an example of capturing images from a webcam with Genie. I don’t know how it’d work on the Pi, but perhaps it could be useful. Here’s the code

> **[Discord - A New Way to Chat with Friends & Communities](https://discord.com/channels/774897545717219328/1141759770542485507/1159873052797063288)**
>
> Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 13, 2024, 12:20pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/3 "2024-03-13T12:20:44Z")

</div>

The link Pere pasted here is one good way to view a live feed of a webcam.

But to get frames from the RPI camera you’ll need to use something else than VideoIO (at least last I tried). You can either use PyCall, or shell out to `libcamera` (which is what I use).

---

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 13, 2024, 2:07pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/4 "2024-03-13T14:07:02Z")

</div>

Thank you for your response. For some reason, the link is not taking me to a text channel in discord. Perhaps I am not part of the right server? Could you possibly paste the code in this thread?

---

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 13, 2024, 2:09pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/5 "2024-03-13T14:09:49Z")

</div>

Yakir,

Interesting, perhaps using `libcamera` would be better than trying to mess with PyCall. Could you include a minimal example of how you shell out to `libcamera` to grab frames from the RPI camera?

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 13, 2024, 2:36pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/6 "2024-03-13T14:36:08Z")

</div>

The following is the code form that post (though this uses `VideoIO`)

```julia
using JpegTurbo, VideoIO
using GenieFramework
@genietools

# a convinience function to convert pixel matrices to jpegs
to_frame = String ∘ jpeg_encode

# open the camera
cam = opencamera()

# use `img` as a container for the frames
const img = Ref(read(cam))

# use `frame` as a container for the jpeg
const frame = Ref(to_frame(img[]))

# fetch fresh frames from the camera as quickly as possible
Threads.@spawn while isopen(cam)
    read!(cam, img[])
    frame[] = to_frame(img[]) # this typically takes less than the camera's 1/fps
    yield()
end

# avoid writing to disk, when the user asks for a frame they get the latest one
route("/frame") do
    frame[]
end

@app Webcam begin 
    @out imageurl = "/frame"
end myhandlers

global model = init(Webcam, debounce = 0) |> myhandlers

# add an (invalid) anchor to the imagepath in order to trigger a reload in the Quasar/Vue backend
Stipple.js_methods(model::Webcam) = """updateimage: async function () { this.imageurl = "frame#" + new Date().getTime() }"""

# have the client update the image every 33 milliseconds (should be changed to the camera's actual 1000/fps or less)
Stipple.js_created(model::Webcam) = "setInterval(this.updateimage, 33)"

# set the image style to basic to avoid the loading wheel etc 
ui() = [imageview(src=:imageurl, basic=true)]

route("/") do 
    page(model, ui) |> html
end

Server.up()

```

and here is how I use `libcamera`:

```julia
function get_buffer_img(w, h)
    w2 = 64ceil(Int, w/64) # dimension adjustments to hardware restrictions
    nb = Int(w2*h*3/2) # total number of bytes per img
    buff = Vector{UInt8}(undef, nb)
    i1 = (w - h) ÷ 2
    i2 = i1 + h - 1
    img = view(reshape(view(buff, 1:w2*h), w2, h), i1:i2, h:-1:1)
    return buff, img
end

struct Camera
    mode::CamMode
    buff::Vector{UInt8}
    img::SubArray{UInt8, 2, Base.ReshapedArray{UInt8, 2, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, Tuple{UnitRange{Int64}, StepRange{Int64, Int64}}, false}
    proc::Base.Process

    function Camera(cm::CamMode)
        w, h, fps = camera_settings(cm)
        buff, img = get_buffer_img(w, h)
        proc = open(`rpicam-vid --denoise cdn_off -n --framerate $fps --width $w --height $h --timeout 0 --codec yuv420 -o -`)
        eof(proc)
        if cm == cmoff
            kill(proc)
        end
        new(cm, buff, img, proc)
    end
end

function Base.close(cam::Camera) 
    kill(cam.proc)
    close(cam.detector)
end

function detect(cam::Camera) 
    read!(cam.proc, cam.buff)
    return cam.img
end

```

---

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 13, 2024, 5:31pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/7 "2024-03-13T17:31:41Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [March 14, 2024, 12:53pm UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/8 "2024-03-14T12:53:24Z")

</div>

If you want to give [Bonito](https://simondanisch.github.io/Bonito.jl/stable/) a try, you can do:

```julia
using FFMPEG_jll, Bonito
# Somehow I can't get VideoIO to work, but FFMPEG works fine...
ffmpeg_cmd = `$(FFMPEG_jll.ffmpeg()) -f dshow -i video="DroidCam Source 3" -preset veryfast -loglevel quiet -vframes 1 -f image2pipe -c:v png -`

App() do session
    bytes = read(ffmpeg_cmd)
    asset = Observable(Bonito.BinaryAsset(bytes, "image/png"))
    @async while true
        bytes = read(ffmpeg_cmd)
        asset[] = Bonito.BinaryAsset(bytes, "image/png")
        yield()
        session.status == Bonito.CLOSED && break
    end
    return DOM.img(src=asset)
end

```

Should work quite similar with `read(video_io_stream)`.

There should be smarter options with streaming an mmpeg video, but it’s more complicated ^^

---

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 16, 2024, 3:55am UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/9 "2024-03-16T03:55:03Z")

</div>

I believe I found your post from 2022 on the raspberry pi forum! Cool!

I am a little new to this. You are reading raw YUV420 pixel format from the RPI camera, do you have to convert that at all before `imageview` would be able to show it on a browser window? How would you manipulate the image matrix to facilitate this?

---

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 16, 2024, 3:56am UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/10 "2024-03-16T03:56:56Z")

</div>

On another note, I noticed that if I used 1920x1080 as my width and height, the resulting `img` view from your `get_buffer_img` function is 1080x1080. Is that purely coincidence and how YUV420 pixel format is structured since the U and V layers are subsampled? I want to make sure I understand this completely.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 16, 2024, 8:30am UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/11 "2024-03-16T08:30:17Z")

</div>

You’re right, the square shape of the frame is just diverging I needed for my application. You can change that.  
But I really just copy pasted my specific setup without any explanation or help… sorry about that. I hope I’ll have some time to expand on it next week.

---

<div class="post-metadata">

**Author:** ![smith-isaac](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/smith-isaac/32/35124_2.png) [@smith-isaac](https://discourse.julialang.org/u/smith-isaac)\
**Post date:** [March 26, 2024, 3:04am UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/12 "2024-03-26T03:04:59Z")

</div>

No need to explain the shape of the image, I was able to figure that part out. I have been able to get the YUV “frames” into an image, got that part all figured out.

Did you use `base64encode` to get your image to show in the `imageview` call? Or did you use the `String(jpeg_encode(img))` method from above? What rates were you getting encoding your image to show in your Genie page? Did you have to make your images smaller?

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 26, 2024, 7:51am UTC](https://discourse.julialang.org/t/stream-help-in-genie/111531/13 "2024-03-26T07:51:17Z")

</div>

> [@smith-isaac](#):
>
> No need to explain the shape of the image, I was able to figure that part out. I have been able to get the YUV “frames” into an image, got that part all figured out.

Awesome!

> Did you use `base64encode` to get your image to show in the `imageview` call?

No.

> Or did you use the `String(jpeg_encode(img))` method from above?

Yeah, I used the `jpeg_encode`, but I never tested the difference between that and `base64encode`. It might be the case that the difference depends on the image size and even complexity…?

> What rates were you getting encoding your image to show in your Genie page?

I didn’t benchmark it all too carefully, but 15 FPS worked fine on WiFi.

> Did you have to make your images smaller?

Absolutely. In my use-case, 300x300 was enough for the image to be useful (even smaller would have worked for me, but it was quick enough, so…).

I’m sure that @sdanisch is right, encoding a stream should be better in all aspects, I just don’t know how to do that. I also plot a bunch of stuff on the frame (markers, object detection, etc) so I would need to fetch frames, plot things, and then stream this out to the client.