Display Matrix{RGB{N0f8} with Genie

Say I have an image in the form of a matrix of RGB values:

img = rand({RGB{N0f8}, 100, 100)

How can I display that in my webapp using Genie, Stipple, etc?

1 Like
# Installation
]add FileIO, ImageIO

# Image Demo App
using Stipple, StippleUI
using Genie.Renderer.Html
using Colors
using FileIO

const IMGPATH = "img/demo.png"

Base.@kwdef mutable struct Dashboard1 <: ReactiveModel
    imageurl::R{String} = IMGPATH
    newimage::R{Bool} = false
end

function restart()
    global model

    cd(@__DIR__)
    model = Stipple.init(Dashboard1(), debounce=1)

    on(model.newimage) do x
        # if button value was set to false, ignore
        x == false && return nothing

        # get and save the new image
        image = rand(RGB, 100, 100)
        save(joinpath(@__DIR__, "public", IMGPATH), image)

        # add a time anchor to the imagepath in order to trigger a reload in the backend
        model.imageurl[] = "/$IMGPATH#$(Base.time())"

        # reset the button
        model.newimage[] = false
    end
end

function ui()
    dashboard(vm(model), [
        heading("Image Demo"),

        row(cell(class="st-module", [
            btn("New Image", "", @on("click", "newimage=true")),
            quasar(:img, src=:imageurl, spinner__color="white", style="height: 140px; max-width: 150px")
        ])),

        footer(class="st-footer q-pa-md", cell([
            img(class="st-logo", src="/img/st-logo.svg"),
            span(" &copy; 2020 - Happy coding ...")
        ]))
    ], title = "Image Demo") |> html
end

route("/", ui)

Genie.config.server_host = "127.0.0.1"

restart()
up(open_browser = true)

Please also see my post in Stipple

1 Like

Alternatively to storing the image as file and updating the file-url with an anchor, you can also embed the image as base64 encoded picture:

using Base64, Images

model.imageurl[] = "data:image/png;base64,$(stringmime(MIME("image/png"), image))"

If you prefer a very light-weight packge, you can also use ImageShow, which Atom and VSCode use for image display. Very small and very large pictures are handled differently for better display.

using Base64: stringmime
using ImageShow

base64png(img) = "data:image/png;base64,$(stringmime(MIME"image/png"(), img))"

I had problems evaluating @btime on the conversion (Benchmarktools crashed), so I defined alternatively

using Base64: base64encode
using ImageShow

show_png(io, x) = show(io, MIME"image/png"(), x)
base64png(img) = "data:image/png;base64,$(base64encode(show_png, img))"

model.imageurl[] = base64png(image)
1 Like

Thank you so much @hhaensel!

I got everything to work (using StippleUI#master though), but I had Julia segfault in both cases, via Images and ImageShow:

signal (11): Segmentation fault
in expression starting at none:0
jl_object_id_ at /home/yakir/julia/src/builtins.c:366 [inlined]
jl_object_id_ at /home/yakir/julia/src/builtins.c:343
type_hash at /home/yakir/julia/src/jltypes.c:981
typekey_hash at /home/yakir/julia/src/jltypes.c:993 [inlined]
lookup_type at /home/yakir/julia/src/jltypes.c:585
inst_datatype_inner at /home/yakir/julia/src/jltypes.c:1157
jl_inst_arg_tuple_type at /home/yakir/julia/src/jltypes.c:1424
arg_type_tuple at /home/yakir/julia/src/gf.c:1823 [inlined]
jl_lookup_generic_ at /home/yakir/julia/src/gf.c:2350 [inlined]
jl_apply_generic at /home/yakir/julia/src/gf.c:2402
_writecallback at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:422
unknown function (ip: 0x7f7dad4b8468)
png_write_data at /home/yakir/.julia/artifacts/b0cf018645acb4148a4234c116eb74799ef63fc1/lib/libpng16.so (unknown line)
png_write_sig at /home/yakir/.julia/artifacts/b0cf018645acb4148a4234c116eb74799ef63fc1/lib/libpng16.so (unknown line)
png_write_info_before_PLTE at /home/yakir/.julia/artifacts/b0cf018645acb4148a4234c116eb74799ef63fc1/lib/libpng16.so (unknown line)
png_write_info at /home/yakir/.julia/artifacts/b0cf018645acb4148a4234c116eb74799ef63fc1/lib/libpng16.so (unknown line)
png_write_info at /home/yakir/.julia/packages/PNGFiles/CQUsD/gen/libpng/libpng_api.jl:92 [inlined]
#_save#15 at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:409
_save##kw at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:344
#13 at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:326
maybe_lock at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:37 [inlined]
#save#12 at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:323
save at /home/yakir/.julia/packages/PNGFiles/CQUsD/src/io.jl:315
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
do_apply at /home/yakir/julia/src/builtins.c:672
jl_f__apply_latest at /home/yakir/julia/src/builtins.c:722
#invokelatest#2 at ./essentials.jl:707
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
do_apply at /home/yakir/julia/src/builtins.c:672
invokelatest at ./essentials.jl:706
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
do_apply at /home/yakir/julia/src/builtins.c:672
#save#6 at /home/yakir/.julia/packages/ImageIO/9APVD/src/ImageIO.jl:43
save##kw at /home/yakir/.julia/packages/ImageIO/9APVD/src/ImageIO.jl:38
inner at ./essentials.jl:710
unknown function (ip: 0x7f7dad4b5fac)
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
do_apply at /home/yakir/julia/src/builtins.c:672
jl_f__apply_latest at /home/yakir/julia/src/builtins.c:722
#invokelatest#2 at ./essentials.jl:711
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
do_apply at /home/yakir/julia/src/builtins.c:672
invokelatest##kw at ./essentials.jl:706
#save#33 at /home/yakir/.julia/packages/FileIO/TyKdX/src/loadsave.jl:233
save##kw at /home/yakir/.julia/packages/FileIO/TyKdX/src/loadsave.jl:217
#show#13 at /home/yakir/.julia/packages/ImageShow/9kpaq/src/showmime.jl:43
unknown function (ip: 0x7f7dad4b36bb)
show at /home/yakir/.julia/packages/ImageShow/9kpaq/src/showmime.jl:28 [inlined]
show_png at /home/yakir/tmp/a.jl:14
unknown function (ip: 0x7f7dad4b3045)
#base64encode#3 at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Base64/src/encode.jl:209
base64encode at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Base64/src/encode.jl:206 [inlined]
base64png at /home/yakir/tmp/a.jl:15
#8 at /home/yakir/tmp/a.jl:85
unknown function (ip: 0x7f7dadf78597)
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
do_apply at /home/yakir/julia/src/builtins.c:672
jl_f__apply_latest at /home/yakir/julia/src/builtins.c:722
#invokelatest#2 at ./essentials.jl:707 [inlined]
invokelatest at ./essentials.jl:706 [inlined]
#setindex!#1 at /home/yakir/.julia/packages/Stipple/cnufP/src/Stipple.jl:83
setindex! at /home/yakir/.julia/packages/Stipple/cnufP/src/Stipple.jl:70 [inlined]
update! at /home/yakir/.julia/packages/Stipple/cnufP/src/Stipple.jl:113 [inlined]
update! at /home/yakir/.julia/packages/Stipple/cnufP/src/Stipple.jl:109
unknown function (ip: 0x7f7dadf783a5)
#7 at /home/yakir/.julia/packages/Stipple/cnufP/src/Stipple.jl:190
unknown function (ip: 0x7f7dadf74cac)
match_channels at /home/yakir/.julia/packages/Genie/SN5kE/src/Router.jl:572
route_ws_request at /home/yakir/.julia/packages/Genie/SN5kE/src/Router.jl:192
handle_ws_request at /home/yakir/.julia/packages/Genie/SN5kE/src/AppServer.jl:246 [inlined]
handle_ws_request at /home/yakir/.julia/packages/Genie/SN5kE/src/AppServer.jl:245 [inlined]
#24 at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Distributed/src/macros.jl:127 [inlined]
run_work_thunk at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
unknown function (ip: 0x7f7dadf66768)
#remotecall_fetch#142 at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:379
remotecall_fetch at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:379
#remotecall_fetch#146 at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421
remotecall_fetch at /home/yakir/julia/usr/share/julia/stdlib/v1.6/Distributed/src/remotecall.jl:421
unknown function (ip: 0x7f7dadefca16)
setup_ws_handler at /home/yakir/.julia/packages/Genie/SN5kE/src/AppServer.jl:229
#7 at /home/yakir/.julia/packages/Genie/SN5kE/src/AppServer.jl:87 [inlined]
#upgrade#8 at /home/yakir/.julia/packages/HTTP/TVRTz/src/WebSockets.jl:160
upgrade at /home/yakir/.julia/packages/HTTP/TVRTz/src/WebSockets.jl:142 [inlined]
#6 at /home/yakir/.julia/packages/Genie/SN5kE/src/AppServer.jl:86
macro expansion at /home/yakir/.julia/packages/HTTP/TVRTz/src/Servers.jl:406 [inlined]
#13 at ./task.jl:406
unknown function (ip: 0x7f7dadefc5c4)
jl_apply at /home/yakir/julia/src/julia.h:1703 [inlined]
start_task at /home/yakir/julia/src/task.c:839
unknown function (ip: (nil))
Allocations: 53377365 (Pool: 53359124; Big: 18241); GC: 56
Segmentation fault

Ultimately, I plan to load the frames, so your base64 implementation sounds very attractive. I’ll move on with saving the frames to disk, but it would be great to avoid that.

Thanks again!

It’s probably related to an issue in PNGFiles.

I’m not totally sure, but you might try the following workaround:

mime(s) = MIME(s)
base64png(img) = base64encode.(show, Ref(@noinline(m)("image/png")), [img])

maybe

base64png(img) = base64encode.(show, Ref(MIME"image/png"()), [img])

is sufficient

1 Like

[got resolved] Hmm, still got segfaults… No worries, I’ll keep trying things. Looks like Tim attributed it to https://github.com/JuliaIO/ImageIO.jl/issues/17, so hopefully this will get resolved in the next few days :slight_smile: (because Tim).