Displaying images with text

I’m working on adapting the fast.ai fastbook with a few other people. You can find our work slowly happening via our website or by the Quarto book here.

As part of this, I want to build a little tool to view images with the class printed right above them. To do this, I need to do the following:

  1. Build a white pane that is slightly larger than the underlying image.
  2. Paste the underying image on top of the white pane.
  3. Add text to the top of the white pane.

Anyone have any ideas on how to go about this? I looked at most of the functions in Images.jl and its sattelite packages but I was not able to find any way to compose images or to add text. Happy to be pointed to something if I missed it!

Once you start to wade into composing and annotating images, it’s time to bail out to a plotting library. Makie makes this sort of thing pretty easy:

julia> f = Figure()

julia> for idx in CartesianIndices(imgs)
           ax = Axis(f[idx.I...], aspect=DataAspect(), 
               title="Letter $(rand('a':'z'))")
           image!(ax, imgs[idx])


Should be easy with GMT. The “image” module I think.

If you want to do it without a plotting library, you can simply copy parts of your image matrix around to compose your result.

You can render the text in Cairo, but when doing so I had to overcome a few pitfalls (e.g. for compatibility with Cairo, I just learnt you need to use the Images.RGB24 color type, and documentation for Cairo.jl seems largely absent).

# Suround a bitmap image with a white margin and add text there

using Images
using Cairo
using TestImages

img = testimage("airplaneF16")
margin = 40
title = "U.S. Air Force F-16"

# prepare a white canvas (image size plus margin)
canvas = fill(RGB24(1.0,1.0,1.0), size(img) .+ 2*margin)

# copy image centered onto canvas
       margin+1:margin+size(img,2)] = img

# prepare another canvas for rendering the text into top margin
# (because Cairo draws with X and Y axis fliped relative to Images)
header = fill(RGB24(1.0,1.0,1.0), (size(canvas,2), margin))
c = CairoImageSurface(header)
cr = CairoContext(c)

# render the text
select_font_face(cr, "Sans", Cairo.FONT_SLANT_NORMAL,
set_font_size(cr, 20.0)
move_to(cr, (size(header, 1) - textwidth(cr, title)) / 2,
        (margin + textheight(cr, title)) / 2)
set_source_rgb(cr,0,0,0)  # black
show_text(cr, title)

# copy the x/y flipped top margin back onto main canvas
canvas[axes(header')...] = header'

# Images.save seems to have a bug when receiving Matrix{RGB24}, so convert
Images.save("test.png", RGB{N0f8}.(canvas))

Nice code, Markus.

But, it did remind me of why I found Cairo.lj a bit awkward to write in. All those underscores! :slight_smile:

Code (for Quarto docs)
using Images, TestImages
import Luxor as 🪲

function imagecaption(img, caption = "";
      margin = 80,
      corner_radius = 30,
      fsize = 16)
    w, h = size(img)
    d = 🪲.Drawing(w + 2margin, h + 2margin, :png)
    🪲.box(Point(0, 0), w + 2margin, h + 2margin, corner_radius, :fill)
    🪲.placeimage(img, centered = true)
    🪲.label(caption, :S, midpoint(boxtopleft(), boxtopright()), 
        offset = margin / 2)
    return d

img = testimage("cameraman")
w, h = size(img)
display(imagecaption(img, "I'm Mandy the mandrill", fsize=30))


Okay, I like the ideas from @stillyslalom, @mgkuhn and @cormullion. But I think there is a gap to fill here from image related dataset visualization pov.

  • Makie solution: Makie is a great high level solution but its heavy like quite heavy
  • Cairo+Luxor: Very flexible and powerful but quite low level for workflow needed by @cpfiffer

I think the solution is likely to eventually come in form of FreeTypeAbstraction.jl(for text render) being part of ImageDraw.jl, to support dataset visualization and such at a slight higher level in form of package extension if MosaicViews.jl is available. Very light and just what we need in a dynamic workflow in my opinion. It could also come from ImageView.jl but right now I think it’s bit heavy.

I would have wanted it in MosaicViews.jl itself but it’s supposed to play role for general mosaic of matrix viewer: Feature Request:Labels on the images · Issue #25 · JuliaArrays/MosaicViews.jl · GitHub

1 Like

I second the idea of a Makie or (better due to being lighter?) Cairo/Luxor solution.

I would like a solution to be usable both from a Pluto notebook (easily interpolatable into e.g. HypertextLiteral’s), in a web application (output some web component I can send off as a response from an Oxygen/HTTP method handler) - or just standalone - render something in an SVG, a PDF or (rendered) in a PNG. Nice if it also works in something building on GTK/QML.

I was thinking of this in relation to an ImageAnnotations.jl-based framework - to support both image classification visualisations, object detection visualisations etc. (as extension packages to ImageAnnotations.jl for MakieViews or LuxorViews etc.): Cf. https://julialang.zulipchat.com/#narrow/stream/390029-image-processing/topic/DL.20based.20tools/near/383544112

… but ImageDraw also seems sufficiently low on dependencies to be an excellent place for implementation. Package extensions for ImageDraw sounds just right - but it would be nice to be able to just stick it an ImageAnnotations AnnotatedImage and get a view back.


ImageDraw.jl will need some enhancement to support PDF and SVG text.

Why would ImageDraw.jl + MosaicViews.jl need to support PDF,SVG in data science context :slight_smile: unless going for publication grade perhaps?

In publication grade case as I see it:

  • Makie.jl provides better structure for drawing and the actual plots
  • Luxor.jl provides way more flexibility

I have no idea what would be needed to get SVG/PDF output (but I would assume Cairo would be helpful in that regard).

I may have gone a bit too far with my wishes (to include SVG and PDF) :slight_smile:

1 Like

I wish Makie was lighter and had Luxor level flexibility :grin:


Do note however that AnnotatedImage is not up to speed yet - at all: It needs to be transformed into something like AnnotatedImage{TArray<:AbstractArray{T,N}} to support actual image data - both eager- and lazily-loaded data (and not just a path to some image file as it is currently). Cf. Slack thread on lazy loading of images from May 2023.

I think Makie can do everything you want - and if you’re alreadymaking plots with it, you really don’t need anything else. It does take a while to install, though…


I guess it would be enough for an ImageDraw Makie-extension to depend on MakieCore - to just get a plot recipe back?

(I have yet (how is that possible?! :scream: :laughing:) to have the pleasure of actually using Makie … and Plots … and Luxor.)

Yes, that’s also an option but it could endup becoming something that’s difficult and seem unneccessary to maintain for a few nice features

OK. Difficult to maintain in which way?

(still completely in the dark wrt. Makie etc.)

No plotting package is light, and GMT is no exception but it’s so simple to do it

using GMT

I = gmtread("stefan_rgba.png");
imshow(I, title="Eu Touco Muito", frame=:none)

1 Like