Errors using Makie plots with Observable metadata Vectors of variable length

I would like to generate a Makie animation that has a variable number of points and polygons per frame, and am running into some odd behavior. I am using scatter() and poly() with Observables holding the data points, and updating the Observables in a loop. The polygons should each have custom transparency, which is held in an Observable of Vector{RGBA}, and the scatter points should each have custom markersize, held in an Observable of Vector{Float}. Because the length of the data and metadata vectors must always be consistent, I am manually updating each Observable Vector, and then only notifying them together after the new data has been loaded.

I ran into a bug where the poly() update fails if the number of objects in the list changes:

ERROR: LoadError: Metadata array needs to have same length as data.
                    Found 6 data items, and 4 metadata items

I had been working on Makie v0.17.3, and discovered that this bug seems to be corrected in v0.18.1, for which the poly plot works as expected. However, the scatter plot which had worked fine in v0.17.3 is now failing with a similar error in v0.18.1.

Is this an odd confluence of related Makie bugs? Am I setting this plot up incorrectly? Or is varying the length of metadata vectors simply not supported? Any suggested workaround or alternate approaches to accomplishing this?

Reproducible example:

using GLMakie, Makie
using Colors

polys = Observable([rand(Point2f,3) for nn=1:2])
colors = Observable(rand(RGBA,2))
pts = Observable(rand(Point2f,2))
sz = Observable(rand(2))
update = Observable(true)

on(update) do _
    NN = rand(1:10)      #only get an error if number of objects is changed...
    #update plot data, but dont notify Plots until all Observables are updated
    polys.val = [rand(Point2f,3) for nn=1:NN]
    colors.val = rand(RGBA,NN)
    pts.val = rand(Point2f,NN)
    sz.val = 10 .* rand(NN)

f = Figure();
ax = Axis(f[1,1])

# scatter(), variable number of points, custom markersize per point
# Works on v0.17.3, breaks on v0.18.1
Makie.scatter!(ax, pts, markersize=sz) 

# poly(), variable number of polys, custom RGBA per poly
# Works on v0.18.1, breaks on v0.17.3
Makie.poly!(ax, polys, color=colors)


for mm=1:10

I thought I decent workaround to this might be to delete the old plot object and generate a new one for each frame. I first tried this per the documentation:

delete!(ax, sp)
sp = scatter!(ax, pts, markersize=sz) 

But this appears to corrupt the axis itself, e.g. all axis tick labels become grey boxes:

(Edit: this rendering issue seems to be specific to GLMakie, and does not occur with CairoMakie)

I also tried manually deleting the plot from the scene, e.g.

deleteat!(ax.scene.plots, findfirst(ax.scene.plots .== sp))

But this does not actually remove the old plot until an explicit redraw (e.g. display(figure)) which is rather slow.