Maybe compare with this recent post, the deletion functionality of Makie is not yet that well figured out:
A GridLayout only describes where to put certain elements, and multiple objects can be in the same location, that’s why your approach doesn’t work. It’s not like a matrix where you overwrite things.