If you write a function label_color(label) which takes your labels — whether they start at zero or whatever else — you can convert your integer matrix to a colour matrix with:
img = [label_color(label) for label in gt_mat]
This array comprehension preserves the shape of the matrix, so produces a matrix of colours.
For example, you might use a dictionary:
julia> using Plots: distinguishable_colors
julia> scheme = Dict(0:28 .=> distinguishable_colors(29));
julia> img = [scheme[label] for label in gt_mat];