so I just figured out how to do it after a lot of digging.
The issue is that in RecipesPipeline.jl there is group_labels = sort(collect(unique(v)))
Now in theory there are CategoricalArrays for this purpose,
however unique(ctg::CategoricalArray) returns an ordinary Vector{String} instead of a new CategoricalArray.
Defining this solves the problem in this case
function Base.unique(ctg::CategoricalArray)
l = levels(ctg)
newctg = CategoricalArray(l)
levels!(newctg, l)
end