Something like this (mat is the number matrix, M is the color matrix generated with the num2col function above:
julia> mat = [0 1 2;2 0 3;1 3 1]
julia> M = num2col.(mat);
julia> heatmap(mat,c=M,xaxis=nothing,yaxis=nothing,legend=nothing,size=(400,400))
Produces
: