Looks like this is underflow due to the fixed-point precision:
julia> cols[2] - cols[1]
RGB{N0f8}(0.008,0.008,0.008)
julia> norm(cols[2] - cols[1])
0.0
julia> norm(RGB{Float64}(cols[2] - cols[1]))
0.00784313725490196
In particular, the problem stems from the underflow:
julia> N0f8(0.008)^2
0.0N0f8
Arguably, this should be changed — since the output of norm is a Float64, this computation should be performed by promoting operands to the output precision. Or at the very least some scaling should be done to prevent spurious overflow/underflow (similar to what norm and hypot do for floating-point vectors).
(Update: Filed issue ColorVectorSpace.jl#183.)