I think your plot has too few points for your liking (you are only plotting on the 21 integers from -10 to 10).
E.g. it does not look that bad:
julia> using Distributions, KernelDensity, Plots
julia> f1 = kde(randn(10^3));
julia> x1 = range(-4,stop=4,length=1000)
julia> y1 = pdf.(Ref(f1),x1);
julia> plot(x1,y1)
Produces this:
