It works fine with Plots.jl’s pgfplotsx() backend:
pgfplotsx()
using LaTeXStrings, Plots; pgfplotsx() plot([1,2,3], label=L"||x-y||^2")