The problem was that I had uninstalled Julia 1.4.2, but the kernel still existed. The solution is to run
jupyter kernelspec list
to get the list of kernels and the paths to them.
Next had to run
jupyter kernelspec uninstall unwanted-kernel
in order to uninstall the old kernels. In my case, remove Julia-1.4 kernel so Julia-1.5 opens by default