I have actually just encountered the same problem. After several tries with different versions of Julia which did not help, I think I have a solution:
First, you need to “use” Julia so that all the usual directories (registries, etc.) are created. Just start the Julia REPL, go to Pkg mode (with “]”) and add any package. After that try starting VS Code again, hopefully after a moment the server will start.