Realised that the "julia.editor": "code" setting was run before the startup.jl script, which I guess makes sense. This meant that ENV["JULIA_EDITOR"] was set to code first, but then startup.jl overwrote it with vim.
I instead set my system editor (EDITOR and VISUAL in linux) to be vim, and didn’t set anything in startup.jl or VSCode. This seem to result in the same behaviour as the manual if ... else ... in the startup.jl, but feels cleaner.
I also realised that the problem on ssh remotes was a known bug, so just have to hope it gets fixed 