How To Use Lua and Julia to Create Plugins for Neovim?

Hi all,

@ExpandingMan and I had a fun conversation about perhaps one day creating neovim plugins that use Julia. Neovim plugins tend to really be written in Lua or Fennel these days and it got me wondering, how do I call Julia from Lua? I suppose creating this bridge may be the first step. I found LuaCall.jl from @ararslan but that goes Julia → Lua. I want Lua → Julia instead.

Thoughts?

~ tcp :deciduous_tree:

Actually this isn’t really what I meant on slack, I was (rather less ambitiously) referring to the possibility of multiple plugins that might need to use Julia for one reason or another (e.g. LSP) all using the same run-time. Having a single plugin that uses Julia is painful (though this is significantly ameliorated as of 1.9), but once you have one, at least in principle, having n additional Julia plugins should be trivial.

I don’t think calling Julia from lua is going to be much fun. One would presumably have to use the Julia run-time C API. On the other hand, post 1.9, the idea of writing nvim plugins directly in Julia is no longer completely crazy, though it’s still really pushing Julia’s use cases due to the latency. The first step would be wrapping nvim’s C API. There is this but presumably something modernized is in order, and I doubt that package ever even considered the possibility of using Julia to write plugins since, again, that was completely untenable before 1.9.

1 Like

Oh interesting – I actually was thinking similarly, but then I decided to take it a step further with using Julia in plugins. :grimacing: Actually, I spoke with the creator of Vhyrro, the creator of Neorg, and they gave me some tips as follows:


Do you want to run the julia code asynchronously in your code or have your plugin wait for the julia binary to complete its work? For either method, here are the two approaches:

      local on_exit = function(obj)
        print(obj.code)
        print(obj.signal)
        print(obj.stdout)
        print(obj.stderr)
      end

      -- Run asynchronously
      vim.system({'julia', '--blah'}, { text = true }, on_exit)

      -----------------------------------------------------------------

      -- Run synchronously
      local obj = vim.system({'julia', '--blah'}, { text = true }):wait()
      -- { code = 1, signal = 0, stdout = '', stderr = 'ERROR: unknown option `--blah`' }

To return values from Julia to Lua is a bit tricky. Here are a few methods:

  1. With the above method, this would be challenging. you’d have to print the return values to stdout as some sorta JSON-encoded struct and then decode that on the neovim side. very unpleasant

  2. if you want to be perfectly integrated then you would probably need an FFI boundary, and that will be a lot of work

  3. another thing you could theoretically do is connect to neovim via RPC and communicate over that, which is the preferred method nowadays. See this for details – especially the first two sections: Remote_plugin - Neovim docs

The most effective approach will probably be 3 using RPC. The first thing you’d want to do is you compile your julia code into a binary or sysimage. then you run:

local handle = vim.fn.jobstart({"your-binary"}, { rpc = true })

This spawns your julia code as a child process of neovim with rpc communication. One important thing to note is that this is messagepack rpc communication , so not JSON-RPC. Find something that supports msgpack :slight_smile: Next up, from the lua side you can send RPC requests to julia, and julia can send RPC requests to neovim.

local response = vim.rpcrequest(handle, "your-julia-function-name", "your-arguments", "here")
print(response) -- you can now use it from lua!

now what you have to research is how to mark your julia functions as “rpc callbable” and how to start an RPC connection over stdio in julia


Looks like there are some packages that could enable sending MessagePack feedback over RPC using this:

So, seems like all the machinery is here… Fascinating.

The stdout / stdin approach is what I used for GitHub - kdheepak/JuliaFormatter.vim: A (N)Vim plugin for formatting Julia code using JuliaFormatter.jl.

I wrote it in vimscript (because I wanted to support vim as well) but did use jobstart for nvim.

See this file for the vimscript jobstart example and this file for the Julia side.

I think the Julia code could be abstracted into a nicer Julia package for making it work with Vim and neovim.

If you want to write a neovim only plugin though, I’d definitely recommend the Neovim.jl plugin @ExpandingMan shared: GitHub - bfredl/Neovim.jl: Neovim client for Julia. Neovim.jl uses MsgPack.jl, and that effectively means Julia and Neovim are separate processes exchanging data through MsgPack via RPC.

For completeness sake, I think there’s a few other unexplored options to write Julia plugins for Neovim.

  1. Using PackageCompiler, build a shared library with @ccallable functions that then can be loaded in neovim as a shared library.
  2. Using PackageCompiler and LuaCall, build a shared library that exposes a Lua module interface that then can be imported in neovim as a Lua module.
  3. Build a shared library in another language as a shim (e.g. Rust?) that loaded Julia using the Julia C API and exposed a Lua module interface that can be imported in neovim as a Lua module.
  4. Use Lua’s ffi package in Neovim to use Julia C API and start a Julia process and execute Julia code.

These approaches are possibly what @ExpandingMan is getting at? If you did the first two, I think you’d have to use PackageCompiler. The last two options though you can get away with without using PackageCompiler.

The 3rd option would require writing some code in another language that can compile to a shared library. I wrote about this approach in a blog post that you can read here. In Rust, there’s already jlrs and nvim-oxi, so even some combination of that would work.

All 4 of these options will result in the Julia process being loaded in the same process as neovim. This means if Julia segfaults, I believe neovim will also segfault and crash. For that reason, I think it will require comprehensive testing and probably be viable only when not running user provided code (imho).

2 Likes

I tried to write neovim plugins in Julia with mixed results. Having tried some of the alternatives already mentioned, I have a couple of questions and a few thoughts.

  • What type of plugins do you have in mind, or what plugins would benefit from being written in Julia?

  • What’s the advantage of having multiple plugins in the same process? (Or is this just a ttfx work-around?)


For “general purpose” plugins, I don’t think it makes sense to use anything other than Lua. In 2021 (before nvim 0.5), it wasn’t entirely clear how the plugin ecosystem would pan out. Nowadays it’s pretty clear that the neovim ecosystem is Lua. Even if you solve RPC (which neovim does ok), the problems of distribution, versioning and environment setup make using other languages too inconvenient.

For Julia specific plugins, the options are not great:

  • LuaCall.jl is just an experiment. It’s not really usable.
  • Anything that requires compiling Julia sounds optimistic. In 2021, when I was first looking into this, I read that small Julia binaries were just around the corner. It’s 2023 and I keep reading the same thing. A compiled library also wouldn’t make sense for plugins that need a compiler to evaluate code.
  • Neovim.jl generates its API bindings using metaprogramming. I don’t know if that’s amenable to precompilation, but if it is, then updating the project to be properly precompiled could be a big step forward, since latency has been the main blocker for usable plugins.

I hope I don’t sound overly negative, but to be honest, I don’t think writing neovim plugins in Julia is a good idea. After I failed trying the same, I started to focus on editor independent stuff that didn’t require a julia runtime (i.e. tree-sitter-julia).

Last time I used LS.jl, it was really bad (and I haven’t used it since), but I think improving it would be the best way to improve Julia’s DX in neovim. For most languages I get by with just the language server, and I wish I could do the same with Julia.