How to use the REPL to speed up modular development by dynamically replacing function definitions

I have a question about workflows in Julia, or alternatively, some aspects of how the REPL can be used.

Allow me to start by describing how I am currently constrained.

I recognized that what I am about to describe will be considered by many here as a sub-optimal situation.

Here is my situation:

There is a server in the cloud somewhere which I have to connect to in order to do my work. It uses a strange version of ssh, so while I can connect a single terminal session to this machine, what I cannot do is scp files to and from it. (This will become important later.)

This server does not have git installed, and it cannot be installed.

I am using a Windows machine to connect to this server. The Windows machine does have git installed. This is how I modify the repository, from the Windows PC.

There is a problem here. While I can make changes to the code in the repository, and push them from my Windows PC, how can I test the changes on the production server?

Here’s how we do it:

We copy the files one by one and paste them into an instance of vim running on the production server.

This creates a very slow development loop. To test a change:

  1. Copy the file from VS Code on Windows.
  2. Exit the julia REPL in terminal session connected to remote prod server
  3. rm <target.jl>
  4. vim <target.jl>
  5. Press i to enter insert mode
  6. Press CTRL+V to paste buffer contents
  7. Press ESC, :wq to save the file and quit vim
  8. Reload the julia REPL
  9. julia> include("entryPoint.jl:)
  10. julia> Main.EntryPointModule.main()
  11. Get feedback from the stack trace if it errors etc
  12. Modify file on Windows machine using VS Code
  13. Repeat

If I could cut out some of these steps, there is potentially a significant time saving to be obtained.

Since Julia is dynamic, and modules can be replaced dynamically, my thoughts are that there might be a way to copy and paste some code into the REPL, and replace the existing function or module definitions.

It would be simpler to do this for individual functions, rather than whole modules.

I tried something, but wasn’t able to get it to work.

Here’s a simple example to test:

Create this file

# MyModule.jl
module MyModule
export exampleFunction
function exampleFunction()
    println("hello world")
end
end

Create a second file which uses the first

# MainModule.jl
module MainModule
include("MyModule.jl")
using Main.MainModule.MyModule
function main()
    exampleFunction()
end
end

Use MainModule.jl from the REPL

julia> include("MainModule.jl")

julia> Main.MainModule.main()
hello world

Example change

Imagine that we want to make the following change to exampleFunction

function exampleFunction()
    println("goodbye world")
end

I would not expect copying and pasting just these lines into the REPL to work, because it will create a name exampleFunction inside the Main module, not the namespace Main.MainModule.MyModule.

I tried to create it using this code, but this did not work.

module Main.MainModule.MyModule
function exampleFunction()
    println("goodbye world")
end
end

The error is:

ERROR: ParseError:
# Error @ REPL[7]:1:12
module Main.MainModule.MyModule
#          ╙ ── invalid identifier

So I tried something else

julia> module MainModule
       module MyModule
       function exampleFunction()
           println("goodbye world")
       end
       end
       end
WARNING: replacing module MainModule.
Main.MainModule

This also didn’t work, because it breaks this, which previously did work…

julia> Main.MainModule.main()
ERROR: UndefVarError: `main` not defined

It makes sense that this no longer works, because I have replaced the existing module named MainModule with something which now does not contain the main() function. (At least this is what seems to be happening.)


Is what I am trying to do possible?

I’m not sure I correctly understand all your constraints, but have you considered Revise, specifically includet?

Revise.jl doesn’t help in the way I was hoping for. I still have to modify the files on disk.

If, rather than exiting the REPL, I were to enter shell mode in order to modify the files, then Revise.jl might remove a few steps.

Looking back at my numerical list, working in this way would remove the need to perform steps: 2 and 9. I don’t think it eliminates any others. The big problem here is needing to modify the files on disk, I think.

I will try it.

Ok maybe I don’t understand what you’re doing here (how would you expect your code to change if you don’t modify the file?) but my point is simply you can do

julia> using Revise

julia> write("testfile.txt", "foo() = 1")
9

julia> includet("testfile.txt")

julia> foo()
1

julia> write("testfile.txt", "foo() = 2")
9

julia> foo()
2

without leaving Julia, as your code will be re-loaded on the fly.

Allow me to provide an even simpler example - which is no longer totally accurate to the situation, because I will remove all the moduleness to the example.

# somefile.jl
function exampleFunc()
    println("hello world")
end

Now imagine I do this in the REPL.

julia> include("somefile.jl")

julia> exampleFunc()
hello world

Next, imagine I want to dynamically change the behavior of exampleFunc. Imagine I found that it had a bug in it of some kind, and I wanted to change the behavior.

julia> function exampleFunc()
    println("goodbye world")
end

julia> exampleFunc()
goodbye world

Here, I have been able to change the behavior of a function called exampleFunc by copying and pasting some code from another machine into the REPL running on the prod server.

Can you configure Vim to @edit exampleFunc() on the remote?

2 Likes

This will work exactly as you show, in this example. You could have also done includet from Revise.jl as @nilshg suggested, and it would update if you changed the function definition in somefile.jl.

I am also a bit unclear about what exactly you’re trying to do and what your limitations are - we might be encountering an xy problem. But here are a couple of caveats:

You mentioned modules, and the behavior would be a little different there, unless you import the name. For example, if you have

module MyMod

export foo

foo() = println("hello")

end

If you have called foo before, and then try to update it from the REPL, that won’t work, but that’s about julia namespaces, rather than something unique to your situation:

julia> module MyMod
       export foo
       foo() = println("hello")
       end
Main.MyMod

julia> using .MyMod

julia> foo()
hello

julia> foo() = println("goodbye")
ERROR: invalid method definition in Main: function MyMod.foo must be explicitly imported to be extended

Instead you can do import MyMod: foo, rather than (or in addition to) using MyMod then update it as expected, or

julia> MyMod.foo() = println("goodbye")

julia> foo()
goodbye

I like your idea, but I cannot get it to work:

julia> include("MainModule.jl")
Main.MainModule

julia> Main.MainModule.main()
hello world

julia> MainModule.MyModule.exampleFunction() = function exampleFunction()
       println("bye")
       end

julia> Main.MainModule.main()
(::var"#exampleFunction#1") (generic function with 1 method)

It is obvious what I am doing wrong here? I am unsure…

I think the wrong part is this:

MainModule.MyModule.exampleFunction() = function exampleFunction()
       println("bye")
end

which has to be

MainModule.MyModule.exampleFunction() = begin
       println("bye")
end

The wrong variant makes the call exampleFunction() return a function, not execute the function body.

1 Like

If you want to do it that way, the function keyword needs to come first,

function MainModule.MyModule.exampleFunction()
    println("bye")
end

You have a lot of very strong and somewhat unusual restrictiong that are going to make it extremely hard to work effectively with Julia code

Really just a single terminal? You can’t open multiple connection / windows? That would allow you edit in one window, and have the REPL in the other window.

Alternatively (and even preferably), can you run a multiplexer (screen or tmux) on the remote? With that, you could do just all of your development on the remote. Edit your code there in vim, and run it in the REPL. Once the project is finished, you could figure out a way to copy it off the server onto your windows machine.

Can you do port forwarding with your SSH connection? That would allow you to run a Jupyter server on the remote, where you could either use a notebook in place of the REPL, or even open files/terminals via a web browser.

That’s a pretty extreme restriction. Why can’t it be installed? Couldn’t you compile it locally? Maybe a replacement like Dulwich could be helpful.

Julia’s package management is very closely tied to git. And, in fact: doesn’t Julia contain a git implementation or JLL wrapper? So, this would be my followup question to the community: Is it possible to clone/update git repos from inside the Julia REPL on a system that does not have git installed?

3 Likes

Also, just throwing it out there in case someone is having similar issues – VS Code has an extension for remote connections: Developing on Remote Machines using SSH and Visual Studio Code

You essentially have the window open on your local machine, but the actual project/files/terminals all live on the remote server.

For this particular use case it might not work due to the restrictions (not sure if the extension relies on scp under the hood), but it might be worth a try. Small caveat though: The extension will take up space and resources on the server side to work properly which may or may not be a problem. It’s an incredibly convenient feature though!


Another option might be to find some way of synchronizing the project directory (semi-)automatically between the local and remote host to cut out some of the manual copy-paste steps (again, probably doesn’t work if scp doesn’t work).

What comes to mind is something like rsync, Cyberduck, WinSCP, probably others?

1 Like

Yep, it appears to be a single bidirectional character feed. Opening more than one session just shows the same text on two terminal interfaces.

git can’t be installed. Neither can screen or tmux.

Nope, also not permitted. Port 22 and ssh are completely firewalled off.

I can compile locally, but it’s pointless because of dependencies on other systems. For example, I do not have any data locally, and there is a very strong coupling between the code and data.

It is my intention to change this, but it’s months of work and not something I can do in overnight. Not even in the next few weeks. It’s a big job.

It’s not permitted. No sudo. Even if I could figure out a way around this, I’d get fired for deliberatly breaking the rules.

This is a potentially interesting idea, but since git depends on port 22, which is firewalled off, I think this is a non-starter.

Well, given the extreme restrictions of your system, I don’t think you’ll be able to have an effective workflow. The one thing (which was already suggested) might be to use @edit/edit from inside the REPL, so that you can run vim without having to close the REPL and losing state there. I would definitely try to exploit that by working exclusively on the remote as much as possible. You’ll want to edit your code there, rather than having the code locally and trying to transfer it to the remote with every change.

You might also want to raise issues internally about how severely the restrictions of the production system limit productivity. Maybe it would be possible to set up a less restrictive “staging area” for testing? To be honest, I’ve never seen anything as locked down as you seem to be describing, and I’ve worked on some pretty locked-down military Cray machines.

Do the admins of the production server have any recommendations on what you should do?

If it were this easy to resolve, it would have already been resolved.

You can certainly run git from Julia without a system git installation, using the Git package (and indirectly the Git_jll package and the artifacts it downloads):

julia> import Git

julia> run(`$(Git.git()) --version`);
git version 2.46.2

Git only relies on port 22 for repositories with ssh:// URLs. But I guess your firewall blocks http, https and git’s native port (9418) as well. Cf. What firewall ports need to be open to allow access to external git repositories? - Server Fault

2 Likes

Yes, it will block all of these AFAIK

How do any of the Julia packages you’re presumably using get installed onto the production server? Or, is this a situation where a fixed set of Julia packages was manually installed in “airgapped” mode by an admin, and you cannot install or update any Julia package?

If you can install Julia packages, that should give you a way to deploy your user code via the same mechanism. After all, you can always package your code.

2 Likes

I am not sure about this.

One problem would be the repository is not organized into nice packages which could be uploaded to a package repo.

I could modularize it, and indeed I intend to do so, but this is not a quick job. It will take weeks. (Made slowed by a lack of testing procedure. Any mistakes I make will only show up when the changes are shipped to production. Which will create a serious problem for us and our clients.)

Another problem would be we do not have our own internal server which hosts a package repo. I could probably create one, but again there will be a huge number of security related issues to get permission to do this.

If we can’t do something simple like use git to move the code from one machine to the target server, I would assume deploying a package hosting server is going to be just as problematic.

There’s also the issue that everything is currently based on AWS. If AWS doesn’t offer an “off the shelf” Julia PKG repo hosting service, it may just be a “no” anyway.

We obviously can’t upload packages to an open source repo for obvious reasons.

So if I’ve gathered this correctly:

  • data is on the production server, so code has to be tested there
  • REPL indeed compiles and runs Julia code on the server
  • EDIT: can only interact with server via one terminal?
  • your IDE VS Code is on your local PC, and you’re moving Julia source code from your PC to the server

which seems like a system set up to test larger code edits without running any code between uploads, which is reflected in the workflow of evaluating all the code in a fresh REPL session and calling a main function.

You could write and run those smaller tweaks in the REPL itself instead of step 12, but that obviously does not change the source file and is cumbersome for large functions that you can’t up-arrow to anyway after include. Revise was suggested for coupling source file changes and REPL tweaks, but that needs you to edit files in the same place you have a REPL session. Julia functions can write to files, but that’s obviously not as ergonomic as a file editor or an IDE, which is why base Julia has edit/@edit for opening editors (worth mentioning that vim is known to need configuration for Revise to work properly). I don’t know if that’s usable because I think you’re saying that a REPL and a file editor can’t be open at the same time on the server, not sure though.

Alternatively, is there any way you can get or generate local data? It doesn’t have to be the actual data, just structured similarly enough for local development without limitations, except perhaps OS dependence, between the real tests on the server.

1 Like