So I’m not sure what precisely you’re doing, but something like the following should just work:
Do also take a look at Chris’ video, if you haven’t already.
Do you change your structs? I think that should be the only change that requires restarting Julia currently. For that @pfitzseb tip should work. (And it is known that it’s currently suboptimal, but there are at least some workarounds). Otherwise a minimal working example (MWE) with your code would probably be helpful.
Just a word on compilation in Julia (which has nothing to do with that issue but if you are unaware it’s good to know anyways). It doesn’t really work like “ask Juno to recompile the whole file” or “module was re-compiled”. (Apologies if you just misspoke, just wanted to make sure.)
Functions (and only those) are only compiled when they are called (at runtime!), not when they are defined (with Shift+Enter). That’s why the first run of something might take a little longer (important for benchmarks) because there is some compilation happening in addition to the calculation. On the second run of a function the already compiled function is used. If you change and redefine your function it needs to be compiled again (but, again, this will only happen when you are calling it).
This, by the way, is the reason why Julia can be dynamically typed (like Python) but still reach speeds of C: at runtime all the types are know. So you can use that knowledge to compile your function for the types you got and voila – amazing speeds.
1 Like
If you are not making a package then you don’t need Revise.
Just put everything in a function in a separate file. Then, have a script file which includes that: include("name or address of file"), and then runs your function from script: myFun().
For debugging in Matlab way I recommend GitHub - JuliaDebug/Infiltrator.jl: No-overhead breakpoints in Julia
And regarding my code, yes for developing you are always in developer mode. Using is when the user has added the package and doesn’t want to change it, so they don’t need Revise.
Here’s a super simple test case that illustrates the issue I am having.
First, two files, a module and a main app:
MyMod.jl:
module MyMod
export MyFunc
function MyFunc()
println(“1”)
end
println(“MyMod loaded”)
end
MainApp.jl
import MyMod
function MyMainApp()
MyMod.MyFunc()
end
println(“MainApp loaded”)
Here’s the sequence I do:
-
Start Juno/Atom, open to my Project. -
Start Julia. (Nothing is in startup.jl for this test) -
Add the project directory to the path:
push!(LOAD_PATH,pwd()) -
Select MainApp.jl window and press the Play icon:
MyMod loaded
MainApp loaded -
Call the function in MainApp:
MyMainApp()
1
-
Modify MyMod.MyFunc to print “2” instead of “1”. -
Ctrl-Shift-Enter to reload MyMod.
WARNING: replacing module MyMod.
MyMod loaded
-
Call the function in MainApp:
MyMainApp()
1 ← Change not picked up
-
Try loading MainApp again, with Play button:
ERROR: LoadError: importing MyMod into Main conflicts with an existing identifier
in expression starting at C:\Users\brad\Documents\Erskine\Jtest\MainApp.jl:1
And now I have to restart Julia.
I understand that, in this case, I could have just hit Shift-Enter to reload just MyMod.MyFunc. That works for me. But what if I have made broad changes to the file, or have changed a half-dozen constants at the top of the file, and want to reload the whole file? This is something I do a lot in development, as I change the distribution of work between functions, factor functions, that sort of thing. I’m not sure which functions or definitions I’ve modified, just “several of them”.
Is there some way to tell Juno “re-evaluate this entire file” rather than just one function?
I’ll dig into the video. I started watching it and it seemed to be a tutorial about programming differential equations, wasn’t sure if I had the right one.
I should have added that I get pretty much the same behavior when using Revise.
Here’s a super simple test case that illustrates the issue I am having.
First, two files, a module and a main app:
MyMod.jl:
module MyMod
export MyFunc
function MyFunc()
println(“1”)
end
println(“MyMod loaded”)
end
MainApp.jl
import MyMod
function MyMainApp()
MyMod.MyFunc()
end
println(“MainApp loaded”)
Here’s the sequence I do:
-
Start Juno/Atom, open to my Project. -
Start Julia. (Nothing is in startup.jl for this test) -
Add the project directory to the path:
push!(LOAD_PATH,pwd()) -
Select MainApp.jl window and press the Play icon:
MyMod loaded
MainApp loaded -
Call the function in MainApp:
MyMainApp()
1
-
Modify MyMod.MyFunc to print “2” instead of “1”. -
Ctrl-Shift-Enter to reload MyMod.
WARNING: replacing module MyMod.
MyMod loaded
-
Call the function in MainApp:
MyMainApp()
1 ← Change not picked up
-
Try loading MainApp again, with Play button:
ERROR: LoadError: importing MyMod into Main conflicts with an existing identifier
in expression starting at C:\Users\brad\Documents\Erskine\Jtest\MainApp.jl:1
And now I have to restart Julia.
I understand that, in this case, I could have just hit Shift-Enter to reload just MyMod.MyFunc. That works for me. But what if I have made broad changes to the file, or have changed a half-dozen constants at the top of the file, and want to reload the whole file? This is something I do a lot in development, as I change the distribution of work between functions, factor functions, that sort of thing. I’m not sure which functions or definitions I’ve modified, just “several of them”.
Is there some way to tell Juno “re-evaluate this entire file” rather than just one function?
I’ll dig into the video. I started watching it and it seemed to be a tutorial about programming differential equations, wasn’t sure if I had the right one.
Using Revise should work pretty well in this case, provided that you includet your main file.
julia> push!(LOAD_PATH, pwd())
4-element Array{String,1}:
"@"
"@v#.#"
"@stdlib"
"/tmp"
julia> using Revise
julia> includet("mainApp.jl")
[ Info: Precompiling MyMod [top-level]
julia> myMainApp()
1
# Edit MyMod.jl
julia> myMainApp()
2
That being said, I would advise you to create a project even for such simple tasks: this can be done very quickly with Pkg.generate(), and it can save you lots of time later when using Revise for example.
julia> using Pkg; Pkg.generate("MyApp")
Generating project MyApp:
MyApp/Project.toml
MyApp/src/MyApp.jl
UUID("0b020b34-c6c0-11e9-12a0-6b6b60a3ffb6")
julia> cd("MyApp")
julia> Pkg.activate(".")
"/tmp/MyApp/Project.toml"
# Populate src/MyApp.jl and src/MyMod.jl
# use Revise before your module
julia> using Revise
julia> using MyApp
[ Info: Precompiling MyApp [0b020b34-c6c0-11e9-12a0-6b6b60a3ffb6]
julia> myMainApp()
1
# edit MyMod.jl
julia> myMainApp()
2
The project directory tree would look like this:
.
├── Project.toml
└── src
├── MyApp.jl
└── MyMod.jl
with the following file contents (very similar to yours):
==> src/MyApp.jl <==
module MyApp
export myMainApp
include("MyMod.jl")
myMainApp() = MyMod.myFunc()
end # module
==> src/MyMod.jl <==
module MyMod
myFunc() = println(2)
end
3 Likes
You’re right, this test case does work with Revise. Thanks for checking.
My real app is not behaving the same way. I’ll see if I can boil my real app down to a simple example that preserves the “Revise isn’t doing the trick” behavior.
Okay, I have the simple example that makes Revise not work. New stuff in red. Everything else about running the test is the same.
Now if you change the println(“1”) and re-run MyMainApp(), the module doesn’t get re-loaded, and the output doesn’t change.
If you comment out the new “using” line, Revise works.
MyMod.jl:
module MyMod
export MyFunc
using MyOtherMod
function MyFunc()
println(“1”)
end
println(“MyMod loaded”)
end
MyOtherMod.jl:
module MyOtherMod
end
MainApp.jl
import MyMod
function MyMainApp()
MyMod.MyFunc()
end
println(“MainApp loaded”)
Several folks including you have urged me towards the package generator. I’ll practice on my test app and see if making it into a package fixes the problem.
I’m trying the simple Pkg test case you so kindly wrote out.
I am having trouble with MyMod. I can’t figure out how to add MyMod to the package dependencies. I tried Pkg.resolve(). I also looked in the online docs about using Pkg.add, but this isn’t a package from “out there”, it’s a module right here in the same folder. See red below. (The module for MyApp is called JTestP.)
Any idea what I’m doing wrong?
Thanks so much to everyone who’s helping me through this.
That being said, I would advise you to create a project even for such simple tasks: this can be done very quickly with Pkg.generate(), and it can save you lots of time later when using Revise for example.
> using Pkg; Pkg.generate("JTestP")
Generating project JTestP:
JTestP/Project.toml
JTestP/src/JTestP.jl
UUID("0b020b34-c6c0-11e9-12a0-6b6b60a3ffb6")
> cd("JTestP")
> Pkg.activate(".")
Activating environment at `C:\Users\brad\Documents\Erskine\JTestP\Project.toml`
# Populate src/MyApp.jl and src/MyMod.jl
# use Revise before your module
> using Revise
> using JTestP
[ Info: Precompiling JTestP [15bcfb71-f55c-4a56-83ac-b63e563f4b8f]
ERROR: LoadError: ArgumentError: Package JTestP does not have MyMod in its dependencies:
- If you have JTestP checked out for development and have
added MyMod as a dependency but haven't updated your primary
environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with JTestP
I was able to work around this using LOAD_PATH.
I had tried LOAD_PATH, but I had it pointed to the root project folder rather than to ./src.
So now I have the test app running as a Package.
But I am still seeing the problem with Revise not seeing the change to MyMod, if the “using MyOtherMod” line is in there.
FYI for all those who have been trying to help me, it’s much appreciated.
After watching videos and studying the Juno docs, I’ve hit on a workflow which is reasonably efficient for me.
I treat my main program as sort of a live script that I work with in the Juno editor, really basically a very smart shell. I can do the equivalent of stepping through it by evaluating cells and lines in whatever order I please.
I can also use the debugger to jump into a function that I want to investigate. Then if I decide I need to change that function, I abort the debugger, edit the function, then hit shift-enter to recompile it.
Now I pop back to my livescript and try it all again. I can do a lot of quick iterations without having to restart Julia.
I feel like maybe I’m on the right track now in terms of what the intended workflow is, at least with Juno.
If the assembled crowd also feels like I’m on the right track, I might try to write down a little bit of a step by step, for people to at least throw darts at. Hopefully it can help the next person that comes along with a Matlab mindset.
I’m still curious as to why Revise doesn’t seem to handle the little test case I put together. Is that a bug, or just some misunderstanding on my part? I’d be interested to get to the bottom of that.
In this workflow, do you have all your code in the “Main” module (that is to say, you never declare a new module to put some of your code)?
Yes, this workflow does work, and has the advantage of requiring minimal setup. The only thing is that I don’t think it scales very well.
This workflows also looks very similar to what people do with notebooks: put all the code in a notebook and evaluate it cell by cell. Have you tried jupyter to see if it would suit your needs?
AFAIU, Revise.includet is only meant to handle cases where all the source code you’re tracking is grouped into a file (i.e. I don’t think it recursively tracks the dependencies of your sources). Your “little test case” is not so little: it has code divided into several modules spanning several files. At this stage, using fully-fledged projects is recommended (and presumably when you get to having multiple modules calling one another, the initial setup cost of generating a project and setting it up is negligible w.r.t the time spent developing code in it). Revise is really designed to handle code structured in this way.
One thing that could act as a hint that you’re doing something wrong: for normal use, you should never have to manually fiddle with LOAD_PATH.
As an example, here is what your example with two modules would look like as a project:
src/JTestP.jl
module JTestP
export myMainApp
include("MyOtherMod.jl")
include("MyMod.jl")
# Note the leading "."
using .MyMod
myMainApp() = myFunc()
end # module
src/MyMod.jl
module MyMod
export myFunc
# Note the leading ".."
import ..MyOtherMod
function myFunc()
println("MyMod: 1")
MyOtherMod.otherFunc()
end
end
src/MyOtherMod.jl
module MyOtherMod
otherFunc() = println("MyOtherMod: 1")
end
Note that Revise works “out of the box”, and you don’t have to manually set LOAD_PATH. Also note that all sub-modules that are defined inside the top-level module of your project (JTestP in this case) have to be referred to with a relative path when using or importing them.
julia> using Pkg; Pkg.activate(".")
"/tmp/JTestP/Project.toml"
julia> using Revise
julia> using JTestP
julia> myMainApp()
MyMod: 1
MyOtherMod: 1
# Edit src/MyMod.jl
julia> myMainApp()
MyMod: 2
MyOtherMod: 1
# Edit src/MyOtherMod.jl
julia> myMainApp()
MyMod: 2
MyOtherMod: 2
3 Likes
Surprising … and for the record: Revise works for me on your example (using Julia 1.1.0, so I’m a bit outdated).
Revise should handle everything you’re asking for without breaking a sweat. I’ve merged that documentation pull request, so now you can see it here: Revise usage: a cookbook · Revise.jl
If you still have trouble making Revise work, make sure you read that “limitations” page.
If I had to guess, a large fraction of your troubles previously might have been from not realizing that Revise can’t handle adding new files to a previously-loaded module, e.g., adding include("mynewfile.jl") to an existing project requires a restart.
The Juno Shift+Enter is awesome, but for what you’re asking for:
But what if I have made broad changes to the file, or have changed a half-dozen constants at the top of the file, and want to reload the whole file? This is something I do a lot in development, as I change the distribution of work between functions, factor functions, that sort of thing. I’m not sure which functions or definitions I’ve modified, just “several of them”.
the answer is: use Revise. (And don’t use Shift+Enter.) Some other things that Revise gives you are:
- better integration with the debugger: without Revise, Juno will take you to the wrong line number if you’ve deleted a bunch of unrelated lines higher up in the same file, even if you’re religious about hitting Shift+Enter. Similarly, the line numbering in stacktraces won’t be correct in the face of modifications without using Revise.
- features like method deletion, which might not matter to you now but may once you start writing more sophisticated code involving multiple methods that provide different implementations for different types of inputs
- automatic incorporation of Pkg updates or toggling between two versions of the code with
git stash, if those changes are within the scope of what Revise can handle.
5 Likes
Ahhh! There are two really big tips in here! However, Revise is still not working for me (end of this message).
AFAIU, Revise.includet is only meant to handle cases where all the source code you’re tracking is grouped into a file (i.e. I don’t think it recursively tracks the dependencies of your sources). Your “little test case” is not so little: it has code divided into several modules spanning several files. At this stage, using fully-fledged projects is recommended (and presumably when you get to having multiple modules calling one another, the initial setup cost of generating a project and setting it up is negligible w.r.t the time spent developing code in it). Revise is really designed to handle code structured in this way.
So the first one is easy. I have modified my JTestP code to match yours. However, after doing Pkg.activate(“.”), I get this error:
using JTest3
ERROR: ArgumentError: Package JTest3 not found in current path:
- Run
import Pkg; Pkg.add("JTest3")to install the JTest3 package.
I shouldn’t have to “add JTest3”, should I?
So I tried the LOAD_PATH trick, but I get the same error.
How to build a project?
Something tells me I haven’t properly set up a “full-fledged project”?
How do I do that? I looked at the Project documentation, but I can’t figure out where to start. Where do the UUID’s come from? Is there a Project builder I should use? Do I just make stuff up and sling the Project and Manifest files manually? Will that fix my “not found” problem?
Status of Revise
Until I get that sorted out, I also tried this with the main file not being a module – I just did includet(“MainApp.jl”) instead. It seems like that should work, right?
But when I do that, Revise does not pick up changes to MyMod.jl or MyOtherMod.jl:
using Revise
includet(“MainApp.jl”)
MyMod2 loaded
MyMod loaded
JTest3 loaded
MyMainApp()
MyMod: 1
MyOtherMod: 3
Edit MyMod.jl
MyMainApp()
MyMod: 1
MyOtherMod: 3
To be clear, here is MainApp.jl now. The other files are identical to your example, with the addition of the “loaded” printouts.
include(“MyOtherMod.jl”)
include(“MyMod.jl”)
import .MyMod
function MyMainApp()
MyMod.MyFunc()
end
println(“JTest3 loaded”)
To use your new package, you either need to Pkg.add it (or, better, Pkg.develop it in some environment. Or you need to activate the package’s environment before using it (>pkg activate . in the package directory).
When working with packages, you should never have to change the LOAD_PATH (that’s kind of the point of having packages deal with the dependencies).
There are lots of details, which I keep writing up here as I figure them out. See section 7 on packages. Disclaimer: My understanding of how things work is highly imperfect. But that perhaps makes the document helpful for beginners.
Aaah… success!!!
By “full-fledge project”, you must have meany “make it a package using Pkg.generate()”? (Also I read Tim Holy’s latest doc updates and was steered again towards package-land.)
I do have a subtree where I’ve tried that. So I tried importing the code you suggested into that tree.
And now Revise works! Hooray hooray hooray! I’ll try this next with my “real” app and verify that it works for it as well.
So it looks like the secret to success, if you’re using multiple files, is to
· Organize them as a package, with the top-level function in a module
· include the sub-modules in the main module (not just import or using)
· Then use import or using with “.submodule” instead of just “submodule”
If that is indeed the secret, that would be a great thing to add to the Revise mini-doc that Tim Holy is maintaining.
Thanks, activate worked!
Great news, again. This workflow is now working for my full application as well. Revise is the bomb.
Thanks to everyone for all the help.
4 Likes