Naive question, but can’t that be worked around by having @breakpoint redefine the function?
1 Like
We added support for stepin targets in the VS Code extension recently. If there are many function calls nested in one line, you can pick into which one you want to step. The UI is a bit hidden in the right click menu.
I think there are really two reasons why the debugging experience for Julia currently is not competitive: 1) speed and 2) a lot of weird small corner cases that are not handled well.
-
is pretty clear, but in practice it pretty much kills the whole experience for me: have something like a CSV file load at the beginning at your script? Well, because that is all written in Julia, things go south already there. Yes, you can try to work your way through it with some clever on off switching of compile mode, but it just is not a good experience. The thing that is really not clear to me is whether the current approach in JuliaInterpreter.jl can be optimized to make things more bearable, or whether we would really need an entirely different approach… The other problem of course is that no one seems to actively work on this, and there are not too many folks that actually can make a dent here, not a simple problem.
-
is also quite important. There are just way too many cases where things don’t work well (at least in the VSCode debugger): generated functions, functions with keyword arguments, tasks and it seems every time I try things I find another one… I think some of these are VS Code specific, some of them would have to be fixed in JuliaInterpreter.jl.
19 Likes
Thank you for this honest explanation on the still existing rough edges on doing debugging. I think this is rather more useful than pretend all is now a sea of roses.
The option to pick into which function one want to stepin is indeed very useful.
Another issue I find less friendly is the fact that when I hover over variables for which I extended Base.:show their content is repeatedly (I mean for each time I hover the mouse on them) printed in the terminal window.
And again, thanks for all the work you guys have done on the debugger front.
That’s fair. And Julia calls to C or Fortran libraries are also fast but there are a lot fewer of them in Julia since most functionality is pure Julia. Automatically switching to compiled mode for built-in functions would be a good direction for someone to explore.
What about Fortran debuggers then? Because I remember those being just as fast (maybe slightly slower) in debug compared to normal mode. But surely they don’t rely on built in functions? Or is that just because everything is compiled already and therefore fast?
And while you have a point in the Julia debugger being as fast as python or matlab debugger, it never feels that way, since all the heavy lifting are done by built in functions. Which means I have the luxury of running all my deep learning in python in debug mode, even the stuff that I know will run without problems and just need 5 days to run.
In any case I am happy we had such a level headed discussion about all this, and I think some valuable knowledge was shared about different perspectives and what it would potentially take to make a debugger that is competitive with what we see in other languages.
Based on all this I will stay out of Julia for the foreseeable future, though I will likely check in again in a year or so and see what have changed and how the wind is blowing then, since I really like the language in general and hope to do all my coding in it eventually.
Fortran (and C, C++) debuggers work differently and run at (nearly) full speed. However, they require you to recompile your program and run it in a different mode. This is possible in Julia as well. That’s how the “original” Gallium debugger for Julia worked. It was very hard to implement and maintain, however, was kind of flaky due to its complexity, and still didn’t satisfy the kind of capabilities that people expect of debuggers in highly dynamic languages. So Debugger.jl was developed instead, using an interpreter, and it has been a massive success. It’s too slow in some cases but it works really reliably and is very feature full. It’s possible that more work on Debugger will be done but that’s only likely if someone funds it or does it themselves. The most promising direction at the moment seems to be automatically running code in compiled mode if it can’t possibly hit a break point.
13 Likes
Which is what MixedModeDebugger.jl prototypes. The prototype seems to work well. Still working out next steps forward with it.
20 Likes
Yeah, MixedModeDebugger.jl looks really promising to me!
From the VS Code side of things, the ideal way forward would be that it gets merged into JuliaInterpreter, and we just update the version of JuliaInterpreter.jl that we use and everything becomes fast 
One constraint to keep in mind is that it is difficult for the VS Code extension to ship very large package dependency trees, so the leaner, the better. I think Cassette.jl would be fine because it doesn’t bring anything else in, but ideally we wouldn’t need to take a dep on Debugger.jl and Revise.jl if we want to bring the functionality into the extension. In general, the dependency order for Debugger.jl and MixedModeDebugger.jl would ideally be reversed, right? I.e. Debugger.jl is the UI, so the underlying debugging engine shouldn’t have a dependency on it?
Yeah, a non-prototype version of MixedModeDebugger.jl would be built into JuliaInterpretter; or at least be at that level of the stack, as a thing debugger front-ends depend on.
Having it outside of JuliaDebugger was for getting something to demo out easier.
FWIW, (unless you want to reduce the features of the debugger) you also want to be able to break on throws and get full stacktraces which makes it harder to figure out when you “can’t possibly hit a breakpoint” (especially considering InterruptException).
Also, often you want to put breakpoints inside a loop, and having things be thousands of times slower just because you set a breakpoint at some hot code seems like a pretty annoying behavior.
Bringing in Cassette and recompiling a lot of stuff on every Julia session also seems non-ideal. I recall minutes of compilation time for pretty simple functions (that probably recurse down in the IO system and recompiles everything there).
So I am personally not that convinced that such a strategy will be of much success.
4 Likes
@kristoffer.carlsson just to make sure I’m not misunderstanding you: you were talking both about the existing compile mode in JuliaInterpreter.jl and the MixedModeDebugger.jl strategy, and think that neither will lead to a really good solution?
Do you (or someone else) have other strategies or ideas?
Well, the “compiled mode” in JuliaInterpreter just turns off the debugger for future calls. So it isn’t really a solution to anything except to work around the slowness for the interpreter in cases where you decide you don’t need it.
Bringing something like WIP: a new serialization format for optimizing & executing lowered IR by timholy · Pull Request #309 · JuliaDebug/JuliaInterpreter.jl · GitHub to a point where it could run something representative might be interesting. However, IIRC, the interpreter right now spends like 20% of the execution time only forming tuples (which is e.g. the return values for the iteration protocol) so if we could do everything else instantly, it would still only be 5x faster. So it needs more optimizations than just executing IR faster.
1 Like
For reference the prototype benchmarks for MixedModeDebugger.jl can be found here.
They look really good, but it is just some micro-benchmarks.
More notes on the pathological case where it is super slow are here.
Also, often you want to put breakpoints inside a loop, and having things be thousands of times slower just because you set a breakpoint at some hot code seems like a pretty annoying behavior.
i am not sure if this is reference to that pathological case in MixedModeDebugger, or to something else.
In the MixedModeDebugger case, mostly noone will see it in practice because it is transition that is slow, which happens at the breakpoint so needs someone to actually hit continue.
Exception to this is conditional breakpoints where the condition is normally false.
I recall minutes of compilation time for pretty simple functions (that probably recurse down in the IO system and recompiles everything there).
That was MagneticReadHead though, right?
MagneticReadHead increased the amount of code generated by about an order of magitude, and julia compile times are quadratic in the length of the code generated.
I have not seen compile times anywhere near that large with MixedModeDebugger, which is a far gentler pass.
Still it is indeed recompiling everything it touches, and it probably does normal cassette breaking of type-inference.
but that seems to be in my microbenchmarks linked above less than the difference between first run and second run in JuliaInterpretter (the whole process finishes in less than the difference between first and second run in julia interpretter)
FWIW, (unless you want to reduce the features of the debugger) you also want to be able to break on throws
I agree anything like this that compiles when there is no breakpoint, more is going to reduce features.
Especially features like break-on-throw.
Maybe not Gallium style, idk about that.
I am not sure what is the actual features people really care about in a debugger though.
One thing that always surprises me is how much people like @bp, and how many people use that as the only way they set breakpoints.
I always assumed being able to set breakpoints without editting the code was a key feature.
(this suggests mixed mode with Infiltrator.jl style is a strong candidate for how to improve things.)
But this just goes to show how little I know about what people want in a debugger.
Maybe most people are OK with break-on-through being disable-able for greater speed.
2 Likes
When using Python basically the only times I ran debugger were after an exception got thrown. After typing %debug into the notebook it allows to traverse the callstack of the last exception, examine values at any level, type and execute expressions in that context. Unfortunately, this is not available in Julia, and I’m not sure if even possible without major slowdown…
3 Likes
For us (the VS Code team) it is simple: we want to implement the full debug adapter protocol, so that gives us a pretty well defined feature set. We probably have something like 80%-90% of that implemented based on JuliaInterpreter.jl, so the functionality in JuliaInterpreter.jl is really excellent in terms of features.
I think for an IDE debugger the situation is super simple: folks add breakpoints by clicking in the left column of the editor, and then long nothing, and then maybe they use other ways to add breakpoints 
I can see how the @bp feature might be more convenient than specifying a filename and line on the command line, or specifying a function breakpoint, but I think as soon as you have the ability to just click in your editor to set a breakpoint, those other options become very distant second options. So I think from the VS Code side of things, the infiltrator approach is not really a good option, we do need the ability to easily set breakpoints by filename/line because that is the backend for the most common UI feature we have.
I think at the end of the day the command line debugger Debugger.jl is super cool, but probably targets more of a niche audience, and the vast majority of users will end up using the debugger via VS Code (or some other IDE). That gives us a pretty decent and well defined target: I think we should try to get the debugger into a state that allows us to have a debugging experience in VS Code that is comparable to other languages in there.
I’m going to show a very short demo of the upcoming notebook support in VS Code in the IDE talk on Wed. While not in that demo yet , we will have full debugging integration into the notebook experience before it ships that will enable all the things you mention.
17 Likes
I don’t see why mixed mode with a boundary at dependencies that aren’t being debugged can’t be made to work as well as debuggers in other languages where that boundary is a language boundary. It seems fine to default to only interpreting calls in packages that have breakpoints set. Sure you can’t stop at an error deep inside some dependency or stdlib, but most users don’t want to—it’s much more useful to drop them into the debugger at the point where their code made a call that eventually led to an error. If I’m a normal user I do not want to be stepping through every function call inside of some API provided by Base or any other dependency. I want to assume that my dependencies—especially Base & stdlibs—are correct and focus on debugging my code.
7 Likes
That is already possible to do by e.g.
push!(JuliaInterpreter.compiled_modules, Base)
However, an issue with this is that control flow very often goes something like Package -> Base -> Package due to method extension. Base frequently provides some generic methods and a package implements a specialized version that then get’s dispatched to. If we stop the debugging as soon we hit Base then a lot of Package code will be run compiled and breakpoints won’t be hit etc. Also, in cases where you use higher order functions, the method that runs belongs to e.g. Base but it is likely we want to debug that code anyway.
For cases where the module is mostly a “leaf” and things in it are not extended, then it works pretty well, for example, I use this quite often to not interpret Pkg.TOML which is quite slow to interpret when debugging the package manager.
4 Likes
Most of the debuggers in VS Code (including the current Julia implementation) have options on how you want to break on exceptions: break on any exception throw (handled or not), break only on unhandled exception throws etc. I think some option a la what @StefanKarpinski described (break where I called into base if there is an exception thrown in base) would be nice, but I think we should keep the option to also just break whenever any exception is thrown anywhere, that seems a very standard feature for the debuggers that are integrated into VS Code and can be really useful.
1 Like
While it’s great that this can be done, I think that the default experience needs to be improved because most people just won’t figure out how to tinker with this.
So what’s needed is the ability to transfer back to the interpreter when execution returns to user-level code. For higher order functions this could be accomplished by replacing Base.map(User.f, args) with Base.map(Debugger.interpreted(User.f), args). Since the call to Base.map is interpreted, the interpreter should be able to do this replacement straightforwardly. Extensions to compiled functions are harder to deal with. It’s worth thinking about what a minimal language feature to enable this would be.
4 Likes
sometimes i feel like the whole debugger just hangs. So coming from R and Python it’s pretty jarring. But I guess given no replacement candidate insight, it is what it is.
This isn’t a complete replacement kind of situation. I’m quite confident that right debugger will be an evolution of the current interpreter-based approach.
3 Likes