Roadmap for a faster time-to-first-plot?

mbauman · August 20, 2019, 10:48pm

3 posts were split to a new topic: Compile time slowdown from 1.1 to 1.2

davidanthoff · August 24, 2019, 5:07am

But Plots.jl doesn’t even use a native (julia) backend by default, right? It has terrible load times and then delegates the actual plotting to a non julia software stack. We have lots of other plotting packages around in Julia that have that same design (Julia API and non-Julia backend) and have much better load times.

I just tried Winston.jl, which I believe is a pure Julia plotting package (caveat, I might have that wrong), and it has way, way, way better load times than Plots.jl. That would indicate that having a plotting package written in pure Julia also doesn’t mean it has to have slow load times.

I have to admit that I never dug deep into the Plots.jl design, but I would expect that having a) a super pluggable backend story would increase load times and b) having the whole recipes system would increase load/compile times. Those are nice features, but if having them means that plot(x,y) takes forever, then I think that is the wrong tradeoff for a main plotting package that beginning users are exposed to. I think we would be better off to steer new users to a package that doesn’t have pluggable backends or a recipe system, but has a fast first plot(x, y).

lobingera · August 24, 2019, 6:58am

Looking into Project.toml might have given you the picture, that it depends on Gtk for Windows and Cairo for rendering. But you are right, Winston is some sort of low-level plotting. So it doesn’t depend much on type inference, parametric types and other ‘high level’ julia language aspects and does only some number and coordinate transformations and hand over to rendering.

tobias.knopp · August 24, 2019, 9:44am

The question then is how to define “pure Julia plotting library”. I tend to think of Winston as a pure Julia plotting library, where only the low level primitives are passed to Cairo. I don’t see any advantages of replacing Cairo with a pure Julia solution since Cairo is very mature. In particular it also has pdf and png output. Such an architecture is actually used by most plotting libraries (Matplotlib, Makie) with the only difference, that the backend in Winston is not pluggable.

I don’t know why Plots.jl has been chosen to be advertised as the prime plotting solution on the Julia web page. IMHO it would be better linking too PyPlot.jl which is the standard in the Python world and has acceptable load and plotting times.

xiaodai · August 24, 2019, 9:59am

Good call!

mkborregaard · August 24, 2019, 10:03am

It isn’t. There are links to loads of plotting packages. It is mentioned first, yes, which makes sense because of its meta-package nature - and it is the most widely used one. It was also the most widely used one for many years while the webpage pointed exclusively to Gadfly and PyPlot. I can only assume there are people who actually find it nice and useful.
TBH I’d find it rather sad if we reverted to only suggesting matplotlib given the richness and dynamism of the julia plotting ecosystem.

kevbonham · August 24, 2019, 12:39pm

You’re off course entitled to your opinion about trade-offs, everyone will have their own. But I think the fact that I can write plot(myobj), where myobj is a matrix, an Hclust, an ordination from MultivarateStats, or any of a few dozen other types (including my own) is essential for my productivity. Every time I’ve been frustrated with something about Plots and tried VegaLite or Gadfly or PGFPlots or what have you, the fact that I need to go through and convert everything to Arrays takes so much time and effort I go back to Plots in less than a day. Load times are trivial by comparison.

Michael had already addressed the last point, and I think this is really telling:

Tamas_Papp · August 24, 2019, 12:53pm

I really admire the time and dedication the Plots.jl team has invested in that package. But I am skeptical that keeping multiple “backends” around to fix the failings of one with another is the right approach in the long run.

I think of it as an interim solution, until something that is (mostly) written in Julia and thus is easily extensible and maintenable emerges.

I think Makie.jl is promising for sophisticated interactive plots, but not really filling the niche of creating 2D (and occasionally 3D) static plots using vector graphics.

mkborregaard · August 24, 2019, 12:53pm

What I do think this discussion shows is that it was a fundamentally good decision not to deliver Julia with built-in plotting. Everyone does have different perceptions of trade-offs, anf Julia makes it easy for users to find their own. As many other nice things in Julia (such as Revise and Pkg) this comes at a little cost to new, non-technical users, but benefits users over the longer run.

Maurizio_Tomasi · August 24, 2019, 12:57pm

Couldn’t agree more. My colleagues are always impressed when I show how easy is to plot complex objects in Julia using plot(obj). My latest package, Harlequin is using this feature intensively, like for the ScanningStrategy type, and I find really awesome!

ChrisRackauckas · August 24, 2019, 1:13pm

Please let’s just stop spreading nonsense. Here’s a post that describes what’s going on, with a roadmap for faster first time to plot.

github.com/JuliaPlots/Plots.jl

A real "first time to plot" analysis

JuliaPlots:master ← JuliaPlots:first_time_to_plot

opened 01:02PM - 24 Aug 19 UTC

ChrisRackauckas

+798 -728

After seeing a bunch of nonsense like [this thread](https://discourse.julialang.…org/t/roadmap-for-a-faster-time-to-first-plot/22956/142), I decided it was a good idea to document why "first time to plot" is a thing. tl;dr, it's because the original author was not careful with types, nothing more nothing less. (I have to do this on my desktop to not segfault, but @vtjnash saw it. Will post the profile when I am back from travelling.) First of all, there is no code loading going on in like a year. GR is baked into the system since that PR was done, so let's stop assuming that code loading nonsense is still the issue since we ruled that out a long time ago... and it doesn't even exist anymore. Get it out of your head. Now let's continue with a clear mind. If you look at the profile using `ProfileView.view(C=true)`, we can distinguish between Julia type inference time and LLVM compile time. That gigantic bump is Julia type inference time, while the sliver between the bumps is LLVM time. Conclusion: it's our fault in type inference and there's no one else to blame. This is actually quite peculiar since most of the time you see LLVM taking a lot of the time, but that's not the case specifically for Plots.jl. And the reason is because Plots.jl's choices with types are a mess. To start, almost everything is stored in a `KW = Dict{Symbol.Any}`. All of the args are handled in this structure, but this item does not show up in the Julia sysimage, so all operations with it require being compiled. Changing this to `KW = Dict{Any,Any}` is actually a little helpful (Requires the similar first time to plot RecipesBase branch). Basically, we want type inference to give up quickly and just go to `Any`. However, this library throws `KW` into small structs, and then has some abstract types over fields. To make it give up, we can just kick to `Any` again. Etc. Usually that can be helpful, but here we kick to Any too much and then we start getting type inference nightmares. We know from around the ecosystem that function-barriers cause a lot of compile time increases: https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl/pull/841 https://github.com/JuliaDiffEq/StochasticDiffEq.jl/pull/207 and this is precisely where we start running into a wall. With every function call, we hit a barrier and so this overhead cost of knowing that it's a dynamic dispatch function barrier will be hit (to be honest, I'm not sure why this extra cost exists, @vtjnash?). But similarly, putting everything into one super long function will cause superlinear compile time growth, so we need to split functions. So as a proof of concept for this idea, I split out the series types and legends handling branches in the `gr.jl` file. If you comment those portions out you see a massive compile time decrease. If you split them out to functions, you don't get that same decrease because you're now hitting the secondary inference compile time issue. So given how these two play against each other, the only real way to solve this is to make Plots not drop so many types. That a giant feat because everything is stored in bad dictionaries, so someone would have to 1) Set the argument types underlying the attributes. 2) Converting to a standard form the moment the arguments are seen like that. 3) Pushing this type information down. Currently this is very difficult, the biggest offender being Measures.jl. Measures just has so much type information on every value, like Units, that it's very difficult to keep the types aligned with them. It's just way too much type information, so it should just be made a struct with a value and symbol, using the symbol to branch behavior, instead of holding each new value with a possibly new type. When you dig into the pipeline, you see that 7 seconds of compilation is on the type handling of the plot.jl `_plot!` function, and then 13 is in the gr_display function (at least with GR). Essentially all of that time could be cut if the types were more inferred, given that the entire issue is that inference is just going insane. Oh, and this splitting shows why it's so hard to fix this issue: 7 seconds comes from mostly 3 function calls, while the other 13 comes from a few different branches, but that's still quite a few hotspots to hit, so just trying to "be type inference" and declare types would still be a long process (and you still lose inference every time you dig into one of these types without information...) In conclusion, don't use Measures.jl and a bunch of non-inferred partially abstractly typed dictionaries and then wonder why it takes forever to compile. I can tell you why: type inference gone amok. The profiles show it, and it's hard to get around since the original tools like function barriers end up contributing to the problem. This PR shows that you can get very minor gains by avoiding that issue, but you actually need to address it if you want to go anywhere. It really has nothing to do with "tradeoffs" or whatever nonsense, it's just about writing code that's easy on inference. We could fix this in Plots, Makie, GR, etc., does matter which. This makes it clear it's not the method but it's the implementation.

The solution is shown to more concretely be:

Get rid of the KW nonsense
Concretize the argument handling earlier has to happen, otherwise you already take a 7 second hit
Concretization of the types should happen by the display function, since 13 seconds of it is type inference into the (hard coded and definitely not runtime loaded) gr_display function. A large part of this is because the superlinear behavior of the compiler.
Get rid of Measures.jl. It has too much type information and causes issues. If anything, replace it with values and associated symbols, and hardcode branches instead of over dispatching. Units are just not a good idea for compile time… this would likely be helpful for Gadfly as well.
Throw a few declarations on the types in the structs that cannot be well inferred, but you know what they are by the time you dig a few functions down. This should only be a few pieces, not every argument that comes out of a KW.

Or if GR.jl does type-stable recipe handling, it would be a good solution. Makie.jl might have similar or different compile time issues: saying anything without digging in would just be spreading nonsense so I’m not going to predict its cause here. That covers the real feasible solutions, since anything that excludes the vast majority of the Julia community by not supporting type handling, 3D/trisurfs, PDF output, etc. is fine for niche uses but just a no-go as a solution to show to new users.

I hope that shows that it’s not actually impossible, but someone just needs to put some time into it. The real issue is that nobody is looking at typing the attributes .

I agree it’s not needed anymore. However, it’s not the issue.

The data doesn’t seem to suggest a tradeoff if it’s done correctly (DiffEq’s argument handling is somewhat more complex than Plots’, but it’s under 3 seconds now because it’s inferred). It just suggests that typing everything abstractly and then running a recursive algorithm is a bad idea, which I thought was obvious. Recipes can be done correctly.

StefanKarpinski · August 24, 2019, 1:27pm

I think this conversation is somewhat in danger of losing sight of the fact that “time to first plot” is not really about plotting—it’s a shorthand for general compilation time issues when trying to run complex package code with lots of interacting dependencies. Even if we had a statically compiled plotting solution written in C that we could force on everyone (hypothetically), so what? We’d have solved the literal “time to first plot” issue, but we’d still have a “time to first table join” issue and a “time to first differential equation issue”. If we try to apply the same “successful” solution to all of those problems, we end up rewriting everything in C, which is obviously counter to the whole premise of Julia.

Plotting is just a good representative of this general problem because it’s really complex and something almost everyone wants to do. So, as we usually do, we pick the hardest instance of a general problem and then try to solve it in a generic way that also addresses all of the lesser versions of that kind of problem. Not by solving one and only one specific instance of the problem.

KestutisMa · August 24, 2019, 1:33pm

I think “time to first plot” is solved by PackageCompiler and Makie.jl (which is very fast) workaround, only thing is that sysimage compilation takes long and must be done every time after updated packages, but it is nice that this also being improved in near future.
This thread better have different name like “Tips and news about compiler latency improvements”.

Tamas_Papp · August 24, 2019, 1:38pm

I am sure that improving compilation time will be a great generic solution, but my understanding of @ChrisRackauckas’s analysis linked above is that the TTFP issue of Plots.jl could be at least improved significantly within the existing framework, “just” with more compiler-friendly code.

StefanKarpinski · August 24, 2019, 1:39pm

Sure, that’s always an option, but one shouldn’t have to be an expert in how to appease the compiler. It should be good at both low latency and running fast. I’m not saying people shouldn’t work on the other side of the problem, of course, just that ideally it shouldn’t be necessary.

ChrisRackauckas · August 24, 2019, 1:44pm

If you can, make your code infer. That makes it compile a lot faster.
If it doesn’t infer, don’t put abstract types over things. Handling calculations with “how abstract it should be” is one of the slowest things. Just make inference call it Any.
If you can, use things in the system image. Dict{Symbol,MyType{T}} where T will require compiling all of the little parts, while Dict{Any,Any} is used in Julia so it won’t.
Avoid dictionaries if you can. Dictionaries are the devil’s type.
Try to avoid putting Tuple in a bunch of types. Accidentally putting ::Tuple into a type adds an abstract type into a struct, which breaks (2), but then every size tuple will need to specialize a ton. If you don’t need type covariance… make it a Vector{T}.

That’s what the current state is. I’ll keep talking with the compiler people to see if there’s anything that could be helpful to make it less crucial for users to “follow the rules”. I think making type inference give up easier is probably the real solution here. @nospecialize doesn’t seem to do enough: I want a @compiler_please_give_up_or_the_entire_internet_will_hate_me macro.

Zach_Christensen · August 24, 2019, 2:05pm

This may be true but it is kind of an unsatisfying conclusion for this thread for those unable to contribute to improving the compilers performance. Personally, I was hoping to find out some way I could contribute, even if it just meant finding bugs in a package and documenting them for those more capable of fixing them. However, if the conclusion is “wait for us to improve the compiler” then perhaps this thread should be closed and if someone has a clear plan on improving a specific plotting package that can be explored in a different, more focused thread.

xiaodai · August 24, 2019, 2:09pm

No. Plots.jl still slow to plot after compilation

StefanKarpinski · August 24, 2019, 2:12pm

That is kind of how I feel about this thread, but other people may have other perspectives and it’s possible that refactoring plotting libraries will help. Making things easier on the compiler is useful even if the compiler gets smarter.

ChrisRackauckas · August 24, 2019, 2:16pm

Then help fix Plots.jl’s inference which is the issue. It’s just a standard Julia engineering task that no one has done. That’s a fix that that requires no compiler changes, but it would take more man-hours than I can donate right now.

For pointers, I would start by refactoring the attributes in the SubPlot to one fixed set of values to index out to concrete Float64s, then dig in and change all accesses from getindex to getproperty access for those values. If you fix SubPlot you fix the vast majority of it