Are you OK with feedback in an issue on MakiE?
This might be a dumb question, but here goes: why not use GR as the default plotting package for 2D graphics (or perhaps graphics, period)? It is powerful, reasonably quick to compile, and definitely quick to generate the plot.
I get that Plots.jl is a frontend for several backends, and if someone needs it that’s fine. But Julia needs a default plotting package. Isn’t GR a good candidate?
Working with GR.jl directly is a very valid choice. Its a great package. That said, I suspect you are not asking about recommendations for your personal setup, but something that applies to everyone in the community. Like an “official plotting package”?
What’s the goal here? What would that imply? It’s clearly not a management problem, because there are no “Plots employees” to manage. And people really put a lot of effort into collaborating already, which Plots clearly demonstrates.
I don’t mean this to sound like a snarky remark. I understand that people wish for a more streamlined personal experience, but its not like the developers are in any way or form unaware of this perception at this point. The people involved in the whole plotting domain are really diverse, with different interests and motivations. I am more worried of those champions devoting less time to this, because being involved in plotting does not seem to be such a rewarding experience these days.
5 Likes
Are you OK with feedback in an issue on MakiE?
Yes please!
What are you missing that cannot be accomplished, e.g. by Plots + GR backend?
My view on why not just GR:
For once I plan to offer quite a lot of unique functionality (e.g. moving Julia callbacks to the GPU), which would simply be not possible otherwise.
In MakiE I also finally want to offer the kind of interactivity and usage of modern OpenGL, that GR might only offer in the future. Not sure how soon that is.
Also, MakiE + GLVisualize is all in Julia, so likely more hackable - but we do need static compilation to reduce the JIT overhead for a nice, julia only solution.
Thanks for the hints. The HDF5 backend looks interesting for my needs.
All suggested solutions seem fine for most purposes. However, the Matlab figures are perhaps still a little more advanced. One can search for graphics objects, which have parent-child relations, modify or delete them, add new ones etc. Not everything from Matlab needs to be recreated, but I find their figure concept particularly useful.
To me it seems that plotting can easily become highly complex. It would perhaps be worth (and a big work) to design a hierarchically structured plot container that can be filled/modified via the usual plot(…) functions calls or from persistent storage and that can be displayed with the different backends, i.e. some kind of high level framebuffer. The Plots HDF5 backend (and Pickle) are a step towards this, but perhaps not quite there yet.
2 Likes
That’s actually I problem had with Plots.jl and also GLVisualize.
There is no clear scene object and what’s there is not really intended for user interaction.
It’s actually the first thing I’ve implemented in MakiE, to make updates and user interaction with the scene as smooth as possible.
So you can do stuff like:
scene = Scene()
scatter!(scene, x, y)
# by default, the scatter plot gets inserted as "scatter$n" without n for the first plot -
# you can also just define a name for the scene node
scene[:scatter, :color] = :black # update color inplace
# share scene attributes and give plot a name
lines!(scene, scene[:scatter, :x], scene[:scatter, :y], name = :lineplot)
for i=1:10
# animate attributes
scene[:scatter, :x] = rand(10)
scene[:scatter, :y] = rand(10)
@ref scene.lineplot.thickness = i # some syntactic sugar
end
I’m not entirely set on how the plots get inserted into the scene.
Either like above, with ! functions and the scene as first argument, or I could also imagine an implicit version - or actually something like this:
p = plot(x, y)
push!(scene, :myplot, p)
Let me know what version you prefer.
I often do this using PlotlyJs.jl and save figures as json.
A common workflow would be to do something like this:
p = ... construct a plot ...
savefig(p, "my_plot.json")
Then later I can do
p_raw = JSON.parsefile(Plot, "my_plot.json") # just raw data
p = plot(p_raw) # adds a frontend display
Once you have p you can edit anything about the plot using the API methods.
I use this often to construct plots on a server/hpc, save as json, then copy the json over to my machine where I can view/tweak the plot.
This works so well because plotly.js plots are just json objects anyway, so I’m saving the “native” version of the plot (as far as the javascript lib is concerned) when I do this.
3 Likes
That is precisely what I have in mind. Julia is often compared to python or Matlab. Perhaps in the sense of a “problem-solving environment”.
Plotting is these days an inseparable part of such PSEs. There is nothing to install, no package to add, no investigation what package I should use in order to plot my data. This is possible with most Python installations (matplotlib) and definitely with Matlab.
In my opinion, Julia needs something like this. Be it just for basic, simple plots, but something where all I have to do is pass my data to a function to get a scatterplot or a bar chart. I thought it had that in Gadfly, but that seems to be off the table now. There needs to be a built-in plotting as part of the basic installation package (module BasicPlot?), so that newbies don’t get discouraged by all the plotting weirdness that is currently commonplace.
It seems to me that GR might be able to satisfy that need. Perhaps with a thin layer of prepackaged functionality on top? The Plots.jl with multiple plotting back ends seems like an overkill for basic plotting, especially given the difficulties of actually getting it to work, not to mention the horrendous start up times when it deals with multitudes of dependencies.
5 Likes
I deinstalled Plots+GR long time ago, re-installed now and the most obvious missing thing is: interactivity. Even resizing the window doesn’t update the drawing. If i switch desktops or the window is partially hidden, redrawing has some problems. And btw: how to open a second window?
I also don’t see clearly how to change font, use anti-aliased lines for plotting, use semi-transparent colors, subset the legend, put additional content into the figure etc.
1 Like
I have some opinions on this, which I will share below. But first, let me respond to the Plots-specific things:
Getting Plots to work should be as simple as, you know, Pkg.add("Plots"). It comes pre-packaged with plotly, so you can start making pretty plots to your browser right away. Some users never do anything else. It’s important to know that just because Plots supports several backends, you don’t need to install all of them, I’d say that certainly is pointless overkill.
WRT the startup times a lot of that has to do with the metadata maintainers requesting precompilation be turned off due to conditional dependencies (I should mention though that precompilation works fine in practice - all juliaplots members have it turned on by default and never experience issues).
I don’t mean to sound snarky, but a good first suggestion for a place to look is the documentation. overwrite_figure = false opens a second window (https://juliaplots.github.io/attributes/), semitransparent colors are controlled with the alpha argument (on the same page), you put additional content in with plot! (https://juliaplots.github.io/basics/). I’m not sure what you mean by “subset the legend” but you probably can’t do that, as the legend support frankly isn’t very good. Anti-aliasing is the domain of the backend.
Yes. Interactivity is just around the corner with GR (Josef demonstrated it a juliacon). Of course you have interactivity in pyplot and plotlyjs backends until then.
The GMT stuff looks great, the automatic projection brings something new to the table and it’d be nice with an opportunity to do projected plots using the interface of Plots. You should be aware that building a backend for Plots requires a lot of maintenance from the backend owner, also when a new feature is added to Plots. If a backend lacks features that others have, people will open issues, and there’s no way the JuliaPlots group can keep all of the functionality up-to-date across all backends.
4 Likes
I think that Plots + one main backend (probably GR currently) is actually a fantastic default plotting interface that surpasses the solutions in other languages (provided some issues are resolved but that work is underway).
The plotting story in R is a good example. R has a very decent base plotting but it has its drawbacks and so Lattice and ggplot2 were developed. Now when someone develops a package they need to choose what solution to support – either to overload the plot() command or add some ggplot-based functionality or both. And then you have plotly etc. As usual, the syntax is very different across these systems and the end user needs to deal with it.
In the Plots framework, one needs to learn one syntax. With one backend coming preinstalled the end user does not need to care about the backend if they don’t want to and as the technology develops the main backend can be changed without the end user having to worry about it at all. The package developers don’t need to make a commitment either because they can support it all through the Recipes framework (without creating heavy dependencies).
At the same time the user can change the backend depending on their needs without having to change syntax. Although many users might not care, it might make a big difference for others. UnicodePlots is a good example – most users probably don’t care about it but others needing to work directly on a remote server through the terminal might be very happy about it and especially about the fact that they don’t need to learn a new syntax to use it.
All this is almost magical and seems to me far better than what R/Python/Matlab have. It is not perfect yet but it is improving constantly and complex software just takes time to mature.
12 Likes
It’s probably not an easy thing to do, but, while refactoring Plots, it’d be really cool to define something like a Plots backend interface so that, as a backend developer you’d know that there is a finite list of basic method (defined in Plots) you need to implement and then everything should just work. As of now, adding a new backend requires a lot of work but I don’t think it has to be this way, as most of the complex plots are built out of simple functionality of the backend (line, shape, marker etc).
3 Likes
Here’s what I think we should do with plotting in Julia:
I don’t think we need a default plotting package - because how do you guarantee your one chosen package is the preferred one? R tried that, but it’s not the one everybody is using, and the reason it even works with that is that everybody has accepted Wickham’s packages as a standard library.
I also think the current understanding, of a cohort of several different plotting packages that compete with each other to be the preferred plotting package is profoundly mistaken. It creates an uneasy feeling about plotting. And it misses the promise of a nicely integrated plotting functionality, though plotting is at least as central to a scientific computing language as is something like data representation.
What we need instead is a modular plotting structure with good support in a shared base package, based on interfaces. This has not been tried before to my knowledge, but represents a unique chance for julia. I don’t think we’ve yet begun to unpack the potential there is in applying an interface model to julia’s plotting ecosystem. The collaborative nature of Julia’s package ecosystem (yet) makes this possible.
What we need instead of a default plotting functionality is a default plotting specification, which should allow package authors to define visualizations for their types in a package-independent way – this is a line, this is red, these are the values, etc. The reason we need this is that Julia is based around user types and dispatch on those types. This means that any package is able to define plot methods for any types it specifies. The role of plotting, after all, is not to create pictures - it’s to generate insight. This has such great potential in terms of data analysis: instead of building diagnostic plots, you can pass any object you call to plot and get visual insights. To me, being able to define user objects in packages, and being able to define visualizations for these objects are intimately intertwined.
One way this is currently suggested to be done is with glue packages. So you make a ConvexHullsGadfly package, a ConvexHullsGR package, an OpenStreetMapGR package etc. The issue is that you get a crazy number of packages, some packages may be missing (eg. what you only have a ConvexHullsGR and OpenStreetMapPyPlot packages but want to combine those in plots). It also becomes unclear who should maintain the packages in the face of updates. A plotting specification living in the package that defines the type is clear, transparent, and easy to maintain.
The rest should be done with interfaces. If you think of the differences, from a user perspective, between different plotting packages, they are two things: 1) the plotting interface – how are plots defined and called; and 2) the aesthetics / looks and output types of the plot. There is no reason these two should go together. One can find Gadfly beautiful but not have a taste for the grammar of graphics, or the other way around. And something like Plots is really both 1) an interface aiming for concise plotting commands, and 2) a plotting implementation (though the backends take care of some of this). But using e.g. GGPlots.jl (unmaintained at the moment) it is 100% possible to use a Grammar of Graphics interface instead, with the same functionality.
I thus suggest this model:
ChosenInterfacePackage-> BasePlotSpecification->ChosenOutputPackage
Plots is not this, at the moment – but it is attempting to implement some of this vision. And RecipesBase takes an important step towards being a broad plotting specification. It currently lies fairly close to the Plots syntax, but there’s really no reason it has to. Unfortunately, in the current perception that plotting packages are competing entities, there isn’t an understanding that something like RecipesBase or a related package is an attempt to unite plotting functionality; instead it is sometimes viewed as competing with plotting packages that don’t support a shared interface. What we need is some core people in julia to identify this as a priority, then agree on a common specification that isn’t perceived to advantage certain plotting packages. A good first step would be for all of us involved in Julia’s plotting ecosystem to sit down together and talk this through.
We can then build a plotting ecosystem based on picking your favourite interface, your favourite output format. In principle, all the other functionality in a plotting package, related to ticks and axes and colors etc, could be implemented in a Utils package that was shared among platforms. This would ensure efficiency, but also continuity, as you wouldn’t to the same degree get the current problem with a plotting package being left by the wayside after a few years when the original author runs out of steam (a huge problem for this ecosystem).
This would ensure ease-of-access, full pluralism of choice regarding interface and output, full integration of the ecosystem, and basic support for visualization across the broader package ecosystem. We just need to decide to do it, rather than believing that plotting must be a competition between small packages.
14 Likes
FWIW, this is the reason why Vega.jl and ECharts.jl don’t implement the Plots API. And like you, I’d love to see everyone working together and supporting each other, rather than trying to get the “one” default way to plot into Base.
3 Likes
I can see why competition to get into Base as the default graphics package may be perceived as a bad thing. However, try to put yourself into a newbie’s shoes: you are exploring the language, what is available, and currently the package featured on the homepage is Gadfly. On the download page, both Gadfly and Plots.jl are featured, as is PyPlot.
When presented with this selection as a newbie I am discouraged. Just getting started, I do not want to make a (uninformed!) decision which package to install and learn. A newbie wants one way of plotting, not several.
And, a newbie wants a guarantee that once he or she learns how to plot, that knowledge is not going to go to waste, same as the knowledge of the basic syntax of the language (leaving breaking changes aside for the moment).
As a power user, I may be perfectly fine with having several competing packages for plotting. However, if we are interested in engaging new people at all levels of experience in Julia, we should pay attention to the newbie.
In a way, presenting some packages for plotting on the Julia-language websites already implies that they are “default”. Gadfly is (used to be?) bundled with Julia Pro, Gadfly, Plots.jl, and PyPlot are preinstalled for Juliabox, …
So I think having one simple package as a default is a good thing. This does not preempt @mkborregaard’s ambitious idea for an innovative plotting framework. I’m just worried that this big-effort sort of thing is not going to happen in time for 1.0, and I think releasing 1.0 without no-fuss plotting would be a mistake.
5 Likes
No. It does cause issues with package development. If you want it to actually act smoothly as a default, you need developers of packages to support it. To tell them they need to support some simple one but then maybe something else when we want more plotting features means we’re spending too much time making people happy because they won’t read docs. At some point there has to be a better way.
This is why I advocate Plots.jl. Yes, it has some issues it needs to work on. But Plots.jl is a developer target. If you want something to act as a default across Julia’s ecosystem you need it to be a common developer target, and Plots.jl serves as that through its recipe system which lets you add plotting functions and to your types without incurring a dependency on a plotting package. This has worked out well and you’ll find recipes are available with many packages so Plots.jl “just works” around Julia quite often. One wart in the setup is that the JuliaStats projects refuse to support it, so you do need a separate package, StatPlots.jl, to add recipes for them. But other than that, it’s pretty universal.
If we could just get people to start helping each other, we could have Gadfly added as a backend again (it was before) and flesh out GGPlots.jl and then Plots.jl would be a developer target for Grammar of Graphics style APIs as well. There’s already starter code but the issue is that most people working on Plots.jl don’t seem to be GoG people, so we just need one person to do it and GoG people would likely follow.
So if there’s a default it’s Plots.jl. With that, there are some issues it needs to fix. Plots.jl has some speed issues it needs to work out. In short, it needs to more smartly use its global CURRENT_BACKEND, use dispatch properly on an internal interface in order to allow it to precompile smoothly (and all of the way through, so this would be two separate speed improvements), and it needs to get rid of functions internally converting many things to Vector{Any} / Vector{Vector{Any}} which seems to have been done to make things easier, but this can be fixed.
2 Likes
@Evizero, @mkborregaard, @ChrisRackauckas: Alas, I just found out that GR was broken on 0.6. When called from Plots.jl, the plot window never appears again after it’s been closed. (Actually, it is also broken on 0.7. Same thing, except GR can only be called directly, since Plots.jl cannot be installed on 0.7.)
At this point, plotting in Julia is in a dismal place indeed.
1 Like
That’s a GR bug which has been around forever. Then the anti-aliasing mentioned earlier is a GR bug on Windows that has been around forever too. That’s why I think PyPlot → PlotlyJS → Plotly should be the defaults (at least on Windows), but @mkborregaard disagrees with me on that.
We’re just defaulting to a backend that doesn’t work well cross platform which is a problem I have been repeating in the chatroom but hopefully it’s more clear that it’s actually an issue now. GR’s going to be great after it changes to the new interactive GUI, but until then it does have Windows issues to sort out.