Vizcon , 29th-30th Oct Berlin

Community
1

We just had “Vizcon”, an open meeting for anyone interested in plotting and visualization with Julia. The idea came up at the Juliacon in London, and an open invitation was announced on the Slack #plotting channel, with follow-up planning at the Slack #vizcon channel. Present at the meeting were Simon Danisch (@SimonDanisch), Pietro Vertechi (@piever), Christof Stocker (@Evizero), Daniel Schwabeneder (@daschw), Moritz Schauer (@mschauer), Josef Heinen (@jheinen), Oliver Schulz (@oschulz), Sebastian Pfitzner (@pfitzseb), Diego Javier Zea (@diegozea) and myself (@mkborregaard). Most of us are associated with the JuliaPlots org, so we decided that the purpose of the meeting should be 1) discuss the future of plotting with Julia, 2) stabilize Plots for Julia 1.0, and 3) kick-start contributions to Makie.

Overall the meeting was a great success, with many important discussions and also quite a lot of actual work done. Here’s an overview of what happened.

Presentations:

One presentation and discussion involved the future of plotting in Julia. Plots is currently one of the most popular plotting packages in Julia, and as such very important for the viability of Julia as an all-in-one programming language solution for many users. In particular the underlying recipe system in the stdlib-like RecipesBase has been very successful, with just short of 400 reverse dependencies. It’s also quite difficult to maintain, because of the relationship to backend packages with very different interfaces. Over the last two years, we’ve reviewed merged on average 5 pull requests per week, and closed like 150 issues per year. But, issues are opened on Plots at roughly twice that rate, with no sign of diminishing returns, and there is currently >300 open issues on Plots. A presentation of this, and categorized tables of issues and pull requests, can be found here , here and here For long-term stability we thus need either more contributors, or, to find a less maintenance-heavy plotting solution.

At the Vizcon there was great enthusiasm about Makie as a solution for the future, which has a nice clear codebase and is really powerful in terms of interactivity and scope. It also has a system for recipes already and backends in the works. It is very possible that it can take the role of Plots in the future, but development is still open. Many of us present expressed an intent to spend some of our coding time contributing to Makie in the present and future, and that has already begun. Simon gave a guided tour of the code and design of Makie + AbstractPlotting.

Josef gave a presentation of GR, which is a much bigger framework than just a backend to Plots like many Julia users know it, acting as both backend and frontend for a major visualization pipeline. The presentation can be found here.

Discussions

The biggest discussions we had were:

Precompilation and time-to-first-plot

The biggest weakness of Plots, and suffers from the issue that it’s hard to tell why this is so slow. Makie is currently even slower. The solution appears to be to get static package compilation to work. GR, which wraps a binary, is very fast from using GR to first plot. We did some work to get Plots precompilation faster, i.e. #1825 and JuliaPlots/PlotUtils.jl#52 .

A new recipe system

The heart and soul of Julia is the primacy of custom types. We found that plotting in julia needs some plotting-package independent way of mapping custom types to a visualization in user packages, while only depending on a small stdlib. RecipesBase goes some of the way for that, but is very tightly linked to Plots and Plots syntax. We decided we need something that is (practically) dependency-free, uses the facilities of julia 1.0 (like named tuples), does not depend on macros, and which has an intuitive syntax that is close to how we tend to use plotting libraries. It should be general enough to be a stdlib usable by all julia packages - using the syntax would have to be done by the plotting package itself or some small glue package, not in a big metapackage like Plots does. We started working on the syntax, and Oliver now has an almost-working prototype: GitHub - oschulz/PlottingRecipes.jl at dev and a prototype glue package for GR GitHub - oschulz/PlottingGR.jl at dev .

We also agreed that Makie needs a recipe system that is much more lightweight than the one currently offered by AbstractPlotting. This could be such a general recipe system but there are things to iron out, like Makie relies a lot on Observables at the very core but other plotting packages don’t. As a response to this, Simon implemented a change to the recipe system so the user will not have to worry about Observables: JuliaPlots/AbstractPlotting.jl#35 .

Declarative grammar / grouping

Many modern plotting packages like ggplot2 and vegalite have a declarative grammar, where you specify which plot elements should be control by which data, and then sensible plots are built automatically. An old idea is that both standard and declarative plotting syntaxes can be united (the library can be multi-paradigm) by expanding the group ing behaviour of Plots. We talked about various ways this could be done, and Pietro now has a working prototype of this for Makie: JuliaPlots/StatsMakie.jl#7

Makie backends

Josef and Simon managed to developing a GR backend for Makie, can now be found here: GitHub - JuliaPlots/GRMakie.jl: GR backend for Makie .
A UnicodePlots backend is also planned.

Other work

Here is a list of pull requests made during the vizcon that have not been covered in the above:
Deprecate glvisualize backend
JuliaPlots/Plots.jl#1818

Transfer portfoliocomposition recipe
JuliaPlots/Plots.jl#1819

Reorganize PlotRecipes
JuliaPlots/PlotRecipes.jl#50

Implement frames axis argument
JuliaPlots/AbstractPlotting.jl#31

Allow recursive merging of plot Attributes
JuliaPlots/AbstractPlotting.jl#32

Refactor AbstractPlotting backend display
JuliaPlots/Makie.jl#200 and JuliaPlots/AbstractPlotting.jl#33

Add new example reference image
JuliaPlots/PlotReferenceImages.jl#33

Initial version of GR Makie backend
JuliaPlots/Makie.jl#201

New implementation of density, histogram, Table compatibility and lifting
JuliaPlots/StatsMakie#10

Remove concept of atomics
JuliaPlots/Makie.jl#203

Add linestyle for axes
JuliaPlots/AbstractPlotting.jl#36

Groupedbar recipe
JuliaPlots/StatsMakie.jl#12

Allow heatmap of nx1 or 1xn matrices
JuliaPlots/Plots.jl#1815

Fix docs build
JuliaPlots/PlotDocs.jl#112

Documenter compatibility update
JuliaPlots/PlotDocs.jl#115

Update reference images for testing
JuliaPlots/PlotReferenceImages.jl#34

Allow customizable RenderMode for txt
Evizero/ReferenceTests.jl#19

Implement cycling color palette for series (/plots)
JuliaPlots/AbstractPlotting.jl#37

Testing UnicodePlots seriously
Evizero/UnicodePlots.jl#85

Allow to use arguments in default theme
JuliaPlots/AbstractPlotting.jl#40

Other developments

• Apart from these things we also worked on various projects that are not covered by the PR’s above.
• Better Makie theming
• Work on getting the PlotlyJS Plots backend 1.0 compliant
• Refactor PlotRecipes to focus on graphs only (GraphRecipes) while moving other recipes elsewhere
• Remove the Plots/StatPlots dependency of PlotRecipes and get it 1.0 compliant.

Overall a very productive and fruitful meeting. The general sentiment afterwards was that meetings like this could be emulated, by ourselves or other organisations, to stimulate progress in the Julia ecosystem.

The NextJournal company in Berlin graciously allowed us to use their meeting and office facilities (and excellent coffee) for the meeting. We’re also very grateful to Julia Computing’s community funding for covering meals during the two meeting days.

60 Likes
2

Awesome, thanks for the thorough summary! And thanks to everyone that works to make plotting in julia great! I’m definitely excited for the future :slight_smile:

I still have a couple of open questions about the present and near future. Basically, what are the recommendations for the next 3-12 months for:

  1. People that just need plots and don’t plan to develop (here, I’m pretty confident the answer is “just keep using what you’re using”)
  2. People that are teaching new users (I suspect the answer here is the same as for #1)
  3. People that want to develop recipes for their own types. Plots recipes are clearly more usable at the moment, but it’s hard to invest effort in something that’s going to be deprecated in the near future.
  4. People that want to make small contributions to plotting packages. As you mentioned Plots.jl still needs a lot of work, but it seems like a lot of the effort of the core plotting folks is going to Makie (which makes sense!). Is the advice here: “fix bugs you run into, but new development goes to Makie?”
1 Like
3

I’m curious about the reasoning behind this choice. I know that inflationary use of macros can obstruct reasoning, but isn’t excluding this feature per se rather limiting?

(we can split if this is too off-topic)

4

I’ve edited the answer here to be more clear:
Nothing has changed, in that Plots is still being maintained (as well as we can, as I outlined above), including enhancements and not just bugfixes. And PRs to Plots are still very helpful for the great many people who benefit from Plots everyday. Keep writing Plots recipes if you would have done so otherwise.
And at the same time we are enthusiastic about Makie and how it is developing.

1 Like
5

Yes please split or ask on the slack :slight_smile: The short answer is that there is nothing wrong with macros per se, but many people find it hard to use Plots recipes because they use macro-driven non-standard julia syntax.

1 Like
6

I agree with Michael’s point of view, but wanted to emphasize that trying to create recipes in Makie is quite useful to see whether the framework is powerful enough to accommodate your use case. There’s work in process on a simplified recipe framework in Makie (that allows the user to not have to worry about interactivity, while preserving all the nice interactive features of Makie) and as soon as that’s done it’s probably a good time to start writing recipes for it.

3 Likes
7

I agree with that view as well!! I just want to stall the impression that Plots.jl recipes are a thing of the past and now we can just move on. That’s getting ahead of the actual development IMHO. Consider the “it’s time for a 1.0 release” remark 6 years ago :slight_smile:

2 Likes
8

From user’s point of view:

  • It would be nice to see Plots + GUI interaction options, such as those in Matlab, where plots are shown in separate figure windows, and axis are simply one type of GUI elements. So you can add any other GUI elements alongside, set-up interactivity by callback connections (such as range+sliders, mouse clicks, buttons, pop-down menus etc.).

  • Also, it would be nice to see some examples of Plots, embedded into working GUI backends (desktop and web), working with Julia 1.0.

1 Like
9

This already works. Just combine Gtk.jl and Winston.jl. I use this in a medium-sized project und it works pretty good.

10

https://github.com/ma-laforge/InspectDR.jl
https://github.com/JuliaImages/ImageView.jl

are just two further examples that use Gtk.jl. From my point of view Gtk.jl is pretty underestimated by the Julia community.

1 Like
11

Sadly, I don’t think that’s true - It is just pretty unstable, the Julia wrapper & the C side, so I wouldn’t call that understimation. It was also ridicilously difficult to theme it on different platforms, which I consider to be pretty essential!
E.g. it is pretty much unusable on windows - and there are ~60% Julia users on windows… Also, OpenGL isn’t supported on OSX and I’ve run into a couple of segfaults [1 2]…
I don’t mean to bash Gtk, but I really gave it a chance, and it was impossible to write stable software for all Julia users with it. It’s probably fine if you have a small team using only unix :wink:

By the way, I added a documentation section for GUIs in Makie:
http://makie.juliaplots.org/stable/examples-interaction.html

3 Likes
12

I was responding to @sairus7 who asked for similar UI support as in Matlab.

Little sad that you take that as an opportunity to make a rant about Gtk.jl
The term “underestimated” did go more in direction of the number of developers that care about Gtk.jl. The bugs that you cited are all fixable but it simply needs help. I use Gtk.jl on OSX and Linux and it works well for me. The segfaults are a Julia “bug” that is solved on master and I hope that the fix will land in Julia 1.1.

1 Like
13

I wonder whether this discussion should be split into a separate thread as GUI design was not a central part of Vizcon (though I definitely think it’s an interesting topic).

Concerning web based GUI (that you can deploy locally via electron, of course not with the same performance of a native solution) Interact is compatible with Plots, see here for more information. In particular the PlotlyJS backend allows you to embed an interactive plot inside a GUI.

For Makie + GUI I personally think there are at least two ways ahead (not mutually exclusive):

  1. Create a web based backend for Makie so that it can be embedded in a web based Interact GUI while preserving plot interactivity (you can of course already embed it as png)
  2. Add Makie as an Interact backend. Interact has a framework to allow backends (originally thought as CSS frameworks) which could be recycled to also support Makie native widgets as an implementation. In this way the same syntax of Interact (say @manipulate and friends) could be used with Makie backend to create efficient native widgets.

I’m quite excited about 2. as it would be a big plus to have a “pure julia” native GUI library: I haven’t looked at it concretely, but there are definitely some low hanging fruit. In particular, it’d be nice for users to be able to use the same syntax for web and native GUIs.

5 Likes
14

Agree :+1: but can I do that or does it require moderator action?

1 Like
15

Sorry @tobias.knopp, didn’t meant to be so negative!

number of developers that care about Gtk.jl.

As I’m a developper who stopped caring about Gtk, I thought it was indeed the right situation to point out my reasons. I know it’s not nice to take the opportunity to rant over libraries other people want to succeed, but to me all those bugs seem out of my control, especially that there is no OpenGL support on osx and also a segfault in the Gtk C library - which was the point at which I decided it’s pointless for me to invest into this, as much as I’d like to have Gtk succeed! I hope I can just come back to Gtk at some point with those bugs fixed, since I like it more than Qt :slight_smile:

16

I think it requires moderator action.

17

Maybe I don’t have the full picture but it seems to me we need at least four abstract packages, AbstractDrawing, AbstractWindows (handling windows and events) and then built on top of that AbstractPlots and AbstractGUIs. That way if Gtk OpenGL drawing backend is buggy one can use Cairo instead (for 2D plots at least).

It looks like we already have several pieces but it’s not very clear to me how/if they fit together (e.g. Interact looks a bit like an AbstractGUIs but at the same time it looks to be tied to web technologies).

18

Maybe I don’t have the full picture but it seems to me we need at least four abstract packages, AbstractDrawing, AbstractWindows (handling windows and events) and then built on top of that AbstractPlots and AbstractGUIs

That’s kind of what Makie (or more precisely AbstractPlotting) is doing :slight_smile: All windowing, drawing and event code is pretty much abstract or wrapped in backend independent objects. So you can completely define interactive guis + plots, even save them to disk, and then just render them with different backends :wink:

1 Like
19

It sounds pretty good, it just seems to much stuff for a single package. But maybe it will split naturally later on as it grows.

20

Maybe - but it’s actually not that much code though and it’s surprisingly hard to separate cleanly (trust me, I tried!)…
At least AbstractWindows + AbstractPlotting is pretty intertwined… AbstractDrawing is also pretty tied to the way you handle windows and create your plotting scene graph :wink:
At some point I didn’t find that many use cases where you’d want them separated, so I also stopped trying… :smiley:

And I think the main important thing is, to enable the use cases you mention, which is still possible with how things are currently with Makie :wink:

1 Like