It seems like the vegalite package offers more or less a direct translation of the JSON spec for vegalite, but does not offer higher level constructors. Is that more or less correct? It seems like there is room for some sort of type hierarchy that specifies various layers like histograms, curves, KDEs, polygons, points, bars, etc, and then have a Vegalite.VLSpec constructor that takes a sequence of these types and builds a corresponding specification. Does that make some kind of sense? Is that something that’s already planned @davidanthoff?
You may want to take a look at Gadfly.jl. AFAIK, it has the closest syntax to ggplot’s grammar of graphics.
Ggplot(ly) is actually the last remaining reason I keep R installed, I haven’t yet found a perfect replacement for ggplot’s syntax and plotly’s interactivity. A recent project has had me struggling with ggplot’s poor performance, so I’m going to try to make the switch to PlotlyJS.jl. I may end up using a combination of that and Gadfly.
I will try out Gadfly. it’s been on my radar and it’s pretty cool that it’s 100% Julia with rendering to svg. On the other hand vegalite has a big vis group doing development behind it so it would be nice to take advantage of that, if there were a sufficiently high level interface to produce plots. It’s hard enough to figure out what’s going on in your data or models… also having to hand hold the vis library through a JSON spec is not going to be friendly to data analysts.
but it seems like it wouldn’t be too hard to have a kind of compiler that takes a bunch of special structs and grinds out the JSON.
There’s also the Plots.jl ecosystem, which leverages Julia’s type system to define “recipes” for plotting all sorts of data structures, and its extension StatsPlots which offers I think all of the things on your list. Look at the readme here: https://github.com/JuliaPlots/StatsPlots.jl
It’s not clear to me how composable the plots/statsplots are. For example, is it possible to do a heatmap and then overlay a specific subset of points, and then overlay 10 posterior samples from a smoother?
And I don’t mean, is it possible to create a type called “HeatMapPlusPointsAndSmoother” and then create a recipe for it… I mean, can I call for a heat map, and then take the output of that and add to it a points plot and add to that a set of curves?
Composability of a graphics system is key in my mind for it to make sense in a data analysis setting.
VegaLite has the explicit notion of a layer built into the underlying library, so this facilitates composability quite a bit. Gadfly also seems to have stacking and layers built in as primitive. I guess I’ll do some trials with Gadfly.
I don’t think that’s a problem (although I’m not a VegaLite user, mainly because I’m a Matlab convert so never felt comfortable with Grammar of Graphics).
I’ll admit that the following is uniquely ugly, but it should give an idea of the kind of composability you describe - basically just use plotting functions with a ! and they’ll add a layer to the existing plot:
Also just to add, I think your HeatMapPlusPointsAndSmoother isn’t quite what the recipes system is trying to do - you don’t create artificial types to then design recipes for them, but rather you create recipes for existing types that have relevance outside a plotting context (e.g. a type that describes a stochastic differential equation and its solution, or a type that describes a probability distribution). The recipes then provide information to Plots.jl (or indeed other plotting packages that implement recipes) about how an instance of this type should be plotted.
There are obvious improvements tbd e.g. allow color=["Smooth"] in plot p1. For p3, there is also Heatmap.jl, which is a work in progress. For p6, we don’t specifically have a stat to do pointintervals (there is Geom.errorbar), but you can write custom statisitcs and guides (and Heatmap.jl is an example of that). See NEWS.md for what’s in the next release (imminent).
thanks very much for the Gadfly examples! I think I will focus on Gadfly for my data analysis tutorials.
Also thanks @nilshg for winning both the “helpful example” award and the “most ugly plot” award
Yes, and that makes good sense. I think your example shows that you don’t want to represent composite plots by types (it’s a combinatorial nightmare!), you want plots for specific meaningful types and then composition ability of those plots. Composition is a really essential component of graphics because you often want to show how several things work together. For example, you might have some points, and then two ways of smoothing, so you want to be able to plot the points, plot smoother type 1 and then plot smoother type 2 and show how they behave differently/similarly. Or you might have a heatmap from data, plus an analytical surface that you want contour plotted onto the heatmap… or whatever. If you had to create a type for each composition it’d be a nightmare. Thanks for your example, since it shows you don’t need special “composite types”, you can just plot! over the top.
Hey @Mattriks, thanks again for those examples. I’m trying to do some plots and sometimes you just want to control colors yourself. Is it possible to control the color of say a point, or the fill underneath a density curve by specifying a color, rather than specifying it as an aesthetic mapped to a data value?
like plot(…,color = “red”) and not have it look for a column named “red” in the dataset but instead just use the color red?
Also is it possible to specify a computed value without computing it and adding it to the dataset, like suppose for example I have a column called “value” and I want to plot value^2?
doesn’t work of course, because you can’t square the symbol value. I’m down with that, but can I supply a function like y = z -> z.value^2 or something similar?
Also, there appears to be something about Gadfly that holds back Turing so recent versions can’t be installed. Is that likely to change soonish at the next release you mentioned?
So plotting a set of points that are the result of applying some function to the values in each row…
Obviously, I can always do d |> @map(....) |> plot(...) to create the data mappings. I just wondered if there was some built in plotting that could describe a more arbitrary mapping than 1-1 from an axis to a single column of the dataset.
That is roughly correct. The general idea is that vega-lite is already a higher level layer on top of vega itself. We do have a number of syntactic shortcuts over the pure JSON case, described here that often make the VegaLite.jl version of a plot significantly more concise than the corresponding vega-lite JSON. But in general we do want to be close to the underlying vega-lite story.
So this is painful right now The point part is simple, i.e. data |> @vlplot(:point, :x, :y), but doing a regression or trend line is very verbose, see here.
There is an open issue on the vega-lite repo to add a macro for this kind of situation here. My expectation is that once that is implemented we’ll be able to just do something like data |> @vlplot(x=:x, y=:y) + @vlplot(:point) + @vlplot(:smooth).
This is also a pretty good example why I’m hesitant to add special high-level constructs to VegaLite.jl for this kind of situation: I think the proper way to add this functionality is in the underlying vega-lite library, and then it automatically surfaces for the Julia wrapper.
Putting them together is a pain because there is no easy way to generate a normalized histogram that I could find, so the y axis of the two individual plots don’t match up nicely. I’ll try to raise this with the vega-lite folks, this seems a pretty common kind of plot that shouldn’t be so complicated.
I think there might also be room for a vega-lite macro here that would allow one to do the density without specifying a manual transform.
In VegaLite.jl you can use the compute transform to do pretty arbitrary computations, see here. Of course, I’m always in favor of using Quer.jl, though I think computations inside the vega-lite spec make most sense if you have an interactive plot and the computation needs to update in response to the user selections. That scenario is difficult to handle with precomputed transformations, but should work with native vega-lite transforms.
One other thing to point out about VegaLite.jl is that there we have DataVoyager.jl as an interactive UI for data exploration that works hand-in-hand with VegaLite.jl (and we have a couple more similar interactive tools in the pipeline, they are just not ready yet). Not sure how useful that would be for your particular scenario, though.
Wow, so Vega/Vegalite is doing its own regression/loess/etc calculations. That’s interesting.
How about composing plots from Julia? Is there some composition type operator, like + which would let you take two different vegalite plots and combine them together to get a layered plot? Or do you always have to do one plot with two layers?
This is an example of a histogram that is precomputed, but I don’t think that makes things more complicated.
Yes, you can compose plots, but I haven’t written that documentation chapter yet. Mostly because I’m still not a 100% satisfied with what I’ve come up with so far. But the short of it is this:
You can concatenate with brackets: [p1 p2] and [p1; p2] for example
You layer with +, but the first plot will not form a layer but instead have properties that affect the whole plot. Say p1 + p2 + p3 will create a layered plot with two layers (p2 and p3), and p1 will have properties that affect the whole plot.
If you do p1 + p2 and p1 is of a type that requires a sub-spec, then p2 will become that sub spec.
Thanks @davidanthoff for explanations and examples. somehow I had missed your gallery of advanced examples. While I can see the appeal of keeping VegaLite close to the metal of the VegaLite library so as to continuously reap the improvements to that library… I am torn with what I think of as a rather verbose syntax to do relatively common things. Perhaps what is needed is an EDA toolkit of opinionated macros that “compile” to vegalite specs… as a secondary Package on top of vegalite… I’m thinking it’d be nice to say something like
@ggvl(histogram(:height, fill=:lightblue)+ density(:height,fill=:green,alpha=0.25)+background(color=:grey)+title("Histogram and Density Plot of Human Heights"))
Or something similar to that. I feel that is similarly concise to your suggestion? The big benefit would be that it wouldn’t require any additional “ideas” on the Julia side of things. The logic here would be that the first @vlplot call defines properties for the whole plot, and the next two @vlplot calls each become one layer in a layered plot.
Having said that, I’ve also started a new higher level Julia package, like what you suggested, see here for a discussion, and the package is QuickVega.jl (but be warned, the package is empty at this point). But there is a crucial difference to your suggestion: in my mind QuickVega.jl will not have a grammar of graphics style API, instead it will just be a “one function -> one figure” type API. But it will be possible to combine the plots that you get via that route with “normal” VegaLite.jl plots via composition.