Pluto plots with PlotlyBase/PlotlyJS with working static html export

Hi everyone,

I recently wanted to use PlotlyJS directly in Pluto and still be able to view the plots in the static html export.
I just managed to somehow do that and wanted to post the procedure here to get feedback and eventually have a reference on discourse if anybody wants to achieve the same.

My need for PlotlyJS functionality in Pluto was due to the specific plot type I wanted to produce (Suburst Charts from the plotlyjs main library).
This trace, in fact, doesn’t seem to be exposed by the plotly/plotlyjs backend in Plots.jl and after asking on slack, I was told it would not be possible to plot this specific unsupported trace with Plots.jl without rewriting some code of the package itself.

While the use of WebIO is not yet officially supported, I managed to have a working PlotlyJS plot in pluto using the branch from this pull request.

All was working except that plots would not display on a static exported html file, which is normal as @fonsp pointed out in my comment in the pull request.

Trying to find a way to also make the static export work, I found a similar problem referenced in this issue.
I then tried adapting the proposed solution from @akdor1154 to have PlotlyBase plots show directly in the output of a pluto cell.

This is the piece of code that for me makes the Plot object correctly appear in the pluto cell both in the live notebook and in the static export.

using PlotlyBase, HypertextLiteral

# Custom abstract-trace array to string function
TraceData(data::Array{<:AbstractTrace}) = join(["[",join(string.(data),","),"]"])

function Base.show(io::IO, mimetype::MIME"text/html", p::PlotlyBase.Plot)
	show(io,mimetype,@htl("""
	<script src='https://cdn.plot.ly/plotly-latest.min.js'></script>
	<div style="height: auto">
		<script>
		let parentnode = currentScript.parentElement

	  Plotly.plot(parentnode,$(HypertextLiteral.JavaScript(TraceData(p.data))),$(HypertextLiteral.JavaScript(string(p.layout))),{responsive: true});
		</script>
	</div>
"""))
end

I suppose this breaks any possible interaction between julia and the plot which is coming from PlotlyJS, but I didn’t need that specifically.
Moreover, this also leaves PlotlyBase as the only dependency for plotting (and no requirement for the WebIO supporting branch of Pluto).

As I basically patched this solution without much knowledge about Javascript/HTML, I wanted to know whether there is something inherently wrong in adding the code above to a pluto cell in order to achieve what I wanted.
A potential issue I see is that I reference script containing the minfied plotlyjs code for each cell that contains a plot, but I have no idea how I could achieve the same functionality without putting the script inside the show function.

Here is also the source of the example .jl notebook for testing

Test notebook code
### A Pluto.jl notebook ###
# v0.14.5

using Markdown
using InteractiveUtils

# ╔═╡ a08c16a0-bc90-11eb-0274-e91d841ba34e
begin
	import Pkg
	Pkg.activate(".")
end

# ╔═╡ fa09788b-d9a6-496c-a126-4a1639211aaa
begin
	using PlotlyBase
	using HypertextLiteral
end

# ╔═╡ 923df2dc-3c00-493e-aac9-fbf03bd58fe7
begin
	# Custom abstract-trace array to string function
	TraceData(data::Array{<:AbstractTrace}) = join(["[",join(string.(data),","),"]"])
	
	function Base.show(io::IO, mimetype::MIME"text/html", p::PlotlyBase.Plot)
		show(io,mimetype,@htl("""
		<script src='https://cdn.plot.ly/plotly-latest.min.js'></script>
		<div style="height: auto">
			<script>
			let parentnode = currentScript.parentElement

		Plotly.plot(parentnode,$(HypertextLiteral.JavaScript(TraceData(p.data))),$(HypertextLiteral.JavaScript(string(p.layout))),{responsive: true});
			</script>
		</div>
	"""))
	end
end

# ╔═╡ 35c4a24e-c7e9-4b37-bc95-defb25f83f87
p = Plot(GenericTrace("sunburst",Dict(
  :type => "sunburst",
  :labels => ["Eve", "Cain", "Seth", "Enos", "Noam", "Abel", "Awan", "Enoch", "Azura"],
  :parents => ["", "Eve", "Eve", "Seth", "Seth", "Eve", "Eve", "Awan", "Eve" ],
  :values =>  [50, 14, 12, 10, 2, 6, 6, 4, 0],
  :leaf => Dict(:opacity => 0.4),
  :marker => Dict(:line => Dict(:width => 1)),
  :branchvalues => "total",
)))

# ╔═╡ Cell order:
# ╠═a08c16a0-bc90-11eb-0274-e91d841ba34e
# ╠═fa09788b-d9a6-496c-a126-4a1639211aaa
# ╠═923df2dc-3c00-493e-aac9-fbf03bd58fe7
# ╠═35c4a24e-c7e9-4b37-bc95-defb25f83f87

Edit: Added the TraceData function to correctly deal with multiple traces in a plot

9 Likes

@disberd This is a remarkable small piece of code. You have nailed it: PlotlyJS working without any problem at all inside Pluto notebooks. Moreover, we can have the plot displayed within Pluto’s window, or we can export it as an HTML file.

I tested your example and I just added some new cells where I experimented with different types of plots (2D, 3D, etc.). They all worked in an immaculate way. We can save the plots in html, png, pdf, svg formats. The only thing that I had to change is the name of the plot function: in VSCode we plot with plot(), in your approach we have to start it with a capital letter Plot().

I tried this piece of code (typical PlotlyJS syntax):

begin
	n = 200
	r = LinRange(1e-4,10,n)
	k = LinRange(0,2π,n)
	g = r * cos.(k)'
	f = r * sin.(k)'
	z = (g.^2 .* f.^2) ./ (g.^4 + f.^4)
	#p5 = Plot(surface(x=g,y=f,z=z),Layout(width=700,height=500))
	p5 = Plot(surface(x=g,y=f,z=z))
end

and it took just 2.5ms to get the plot below (while it takes 4s in Jupyter!!!). In case someone has doubts, here it is:

Congrats. Wonderful job.

7 Likes

I am glad I was able to help someone else with thi! :slight_smile:

This is simply due to the fact that the plot() explosed by PlotlyJS is a convenience method to create a SyncPlot to get the WebIO interactivity and attaching the standard Plot object from PlotlyBase to it.
In the approach I put above we are just using the Plot object directly as we are skipping the WebIO integration.
If you wish to also keep the plot() synthax you could simply generate your own method that calls Plot() underneath, but I preferred to keep them separate to avoid potential clash with other plotting packages.

1 Like

Ho no, I’m pretty happy using a capital letter in the plot function. I just mentioned it because some people may get stuck here. Great job.

Cool! I was doing similar experiments for the plotly backend of Plots.jl. You can use 💁 API to make objects available inside JS by fonsp · Pull Request #1124 · fonsp/Pluto.jl · GitHub to make it significantly faster:

https://github.com/fonsp/disorganised-mess/blob/main/very%20very%20fast%20plotly.jl

I lost track of all the different Plotly+Julia combinations, but I thought I would quickly share this in case it is helpful. I plan to revisit plotly+pluto at some point.

4 Likes

Also note that the 2.5 ms runtime is an underestimation, it only measures the time to run the cell’s julia code. This does not include:

  • time to render the result object, i.e. call Base.show(io, best_mime, cell_result)
  • time to transfer data to the browser
  • time for the browser (and plotly) to render the result

The best way to test performance is to use a slider or PlutoUI.Clock to continually trigger the plot to refresh, and use your eyes to guess the frames per second that you are getting. Use bigger datasets to make it easier to see.

1 Like

@fonsp , yes, I know that. Anyway, I can assure you that Pluto is much faster than Jupyter. I have tried both in real time, changing the number of grid points by a significant amount, and in Pluto, the new plot takes less than1s to pop up, while it takes around 5.8s to appear in Jupyter.

I used a slider in Pluto. I still have to see how the PlutoUI.Clock works, cause I could not find it in the docs, but the mistake may be mine.

1 Like

So I had some time to did some further testing based on the input from @fonsp and I have a few updates.

The first one is that indeed there is a speedup when using the code example from fonsp but I suspect it mostly comes from the use of Plotly.react when the div already exists rathern than Plotly.newPlot.
The use of MsgPack as opposed to JSON does bring some potential minor benefit from the visual tests with the PlutoUI.Clock but they don’t seem significant using the example surface plot referenced by @VivMendes.

Another update comes from the fact that PlotlyBase already exports a json() function that takes a Plot variable and directly creates the relevant JSON string to pass to the function, being more precise than my mockup TraceData function above and also including the layout characteristics.

Given the above considerations, my suggested function for showing PlotlyBase plots is the following

function Base.show(io::IO, mimetype::MIME"text/html", p::PlotlyBase.Plot)
	show(io,mimetype,@htl("""
<script src='https://cdn.plot.ly/plotly-latest.min.js'></script>
	<div style="height: auto">
		<script id=plotly-show>
			const PLOT = this ?? document.createElement("div");
			(this == null ? Plotly.newPlot : Plotly.react)(PLOT,$(HypertextLiteral.JavaScript(json(p))));
			return PLOT
		</script>
	</div>
"""))
end

I also uploaded the example notebook used for the tests here:

Hi @disberd. I have experimented with your new function. It looks a little bit faster but there is a point that I think deserves some consideration. In your first approach, if we changed the width of Pluto’s window (for me the default size is impracticable for teaching), the plots’ width follows the change we introduce into the notebook. For example, if I choose

html"""<style>
main {
    max-width: 900px;
    align-self: flex-start;
    margin-left: 200px;
}
"""

the plots will adjust to the new size of the window, which is good indeed. In your new approach, apparently, this functionality is not available: plots keep the default size. I don’t know if I am doing something wrong, or if this is just a point that was not considered in the move from the first to the second version. I think tables adjust to the new width, but plots do not. Both followed the change in width window in your first version.

Indeed you are right.
I completely disregarded the responsivity of the plot that was “hard-coded” in my original example (it was the {responsive: true} as the last argument of the newPlot function.

The issue I am facing now is that I am not able to get responsivity to the div width in my plots anymore, not even with the original code…
So I have to understand what problem is causing this at the moment…

Edit:
I realized what was the issue. I was trying to trigger a resizing just by changing the value inside the cell:

html"""<style>
main {
    max-width: 900px;
    align-self: flex-start;
    margin-left: 200px;
}
"""

The plot resize from plotly from my understanding instead only triggers when you resize the actual window of the browser.
Using this modified code below:

function Base.show(io::IO, mimetype::MIME"text/html", p::PlotlyBase.Plot)
	show(io,mimetype,@htl("""
<script src='https://cdn.plot.ly/plotly-latest.min.js'></script>
	<div style="height: auto">
		<script id=plotly-show>
			const PLOT = this ?? document.createElement("div"); 
			(this == null ? Plotly.newPlot : Plotly.react)(PLOT,$(HypertextLiteral.JavaScript(json(p.data))),$(HypertextLiteral.JavaScript(json(p.layout))),{responsive: true});
			return PLOT
		</script>
	</div>
"""))
end

I was able to get plot resizing when changing the size of the window.
There is still some investigation on how to avoid hard-coding responsivity and accepting other plotly config options.
According to this issue:

It was a design choice by Slygon to move the “config” part of the plot to PlotlyJS rather than plotlybase.

This makes it a bit more difficult to handle it nicely but we defeinitely can do better.

Hi. Your first version works very well. It is always responsive to the change of Pluto’s window width and the plots come out quite fast. At least 6x faster than Jupyter, which is quite something if we talk about heavyweight plotting. I tested also your third version, and in my case, it still forces the plots to go back to the default window size whenever I resize.

I made a PR to Plots.jl to use the leaner HTML show method: embeddable html show for performance in Pluto by fonsp · Pull Request #3559 · JuliaPlots/Plots.jl · GitHub

2 Likes

I did some more testing in the past days and currently settled for the current solution for fast and responsive plotting:

begin
	plotlylower(x) = x
	plotlylower(x::Vector{<:Number}) = x
	plotlylower(x::AbstractVector) = Any[plotlylower(z) for z in x]
	plotlylower(x::AbstractMatrix) = Any[plotlylower(r) for r in eachrow(x)]
	plotlylower(d::Dict) = Dict(
		k => plotlylower(v)
		for (k,v) in d
	)
	
	to_js(x) = HypertextLiteral.JavaScript(Main.PlutoRunner.publish_to_js(plotlylower(x)))
	
	function Base.show(io::IO, mimetype::MIME"text/html", plt::PlotlyBase.Plot)
		show(io,mimetype, @htl("""
			<div>
			<script id=asdf>
			const {plotly} = await import("https://cdn.plot.ly/plotly-2.2.0.min.js")
			const PLOT = this ?? document.createElement("div");
		
			if (this == null) {
		
				Plotly.newPlot(PLOT, $(HypertextLiteral.JavaScript(json(plt))));


			} else {				
				Plotly.react(PLOT, $(to_js((map(plt.data) do x x.fields end))),$(to_js(plt.layout.fields)));
			}
		
			const pluto_output = currentScript.parentElement.closest('pluto-output')

			const resizeObserver = new ResizeObserver(entries => {
				Plotly.Plots.resize(PLOT)
			})

			resizeObserver.observe(pluto_output)

			invalidation.then(() => {
				resizeObserver.disconnect()
			})
		
			return PLOT
			</script>
			</div>
"""))
	end
end

This should combine the best of both worlds and keep the active plot resizing when you change the cell width from within the notebook.

P.S. The new version of PlotlyBase also officially supports the config variable (which was relegated to PlotlyJS before) so it is even easier to pass in other custom parameters apart from responsive: true.
In my current script the code used when the cell is newly created or manually ran uses json(plt) that now also includes the config specified during Plot creation

@disberd , wow, it looks much faster than your first version: heavyweight plots pop up like popcorn. Moreover, it combines responsiveness when we resize Pluto cells’ width, which is critical for me. Great stuff.

I have tested this new version in PlotlyJS v0.14.1 and PlotlyBase v0.4.3 (some interdependencies are precluding me from upgrading the Plotly family), and it works just perfect. For teaching, the combination of PlotlyJS and Pluto is challenging to beat. The world would be close to perfection if @fons could develop a way to auto-numerate equations in LaTeX, and someone could tell me how to do a yy-plot in PlotlyJS.

I would also be interested in equation numbering and I’ll try to look if there is a hack that could be used for that…
Regardint the yyplots, do you mean something like this?

Yep, that’s exactly what I mean. I have tried to adapt that specific code to PlotlyJS, but I am a miserable programmer, so I had no success. I am an economist and have to deal with time series of very different means and variances, and that kind of yy plot (one y-axis on the left, the other on the right) makes two (or more) series easy to be visually compared. I can apply linear transformations on one-time series and make them comparable, but this is too much for undergrad economics students. The yy-plot does the job quickly. PGFPlotsX and Plots do it quite easily, but I prefer PlotlyJS for obvious reasons for teaching or doing presentations. For its interactive functionalities, PlotlyJS is fantastic.

Here is the code translated to the PlotlyBase/PlotlyJS lingo:

trace1 = scatter(
  x = [1, 2, 3],
  y = [40, 50, 60],
  name = "yaxis data",
);

trace2 = scatter(
  x = [2, 3, 4],
  y = [4, 5, 6],
  name = "yaxis2 data",
  yaxis = "y2",
);

layout = Layout(
  title = "Double Y Axis Example",
  yaxis_title = "yaxis title",
  yaxis2 = attr(
    title = "yaxis2 title",
    titlefont_color=  "rgb(148, 103, 189)",
    tickfont_color = "rgb(148, 103, 189)",
    overlaying = "y",
    side = "right",
  )
);

Plot([trace1,trace2],layout)

And here is the result on Pluto:

Once you know the rules to apply the translation between standard ploltyjs and the julian counterpart is quite straightforward:

  • Translate every : into = to assign a variable
  • Change strings from using 'text' to using "text"
  • Change nesting of javascript object using either _ or attr():
    • _ is convenient when you child has only one attribute like it was the case of yaxis: {title: yaxis title}} in the javascript
    • attr() is more convenient when you have nested object with multipl attributes, like the yaxis2 above

Most of the plotly traces type have corresponding convenience functions in the julia version (see how I used directly the scatter function above to represent the trace objects with type: 'trace').

For the ones that don’t, like the Sunburst Charts I referenced in the original post above, you would have to use the GenericTrace from PlotlyBase.

Most of what I put above is explained in more detail in the PlotlyJS Documentation

1 Like

@disberd, thank you very much. I had been close to what you sent me, but not close enough. In your case, the code works, and it allows me to understand the logic behind PlotlyJS, which is quite simple but powerful. I am accustomed to doing this type of yy-plotting with PGFPlotsX, which requires quite a lengthy adaptation of raw code from the LaTeX “pgfplots” and TikZ packages, and it takes much more lines of code than with PlotlyJS.

This plot was produced with your code in PlotlyJS (around 30 lines)

This with PGFPlotsX (twice as many lines):

Thanks for your help. It has been invaluable.

1 Like

Hi @disberd. I have been experimenting with your latest code to integrate PlotlyJS into Pluto notebooks. But first, let me try to summarize what I get from your first and your latest approaches:

  1. Your first approach works exceptionally well. I haven’t found one single undesirable aspect out of it. It is fast, PlotlyJS works as expected and automatically adjusts the plots’ when we resize Pluto’s window, with no flaws at all.

  2. Your latest approach increases the speed significantly. It also increases the plots’ reactivity (i.e., zoom in/zoom out are automatic; we do not need to click anywhere to get this functionality available). In the notebooks that include many plots, the speed is easily noticed.

The increase in speed and reactivity may have led to an undesirable aspect that I have spotted. Due to the pandemic, I am putting many exercises in the form of Pluto notebooks. So I have played a lot with plots and PlotlyJS and found this (in your latest approach): when I run a particular cell, all cells are re-run, but Pluto does not take into account the information in the layouts of the other cells, neither when one plot is composed by two (or more) subplots. Indeed, if I manually re-run all cells after running a particular cell, the outputs will come out OK. But if I have, say, 10 independent exercises, the change of one single cell will produce many changes and much work to do. Moreover, manipulating a slider leads to much confusion in the plotting output in a large notebook. This is not an issue of PlotlyJS because the same exercises do not suffer from this problem in your first approach.

Let me frame this in the simplest possible example. Suppose I have two separate plots, and the third one uses the first two as subplots. Sorry for the lengthy MWE below, but this is not easy to explain without a complete example. To avoid multiple definitions, variables and parameters all have a 5 in this case (Exercise 5):

#cell1

begin
	using PlotlyBase
	using HypertextLiteral
	using PlutoUI
end

#cell 2

begin
	Abar5 = 7.6
	λ5 = 0.5
	m5 = 2.0
	ϕ5 = 0.2
	rbar5 = 2.0
	γ5 = 4.5
	Yᴾ5 = 14.0
	πᵉ5 = 2.0
	ρ5 = 0.0
	πd_max5 = (Abar5 / (λ5 * ϕ5)) - rbar5/λ5 
	πs_min5 = πᵉ5 - γ5 * Yᴾ5 + ρ5 
end

#cell 3

begin
	Y5 = 12.0:0.01:15.0
	
	πd5 = πd_max5 .- ((1 ./(m5 .* ϕ5 .* λ5)) .* Y5) 
	πs5 = πs_min5 .+ γ5 .*Y5             
	
	trace9_0 = scatter(; x = [14,14] , y = [-7,14] , mode = "line", 
		line_width = "4", name = "Yp", line_color = "Cyan")
	
	trace9_1 = scatter(; x = Y5, y = πd5, mode="lines" , 
		line_color = "Blue", line_width = "3", name = "AD")
	
	trace9_2 = scatter(; x = Y5, y = πs5, mode="lines" , 
		line_color = "Red", line_width = "3", name  = "AS")
	
	trace9_3 = scatter(; x = [14.0], y = [2.0], text =["1"], 
		textposition = "top center", name = "Eq.", mode = "markers+text", 
		marker_size = "12", marker_color = "Blue", textcolor = "Black")
	
	
	layout9_3 = Layout(;title="Macroeconomic Equilibrium",
        xaxis = attr(title=" GDP trillion dollars (Y)", 
			showgrid=true, zeroline=false),
        xaxis_range = [13.5, 14.5],	
        yaxis = attr(title="Rate of inflation (π)", zeroline = false),
		yaxis_range = [-0 , 5])

   p9_3 = Plot([trace9_0, trace9_1, trace9_2, trace9_3], layout9_3)
	
end

The plot that comes out is this one:

#cell 4

begin
	r5 = 0.0:0.01:5.0
	
	πmp5 = - (rbar5 ./ λ5) .+  r5 ./ λ5 
	
	trace4_0 = scatter(; x = r5, y = πmp5, mode = "lines", line_color = "Brown", 
		line_width = "3", name  = "MP")
	
	trace4_1 = scatter(; x = [3.0], y = [2.0], text =["1"], 
		textposition = "top center", name ="Eq.", mode="markers+text", 
		marker_size = "12",
		marker_color = "Brown", textcolor = "Black")
	
	layout4_1 = Layout(;title="MP Function",
        xaxis = attr(title=" Real interest rate (r)", showgrid=true, 
			zeroline=false),
        xaxis_range = [2.0, 4.5],
		xaxis_tickvals = [2, 2.5, 3, 3.5, 4, 4.5],
        yaxis = attr(title="Rate of inflation (π)", zeroline = false),
		yaxis_range = [-0 , 5])

   p4_1 = Plot([trace4_0, trace4_1], layout4_1)
end

The plot that comes out is this one:

#cell 5

p02 =[p4_1   p9_3]

The plot that comes out is this one:

Now I go back and run #cell 3. The entire notebook is re-run, and now the plot of #cell 5 looks like this:

I don’t know what causes this problem. But having played a lot with plots, it seems to me that the information in the Layouts seems to be overlooked in the re-runs, particularly the information concerning the x-axis.

I am delighted with your first approach. I need PlotlyJS, and I need Pluto; together, they are working extremely well. I am pointing this out because it seems that you really want to improve on your first approach. I hope this my MWE is readable.

Edited: I edited the code in a more friendly way.

Thanks a lot for taking the time to write this post even though you were fine with the first solution.
As you correctly inferred, I am trying optimize the plotting as much as possible because I rely heavily on Pluto nowadays for my own research and investigation at work and want to streamline the workflow as much as possible.

Indeed I didn’t try thoroughly the new method but it does breaks with custom layout (especially the xaxis_range in your example case).
I did some other tests and decided to revert to the json function from PlotlyBase as the speed was anyway mostly given by using Plotly.react from some preliminary tests.

Please try this code to see if you also get working behavior:

function Base.show(io::IO, mimetype::MIME"text/html", plt::PlotlyBase.Plot)
       # Remove responsive flag on the plot as we handle responsibity via ResizeObeserver and leaving it on makes the div flickr during updates
	hasproperty(plt,:config) && plt.config.responsive && (plt.config.responsive = false)   
	show(io,mimetype, @htl("""
			<div>
			<script id=asdf>
			const {plotly} = await import("https://cdn.plot.ly/plotly-2.2.0.min.js")
			const PLOT = this ?? document.createElement("div");
		

		
			Plotly.react(PLOT, $(HypertextLiteral.JavaScript(PlotlyBase.json(plt))));


		
			const pluto_output = currentScript.parentElement.closest('pluto-output')

			const resizeObserver = new ResizeObserver(entries => {
				Plotly.Plots.resize(PLOT)
			})

			resizeObserver.observe(pluto_output)

			invalidation.then(() => {
				resizeObserver.disconnect()
			})
		
			return PLOT
			</script>
			</div>
"""))
end

I check at the beginning for the responsive flag inside the Plot.config as now responsivity is handled with the ResizeObserver and keeping the responsive: true flag makes the div disappear very briefly everytime the plot is recreated.
I check to see if the plot object has a config field (it won’t if you use PlotlyBase < 0.6.0) so it shouldn’t produce an error for your case.

1 Like