# How to plot ~10 billion datapoint time series efficiently?

**URL:** <https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228>\
**Category:** Visualization\
**Tags:** question, plotting, diffeq\
**Created:** [May 18, 2022, 1:09am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228 "2022-05-18T01:09:30Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![hk69](https://avatars.discourse-cdn.com/v4/letter/h/c57346/32.png) [@hk69](https://discourse.julialang.org/u/hk69)\
**Post date:** [May 18, 2022, 9:16pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/23 "2022-05-18T21:16:58Z")

</div>

I am plotting a simple x,y line plot. See the partial plot I posted in reply to the other post

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [May 18, 2022, 9:33pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/24 "2022-05-18T21:33:20Z")

</div>

So, keeping it simple. For example:

```julia
julia> using Plots

julia> x = 0:1e-6:2π ; y = sin.(x); # ~2π million points

julia> plot(layout=(2,1))

julia> plot!(x,y,linewidth=2,subplot=1)

julia> sample = 1:1000:length(x);

julia> plot!(x[sample],y[sample],linewidth=2,subplot=2)

```

Gives you two essentially identical plots, but the second one has 1/1000 of the points:

![image](https://global.discourse-cdn.com/julialang/original/3X/a/5/a5545f55eb633dd2323e4184b26324ab06677b52.png)

You perhaps need a smarter sampling strategy if some part of the plot doesn’t look like it should, but that can even be done “by hand”.

---

<div class="post-metadata">

**Author:** ![hk69](https://avatars.discourse-cdn.com/v4/letter/h/c57346/32.png) [@hk69](https://discourse.julialang.org/u/hk69)\
**Post date:** [May 18, 2022, 9:45pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/26 "2022-05-18T21:45:57Z")

</div>

Yes, this is the first thing I tried but it doesn’t preserve the shape of the data. Do you have any suggestions for a smarter sampling strategy?

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [May 18, 2022, 10:01pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/27 "2022-05-18T22:01:19Z")

</div>

I was thinking that if you are ok with that syntax, you could simply concatenate by hand different sampling frequencies tuned to the data variation in each region. That, of course, because it seems that you are only doing this for a single figure. For a more automated and sophisticated solution, we should implement one smarter sampling method as suggested above.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [May 18, 2022, 10:01pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/28 "2022-05-18T22:01:27Z")

</div>

If your plot is log-spaced, your sampling indices should likewise be log-spaced using something like this:

```julia
julia> function logsample(x, y, N)
           length(x) == length(y) || ArgumentError("Mismatched input sizes")
           idx = unique([round(Int, exp10(v)) for v in LinRange(0, log10(length(x)), N)])
           x[idx], y[idx]
       end

```

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [May 19, 2022, 1:41am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/29 "2022-05-19T01:41:06Z")

</div>

I recall people being satisfied with the publication quality graphs they constructed using [PyPlot.jl](https://github.com/JuliaPy/PyPlot.jl). It wraps matplotlib, which is a well known and highly developed Python plotting package.

---

<div class="post-metadata">

**Author:** ![jheinen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jheinen/32/2787_2.png) [@jheinen](https://discourse.julialang.org/u/jheinen)\
**Post date:** [May 19, 2022, 6:41am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/30 "2022-05-19T06:41:48Z")

</div>

I don’t understand why GR should not produce publication quality plots. Could you provide a MWE?

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [May 19, 2022, 7:16am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/31 "2022-05-19T07:16:14Z")

</div>

I suspect it’s because that plot makes a heatmap (with a very colorful color map), which doesn’t reproduce well in print. For the 2D line data of OP a (simplified) line plot would probably be best in terms of plot quality.

---

<div class="post-metadata">

**Author:** ![hk69](https://avatars.discourse-cdn.com/v4/letter/h/c57346/32.png) [@hk69](https://discourse.julialang.org/u/hk69)\
**Post date:** [May 19, 2022, 4:12pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/32 "2022-05-19T16:12:08Z")

</div>

Yes, PyPlot is nice but it can’t plot 10 billion data points efficiently.

---

<div class="post-metadata">

**Author:** ![hk69](https://avatars.discourse-cdn.com/v4/letter/h/c57346/32.png) [@hk69](https://discourse.julialang.org/u/hk69)\
**Post date:** [May 19, 2022, 4:15pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/33 "2022-05-19T16:15:03Z")

</div>

GR.shade is producing a grainy curve since it is sort of a heatmap of the data

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 19, 2022, 4:50pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/34 "2022-05-19T16:50:44Z")

</div>

GMT let’s you plot as many billion points as you want, but it’s constrained to the same memory conditions as the others so you would have to do it by chunks and expect `ghostscript` is able to convert the huge resulting postscript file.

However, I maintain that that no-one is able to see billions of points in a plot so don’t understand this request.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [May 19, 2022, 5:15pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/35 "2022-05-19T17:15:01Z")

</div>

> [@hk69](#):
>
> Yes, PyPlot is nice but it can’t plot 10 billion data points efficiently.

One question then is how long are you willing to wait for the plot to finish, and how often do you need to repeat it? Or is the memory requirement the issue (as PyPlot can’t stream the data, don’t know actually)?

> [@joa-quim](#):
>
> However, I maintain that that no-one is able to see billions of points in a plot so don’t understand this request.

The goal is probably not to directly show all billions of points, but to produce faithful visual representation of the underlying data that shows the overall trajectory in a certain amount of detail. And the challenge here is that the underlying data isn’t straightforward to subsample/decimate without losing the overall shape of the curve. Or at least, it takes a specific type of decimation algorithm, like one of the algorithms/papers mentioned earlier (or something like [CGAL 5.6 - 2D Polyline Simplification: User Manual](https://doc.cgal.org/latest/Polyline_simplification_2/index.html)). And plotting as a heat-map doesn’t make much sense for a trajectory.

So brute-forcing plotting the 2D line segments can still be the best approach as it is simple (but somewhat time-consuming), while not introducing any smoothing artifacts that decimation would most likely introduce.

Also, I somewhat feel that doing a simulation at such a high resolution and saving that data, and then **not** looking at the evolution in its full resolution seems a waste. Usually only a subset of simulation timesteps is saved (mostly due to storage restrictions), which provide enough insight into the system behaviour, and then OP’s problem would simply not exist. But apparently there’s a reason here to save such a high number of time steps _and_ visualize them.

Edit: bad grammar fixes, missing word, clearified question

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 19, 2022, 9:15pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/36 "2022-05-19T21:15:07Z")

</div>

> [@paulmelis](#):
>
> But apparently there’s a reason here to save such a high number of time steps _and_ visualize them.

Sure, but the point is that they can’t be visualized **all** at same time.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [May 20, 2022, 6:54am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/37 "2022-05-20T06:54:25Z")

</div>

> [@joa-quim](#):
>
> Sure, but the point is that they can’t be visualized **all** at same time.

That depends on your definition of “at the same time”. If you take those billions of points and represent them with a heat map (which would involve mapping/plotting them all) then the plot still shows them “all at the same time” 😉 And making a heatmap of a subset of the data will most likely lead to a different plot output (depending on the plot setup). But I agree that for the goal of providing an overview of the data it’s not very useful to directly plot all those points.

I’ve seen examples of N-body simulations of billions of mass particles where an interactive viewer was built to be able to visualize both the high-level structure, as well as being able to zoom down to the individual point level. Yet, such exploratory visualization doesn’t seem to be the goal here.

---

<div class="post-metadata">

**Author:** ![liuyxpp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/liuyxpp/32/9870_2.png) [@liuyxpp](https://discourse.julialang.org/u/liuyxpp)\
**Post date:** [May 20, 2022, 6:57am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/38 "2022-05-20T06:57:47Z")

</div>

> [@paulmelis](#):
>
> examples of N-body simulations of billions of mass particles where an interactive viewer was built to be able to visualize both the high-level structure, as well as being able to zoom down to the individual point level.

Wow, that should be a daunting job. I can only admire and envy someone has a tool like this.

---

<div class="post-metadata">

**Author:** ![jonathanBieler](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)\
**Post date:** [May 20, 2022, 8:45am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/39 "2022-05-20T08:45:43Z")

</div>

I would try something like this, basically I go over the vectors and add indices for plotting only when the angle changes significantly, so that if many points follow the same direction they are skipped. You might want to add a minimum step to make sure you reduce enough and also do something to deal with the log-scale.

```julia
function downsample(x, y, tol) 
    idx = [1]
    last_idx = 1
    last_angle = 0
    for i in 2:length(y) #don't use OffsetArray ;)
        δy = y[i] - y[i-1]
        δx = x[i] - x[i-1]
        angle = atan(δy, δx)
        if abs(last_angle - angle) > tol 
            push!(idx, i)
            last_idx = i
            last_angle = angle
        end
    end
    idx
end

tol = 0.1
x = LinRange(0,2π, 10_000)
y = @. sin(x) + cos(2x)^4

idx = downsample(x, y, tol)

@show length(idx)
plot(x = x[idx], y = y[idx], Geom.line, Geom.point)

```

 ![image](https://global.discourse-cdn.com/julialang/original/3X/2/9/29156b061a4f8eed5f03f52d6ea33cbc9dde183a.png)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [May 20, 2022, 9:32am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/40 "2022-05-20T09:32:12Z")

</div>

That is a nice idea. From quick test below, using the derivative instead of atan may provide better coverage for the same level of data reduction?

 ![Plots_downsample_based_on_angle_vs_derivative](https://global.discourse-cdn.com/julialang/original/3X/f/7/f7791f97a670c833269d44a4e3b781487d0f9ba8.jpeg)

> **Code**
>
> ```julia
> 
> function downsample(x, y, tol) 
> idx = [1]
> last_idx = 1
> last_angle = 0
> for i in 2:length(y) #don't use OffsetArray ;)
> δy = y[i] - y[i-1]
> δx = x[i] - x[i-1]
> angle = atan(δy, δx)
> if abs(last_angle - angle) > tol 
> push!(idx, i)
> last_idx = i
> last_angle = angle
> end
> end
> idx
> end
> 
> function downsample2(x, y, tol) 
> idx = [1]
> last_idx = 1
> last_derivative = 0
> for i in 2:length(y) #don't use OffsetArray ;)
> δy = y[i] - y[i-1]
> δx = x[i] - x[i-1]
> δx==0 ? deriv = Inf : deriv = δy/δx
> if abs(last_derivative - deriv) > tol 
> push!(idx, i)
> last_idx = i
> last_derivative = deriv
> end
> end
> idx
> end
> 
> tol = 0.1
> x = LinRange(0,2π, 10_000)
> y = @. 2x * sin(x)^2 + 4x * cos(2x)^4
> 
> idx = downsample(x, y, 0.1)
> idx2 = downsample2(x, y, 2.6)
> 
> using Plots
> p1 = plot(x,y, c=:blue)
> plot!(x[idx], y[idx], marker=:o, ms=2, ls=:dash, c=:red, title="atan angle ($(length(idx)) points)")
> p2 = plot(x,y, c=:blue)
> plot!(x[idx2], y[idx2], marker=:o, ms=2, ls=:dash, c=:green, title="derivative ($(length(idx2)) points)")
> plot(p1,p2, size=(1000,500))
> 
> ```

In any case, for examples like this, the use of FFT-based data compression is more common?

---

<div class="post-metadata">

**Author:** ![drvi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drvi/32/13236_2.png) [@drvi](https://discourse.julialang.org/u/drvi)\
**Post date:** [May 20, 2022, 9:40am UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/41 "2022-05-20T09:40:29Z")

</div>

This blog post sounds relevant [Drawing 2.7 billion Points in 10s | by Simon Danisch | HackerNoon.com | Medium](https://medium.com/hackernoon/drawing-2-7-billion-points-in-10s-ecc8c85ca8fa)

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 20, 2022, 2:52pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/42 "2022-05-20T14:52:05Z")

</div>

Guys, those [Douglas-Peucker](https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm) guys (sorry Ramer, I didn’t know about you) have worked on this

```julia
using GMT

x = LinRange(0,2π, 10_000);
y = @. 2x * sin(x)^2 + 4x * cos(2x)^4;
xy = gmtsimplify([x y], tol=0.005);
imshow(xy, marker=:point, lw=0.5, figsize=(12,12), title="Douglas-Peucker ($(size(xy,1)) points)")

```

 ![GMTjl_tmp](https://global.discourse-cdn.com/julialang/original/3X/9/f/9f856b475bb4e26074b130ed4c3fe643ecb9444b.png)

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 20, 2022, 3:00pm UTC](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228/43 "2022-05-20T15:00:07Z")

</div>

Very ingenious but since the image has 600 x 960 pixels and continents are about 1/3 of Earth surface and more than half of if have no points, it turns out that of those 2.7 bilions points we are only _seeing_`600 * 960 / 3 / 2 ~ 100_000` of them.

[Previous page](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228.md?page=1)

[Next page](https://discourse.julialang.org/t/how-to-plot-10-billion-datapoint-time-series-efficiently/81228.md?page=3)