Plot without holding data in memory

Specific Domains Visualization
, , ,
1

I’d like to overlay hundreds of gigabytes of data onto a single plot. I don’t need this plot to be interactive, I just want to “paint pixels”.

Is there any plotting backend, or option with Plots.jl that allows for plotting without holding the data in memory?

1 Like
3

Well, you cannot plot such an amount of data directly, because no screen has that many pixel. You need to display a summary of the data anyway, and so you need to write a script that extracts the summary you want to see and than plot that.

1 Like
4

Gotcha. I was wondering if the capability already existed in some plotting package out there. I want to set the axes limits, and then plot more data than can fit in RAM at any given time.

This wouldn’t be possible if the axes weren’t fixed, because after receiving new data the plotting package would want to re-scale all data to produce better axes, data interpolating, etc.

But if the axes are fixed, then theoretically you could just plot each trajectory independently and plot more data than memory can hold.

Basically, I want to update a PNG (or some other static image format equivalent) file with hundreds of GB of data (more than a computer’s memory can hold).

5

I have found this post. Your algorithm could look like

for all chunks of data 
  read chunk
  convert chunk to pixels
  set pixels
end
6

Yup, that’s exactly what I want to do. The “convert chunk to pixels” step is what I was hoping already exists in a plotting library. The post linked asks a similar question to mine. The solution to that post was to use a scatter plot, which would still hold all of the data in a figure object.

7

The same pixel would have multiple values? Then you’d probably have to collect the pixels yourself in a preprocessing step.

Just to be clear: plotting out of core data sets doesn’t seem like an everyday requirement to me.

8

In my experience, if you plot to PDF or PS (with Gadfly.jl, for example), the result is a PS/PDF that has all the points inside it, and the PS/PDF reader will create the exact image when the page is rendered, in other words, the PS/PDF is more akin to an SVG than a PNG.

1 Like