# Plot without holding data in memory

**URL:** https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444
**Category:** Visualization
**Tags:** package, plotting, memory, plots
**Created:** [April 13, 2022, 5:25pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444 "2022-04-13T17:25:33Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![jcarpi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jcarpi/32/35416_2.png) [@jcarpi](https://discourse.julialang.org/u/jcarpi)
#### Post date: [April 13, 2022, 5:25pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/1 "2022-04-13T17:25:33Z")

</div>

I’d like to overlay hundreds of gigabytes of data onto a single plot. I don’t need this plot to be interactive, I just want to “paint pixels”.

Is there any plotting backend, or option with `Plots.jl` that allows for plotting without holding the data in memory?

---

<div class="post-metadata">

### Author: ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [April 14, 2022, 11:15am UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/3 "2022-04-14T11:15:16Z")

</div>

Well, you cannot plot such an amount of data directly, because no screen has that many pixel. You need to display a summary of the data anyway, and so you need to write a script that extracts the summary you want to see and than plot that.

---

<div class="post-metadata">

### Author: ![jcarpi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jcarpi/32/35416_2.png) [@jcarpi](https://discourse.julialang.org/u/jcarpi)
#### Post date: [April 15, 2022, 7:20pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/4 "2022-04-15T19:20:34Z")

</div>

Gotcha. I was wondering if the capability already existed in some plotting package out there. I want to set the axes limits, and then plot more data than can fit in RAM at any given time.

This wouldn’t be possible if the axes weren’t fixed, because after receiving new data the plotting package would want to re-scale all data to produce better axes, data interpolating, etc.

But if the axes _are_ fixed, then theoretically you could just plot each trajectory independently and plot more data than memory can hold.

Basically, I want to update a PNG (or some other static image format equivalent) file with hundreds of GB of data (more than a computer’s memory can hold).

---

<div class="post-metadata">

### Author: ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [April 15, 2022, 7:29pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/5 "2022-04-15T19:29:23Z")

</div>

> [@jcarpi](#):
>
> Basically, I want to update a PNG (or some other static image format equivalent) file with hundreds of GB of data (more than a computer’s memory can hold).

I have found this [post](https://discourse.julialang.org/t/how-to-draw-a-pixel-in-julia-graphics/37316). Your algorithm could look like

```julia
for all chunks of data 
  read chunk
  convert chunk to pixels
  set pixels
end

```

---

<div class="post-metadata">

### Author: ![jcarpi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jcarpi/32/35416_2.png) [@jcarpi](https://discourse.julialang.org/u/jcarpi)
#### Post date: [April 15, 2022, 7:36pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/6 "2022-04-15T19:36:53Z")

</div>

Yup, that’s exactly what I want to do. The “convert chunk to pixels” step is what I was hoping already exists in a plotting library. The post linked asks a similar question to mine. The solution to that post was to use a scatter plot, which would still hold all of the data in a figure object.

---

<div class="post-metadata">

### Author: ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)
#### Post date: [April 15, 2022, 7:39pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/7 "2022-04-15T19:39:53Z")

</div>

> [@jcarpi](#):
>
> The solution to that post was to use a scatter plot, which would still hold all of the data in a figure object.

The same pixel would have multiple values? Then you’d probably have to collect the pixels yourself in a preprocessing step.

Just to be clear: plotting out of core data sets doesn’t seem like an everyday requirement to me.

---

<div class="post-metadata">

### Author: ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [April 15, 2022, 9:19pm UTC](https://discourse.julialang.org/t/plot-without-holding-data-in-memory/79444/8 "2022-04-15T21:19:41Z")

</div>

> [@goerch](#):
>
> The same pixel would have multiple values?

In my experience, if you plot to PDF or PS (with Gadfly.jl, for example), the result is a PS/PDF that has all the points inside it, and the PS/PDF reader will create the exact image when the page is rendered, in other words, the PS/PDF is more akin to an SVG than a PNG.
