# Infinite Increasing Memory Usage Until Crash
**URL:** https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254
**Category:** General Usage
**Tags:** question
**Created:** [April 26, 2020, 6:34pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254 "2020-04-26T18:34:53Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@duncanam](https://discourse.julialang.org/u/duncanam)
#### Post date: [April 26, 2020, 6:34pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/1 "2020-04-26T18:34:53Z")
I’ve made a small script to assist with visualizing some data. The script `include()`s another script that has a module to do all my isosurface and 3D plotting using Makie, as well as all the reading in with CSV. I save all the images to file, and then use FFMPEG to compile them together into a movie. I’ll make slight changes to my visualization script inputs that get passed to the module functions, such as different isolevels and plotting parameters and re-`include()` the visualization script to run it and produce my movies. Each time I do this, the amount of RAM Julia uses increases. Every single one of my processes is containerized into a function, including my main script which is inside a `main()`, so there can’t even be global variable usage. I am confused why there is this “memory leak”, per-se, as currently I have to just restart Julia once I’ve run the script a certain number of times when tweaking visualizations otherwise my computer will crash. Why is this necessary?
---
### Author:  [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)
#### Post date: [April 26, 2020, 6:38pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/2 "2020-04-26T18:38:50Z")
This will be very difficult to help you debug without code (preferably a mwe).
---
### Author:  [@duncanam](https://discourse.julialang.org/u/duncanam)
#### Post date: [April 26, 2020, 6:55pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/3 "2020-04-26T18:55:18Z")
I was hoping to avoid this as I’m worried this will be confusing, but I’ll see if I can distill it down to a minimal nonworking example as an overview:
**vis.jl**
```julia
using CSV, Makie, FFMPEG
module wave
export read3D, isosurface3D
function read3D(filepath,timeend)
for i=1:timeend
data[:,:,:,i] = Array(CSV.read(filepath*string(i)*".txt")
end # end for loop
return data
end # end read()
function isosurface3D(data,timeend)
for i=1:timeend
scene = volume(data[:,:,:,i],algorithm=:iso)
end # end for loop
# Save 3D image
Makie.save("/somepath/",scene)
# Make movie
FFMPEG.exe("options for movie")
end # end isosurface3D()
end # end module
```
**analysis.jl**
```julia
include("vis.jl")
using .wave
function main()
filepath = "/somepath/"
timeend = 10
data = read3D(filepath,timeend)
isosurface3D(data,timeend)
end
main()
```
I then run **analysis.jl** using:
`julia> include("analysis.jl")`
I think that’s the gist of it. I’m a bit puzzled as to why this would eat more memory each time.
---
### Author:  [@pixel27](https://discourse.julialang.org/u/pixel27)
#### Post date: [April 26, 2020, 6:56pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/4 "2020-04-26T18:56:41Z")
> **[GitHub - JuliaCI/Coverage.jl: Take Julia code coverage and memory allocation...](https://github.com/JuliaCI/Coverage.jl)**
>
> Take Julia code coverage and memory allocation results, do useful things with them - GitHub - JuliaCI/Coverage.jl: Take Julia code coverage and memory allocation results, do useful things with them
Check the Memory Allocation section.
---
### Author:  [@asinghvi17](https://discourse.julialang.org/u/asinghvi17)
#### Post date: [April 29, 2020, 3:27am UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/5 "2020-04-29T03:27:52Z")
The reason your script is taking so much memory is because you’re continually creating Scenes, which are quite expensive (and, since you’re using GLMakie, have to be stored on the GPU as well). Instead of creating and then saving individual scenes, you can use Makie’s inbuilt animation capabilities to achieve the same thing.
First, here’s an example of your code changed to use Observables (such that you only mutate a single Scene, and don’t create multiple Scenes):
```julia
function isosurface3D(data,timeend)
scene = volume(data[:,:,:,1],algorithm=:iso)
volumeplot = scene.plots[end] # the last plot plotted to the Scene.
for i=1:timeend
volumeplot[4][] = data[:, :, :, i] # update the data of the plot
Makie.save("scene_$i.png", scene) # save each frame separately
end # end for loop
# Save 3D image
Makie.save("final.png",scene)
# Make movie
FFMPEG.exe("options for movie")
end # end isosurface3D()
```
Makie actually uses FFMPEG internally for animations, and we’ve built in a bunch of optimizations throughout the pipeline to make recording fast. On the latest version of AbstractPlotting.jl (Makie is the metapackage, AbstractPlotting is the bulk of the plotting code):
```julia
function isosurface3D(data,timeend)
scene = volume(data[:,:,:,1],algorithm=:iso)
volumeplot = scene.plots[end] # the last plot plotted to the Scene.
record(scene, "movie.mp4", 1:timeend; framerate = 30) do i
volumeplot[4][] = data[:, :, :, i] # update the data of the plot
end # end recording loop
# Save 3D image
Makie.save("final.png",scene)
end # end isosurface3D()
```
and you’re done! If you want, you can also pass `sleep = false` as a keyword argument to record, to make it record as fast as possible. If you don’t pass it, the animation will be shown on the screen in real-time.
You should also take a look at [http://makie.juliaplots.org/dev/animation.html](http://makie.juliaplots.org/dev/animation.html) and [http://makie.juliaplots.org/dev/interaction.html](http://makie.juliaplots.org/dev/interaction.html), which have some more info about using Observables.
Let me know if you have any questions!
---
### Author:  [@duncanam](https://discourse.julialang.org/u/duncanam)
#### Post date: [April 30, 2020, 10:47pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/6 "2020-04-30T22:47:26Z")
This is very helpful, thank you. I had looked at Makie’s animation section, but to be truthful, I was a little overwhelmed with the `Node()` stuff hence me turning to stitching together frames manually. I looked through the second link you sent, but I’m afraid I’m still not fully understanding this `scene.plots[end]` section. In general, I guess I am internally thinking that `scene` is a variable holding the plot, so each time I call `volume()` and write to `scene` it is overwriting this, but in reality I suppose this is not the case. I’m assuming that the `scene` object is thus then updated (the point of Observables, as far as I can tell) with `plots`, but `[end]` seems to indicate that there are multiple.
---
### Author:  [@asinghvi17](https://discourse.julialang.org/u/asinghvi17)
#### Post date: [May 1, 2020, 2:03am UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/7 "2020-05-01T02:03:29Z")
It might be useful to think of the Scene as a holder for plots, much like Matplotlib’s figure. `scene.plots` is an array of the plots in the Scene, and `scene.plots[end]` is simply the latest plot to have been plotted to the Scene (in this case, the volume plot).
You can access the arguments of a Plot using linear indexing (`plot[i]`). Volumes actually have 4 arguments - an x range, y range, z range, and the 3D array of values. Since you only provided one in the call to `volume`, it was expanded into 4 arguments internally, so to update the value you must update the 4th argument.
---
### Author:  [@YohanSassi](https://discourse.julialang.org/u/YohanSassi)
#### Post date: [March 16, 2021, 12:53pm UTC](https://discourse.julialang.org/t/infinite-increasing-memory-usage-until-crash/38254/8 "2021-03-16T12:53:56Z")
Hi @asinghvi17 ,
I’m trying to follow your advice for creating a scene and then just update it iteratively to avoid memory usage to increase until crash. I’m working with video, I would like to add a grid on each frames and extract them iteratively. Here you will find the leak I firstly followed to avoid memory leak [Makie memory leak?](https://discourse.julialang.org/t/makie-memory-leak/52362) but it seems that it’s not sufficient when using it on large video… With your advice my problem is to be able to extract from the scene the information corresponding to the image and replace it by another one… Thanks in advance for your help !
This piece of code is my MWE:
```julia
using VideoIO, FileIO
using GLMakie,AbstractPlotting, Makie
function save_png(img, i, grid)
fn=string(i)*".png"
scene = AbstractPlotting.image(rotr90(img), show_axis = false);
AbstractPlotting.linesegments!(scene, grid, color = "white", linewidth = 2, show_axis = false);
AbstractPlotting.save(fn, scene; resolution = (720,450))
close(GLMakie.global_gl_screen())
nothing
end
function ext_frames(vid_name::String, grid) ## to Add Grid and extract frames
io = VideoIO.testvideo(vid_name) ## for testing purposes
f = VideoIO.openvideo(io); ## Open the video
img = read(f); ## Take the first frame
n = 1
save_png(img,n, grid) ## Add grid and save the frame
while !eof(f) ## While the video is not finished
next_img = read!(f, img); ## Take the next frame
n = n + 1
save_png(next_img,n, grid)
end
close(f)
end
grid = Tuple{Float32,Float32}[(203.18451, 391.91782), (203.18451, 34.85404), (359.62997, 391.91782), (359.62997, 34.85404), (516.07544, 391.91782), (516.07544, 34.85404), (46.739048, 124.11998), (672.52094, 124.11998), (46.739048, 213.38593), (672.52094, 213.38593), (46.739048, 302.6519), (672.52094, 302.6519)]
ext_frames("annie_oakley",grid)
```
So the idea would be to keep the first part because in my real code I’m doing specific stuff there, but in the while loop I would like to change a bit the save\_png function to create the scene with the image + the grid and then iteratively change the image under the grid and save it. Ideally I would like to couple this process with an encoder to recreate the video with the added grid.
**Important note** : I’m working with 30min videos at 30 fps so it’s a lot of frames to work with → all is about avoiding RAM usage and speed up the process of saving frames.