Plots/GR + png() results in "Too many open files"

General Usage
1

Hello!

I have a piece of Julia code where I generate (a lot of) scatter plots with the scatter function from Plots (with the GR backend), and save the plots to PNGs, using the png(filename) function (to later turn them into an animation using apngasm). The problem I’m facing is that after a while, my code crashes with

socket: Too many open files
GKS: can’t connect to GKS socket application

GKS: Open failed in routine OPEN_WS
GKS: GKS not in proper state. GKS must be either in the state WSOP or WSAC in routine ACTIVATE_WS
ERROR: LoadError: SystemError: opening file “/tmp/jl_RcjyRH.png”: No such file or directory
Stacktrace:
[1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
[2] #systemerror#48 at ./error.jl:167 [inlined]
[3] systemerror at ./error.jl:167 [inlined]
[4] open(::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing) at ./iostream.jl:284
[5] open at ./iostream.jl:273 [inlined]
[6] open(::Base.var"#296#297"{String}, ::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:323
[7] open at ./io.jl:323 [inlined]
[8] read at ./io.jl:410 [inlined]
[9] _show(::IOStream, ::MIME{Symbol(“image/png”)}, ::Plots.Plot{Plots.GRBackend}) at /home/erik/.julia/packages/Plots/hyS17/src/backends/gr.jl:1780
[10] show at /home/erik/.julia/packages/Plots/hyS17/src/output.jl:215 [inlined]
[11] png(::Plots.Plot{Plots.GRBackend}, ::String) at /home/erik/.julia/packages/Plots/hyS17/src/output.jl:7
[12] png(::String) at /home/erik/.julia/packages/Plots/hyS17/src/output.jl:10
.
.
.

I presume either the *.png files I’m writing, or some temporary files being used in the processes, are opened, but never closed. Is there something I could/should do to close those files once I’m done writing them?

2

Which GR version are you using?

using GR
version()
] st GR
3

I seem to be using version “0.52.0”.

Here’s a small test program to demonstrate the problem:

using Plots
using Printf

ENV["QT_QPA_PLATFORM"] = "offscreen"

gr()

for i = 1:2000
    scatter(rand(5), rand(5))
    fn = @sprintf("hepp-%4.4i.png", i)
    println(i)
    png(fn)
    sleep(0.1)
end

Running it results in output like

.
.
.
1024
1025
1026
1027
socket: Too many open files
GKS: can't connect to GKS socket application

GKS: Open failed in routine OPEN_WS
GKS: GKS not in proper state. GKS must be either in the state WSOP or WSAC in routine ACTIVATE_WS
ERROR: LoadError: SystemError: opening file "/tmp/jl_bgLbsZ.png": No such file or directory

Notes:

  • The exact number of png files I can generate before the crash varies a bit, but it’s usually a little bit more than 1000.
  • If I don’t put the sleep(0.1) there, I get a lot of
read: Connection reset by peer
send: Broken pipe

That, however, was not required in my “real” program. Possibly because it generates bigger plots, so it’s a bit slower (maybe png() doesn’t like to be called too often?)

4

I could not reproduce this behaviour on macOS. Which OS are you using?

5

Linux. Ubuntu 20.04 to be precise.

6

I can confirm this, on Ubuntu 20.05, clean installation of Julia v1.5.2, Plots and GR:

[...]
292
293
294
read: Connection reset by peer
send: Broken pipe
295
GKSserver: Failed to listen to port 8410
read: Connection reset by peer
send: Broken pipe
296
297
read: Connection reset by peer
send: Broken pipe
298
299
300
[...]
7

Same happens in a CentOS-7 container, so I don’t think it’s distribution-specific.

8

This is a different issue. Relates to the socket connection - the required port isn’t free’d in time.

1 Like
9

Ah, right, of course!

10

One important factor here is

ulimit -a | grep "open file"
open files                      (-n) 1024

If I do ulimit -n 2048 the problem is solved - sort of. I feel there is hardly any reason why so many files would need to be kept open all at the same time. Once it’s done writing a file, the file should be closed.

11

In the meantime I could reproduce the problem on Ubuntu 20.04. The point is, that due to the plugin setting (QT_QPA_PLATFORM) the GKS QtTerm socket is never closed.

If you want to produce the PNGs only, then it’s much easier to use the following script:

using Plots
using Printf

ENV["GKSwstype"] = "100"

gr()

for i = 1:2000
    scatter(rand(5), rand(5))
    fn = @sprintf("hepp-%4.4i.png", i)
    println(i)
    png(fn)
end
3 Likes
12

Aha! Yes, that works. And, as an added bonus, it’s a bit faster as well :slight_smile:

Thanks a lot!

13

when I used the using GR
WARNING: using GR.plot in module Main conflicts with an existing identifier.
what should I do ? I asked in a seperate question about my problem but none answered me:(

14

Both Plots and GR export commands like plot. So you can only use one of them: using Plots (with the GR backend) or using GR.

15

If you need to access the commands, do import Plots or import GR instead. If you import GR, then you’ll have to use GR.plot to explicitly refer to the plot method in the GR module.

16

Right, and that said, if it isn’t already fixed (reading here it seemed so), while you can use ulimit, there’s an outstanding PR:

With another just opened PR: “Set VERSION to 1.8.0-DEV, move 1.7 NEWS to HISTORY” I’m not sure if the above one made it in, since it seems very harmless and only potentially helpful, I would like to see it in, even if it would only make buggy software less of an issue. Possibly it will come in 1.7.x? I’m not sure were they draw the line at adding stuff in such a release.