Running JuMP using KNITRO: killed

I am running a bunch optimization problems in a row with different parameters using JuMP and KNITRO and after about 5-7 hours the process simply says Killed and is sent back to the terminal with no additional information. Might this be a small memory leak? Should I somehow clear the RAM after each run?

I also encountered similar problem with some long running script. The memory usage gradually stack up and finally get killed. I have not got a MWE yet.

I am Not using JuMP though, so this could be a potential Julia memory leak bug.

There is also some memory leak issues of distributed computation in Julia repo, but I think I am not using them, not sure about other dependent packages. I do used @async though.

1 Like

The Killed message means out of memory. The kernel triggers the out of memory killer. You can see the logs at /var/log/messages or equivalent

1 Like

@bbrunaud thank you for the reply! There are many different files here, which ones should I look at:

febbo@febbo-HP-Z220-SFF-Workstation:/var/log$ ls
alternatives.log        alternatives.log.7.gz  apport.log.5.gz  auth.log.2.gz  dist-upgrade    dpkg.log.5.gz   gpu-manager.log  lightdm            syslog.6.gz          Xorg.0.log.old
alternatives.log.1      alternatives.log.8.gz  apport.log.6.gz  auth.log.3.gz  dmesg           dpkg.log.6.gz   installer        speech-dispatcher  syslog.7.gz          Xorg.1.log
alternatives.log.10.gz  alternatives.log.9.gz  apport.log.7.gz  auth.log.4.gz  dpkg.log        dpkg.log.7.gz   kern.log         syslog             teamviewer12         Xorg.1.log.old
alternatives.log.2.gz   apport.log             apt              boot.log       dpkg.log.1      dpkg.log.8.gz   kern.log.1       syslog.1           unattended-upgrades  Xorg.2.log
alternatives.log.3.gz   apport.log.1           aptitude         bootstrap.log  dpkg.log.10.gz  dpkg.log.9.gz   kern.log.2.gz    syslog.2.gz        upstart              Xorg.2.log.old
alternatives.log.4.gz   apport.log.2.gz        aptitude.1.gz    btmp           dpkg.log.2.gz   faillog         kern.log.3.gz    syslog.3.gz        wtmp                 Xorg.3.log
alternatives.log.5.gz   apport.log.3.gz        auth.log         btmp.1         dpkg.log.3.gz   fontconfig.log  kern.log.4.gz    syslog.4.gz        wtmp.1
alternatives.log.6.gz   apport.log.4.gz        auth.log.1       cups           dpkg.log.4.gz   fsck            lastlog          syslog.5.gz        Xorg.0.log

Also if I do:

dmesg -T| grep -E -i -B5 'killed process'

And look at the lines just before it is killed I get:

febbo@febbo-HP-Z220-SFF-Workstation:~/Documents//results/pp$ dmesg -T| grep -E -i -B5 'killed process'
[Tue Feb 20 04:24:24 2018] [ 6212]  1000  6212   353344      577      83       5      966             0 TVGuiSlave.64
[Tue Feb 20 04:24:24 2018] [ 6213]  1000  6213    33381      115      38       2     1824             0 TVGuiDelegate
[Tue Feb 20 04:24:24 2018] [ 6275]  1000  6275  8016419  3941396   15269      33  3601973             0 julia
[Tue Feb 20 04:24:24 2018] [ 6778]  1000  6778   137757      280     120       3      940             0 unity-panel-ser
[Tue Feb 20 04:24:24 2018] Out of memory: Kill process 6275 (julia) score 933 or sacrifice child
[Tue Feb 20 04:24:24 2018] Killed process 6275 (julia) total-vm:32065676kB, anon-rss:15765584kB, file-rss:0kB
--
[Wed Feb 21 04:55:33 2018] [ 6213]  1000  6213    33381       60      38       2     1928             0 TVGuiDelegate
[Wed Feb 21 04:55:33 2018] [19292]  1000 19292   165148       42     140       4     2339             0 gnome-terminal-
[Wed Feb 21 04:55:33 2018] [19299]  1000 19299     5845        1      18       3      670             0 bash
[Wed Feb 21 04:55:33 2018] [19456]  1000 19456  7863062  3953755   15203      33  3547061             0 julia
[Wed Feb 21 04:55:33 2018] Out of memory: Kill process 19456 (julia) score 928 or sacrifice child
[Wed Feb 21 04:55:33 2018] Killed process 19456 (julia) total-vm:31452248kB, anon-rss:15815020kB, file-rss:0kB
--
[Thu Feb 22 03:09:35 2018] [ 5770]  1000  5770   184534        1     154       4      666             0 gvfsd-smb-brows
[Thu Feb 22 03:09:35 2018] [ 5779]  1000  5779    90429        0      45       3      252             0 gvfsd-dnssd
[Thu Feb 22 03:09:35 2018] [ 5865]  1000  5865  7852110  3946446   15138      33  3571519             0 julia
[Thu Feb 22 03:09:35 2018] [ 7250]  1000  7250   137749      259     121       4     1473             0 unity-panel-ser
[Thu Feb 22 03:09:35 2018] Out of memory: Kill process 5865 (julia) score 930 or sacrifice child
[Thu Feb 22 03:09:35 2018] Killed process 5865 (julia) total-vm:31408440kB, anon-rss:15784840kB, file-rss:944kB
--
[Fri Feb 23 02:15:37 2018] [ 5779]  1000  5779    90429       25      45       3      211             0 gvfsd-dnssd
[Fri Feb 23 02:15:37 2018] [31248]  1000 31248   178707        0     198       4    14099             0 update-manager
[Fri Feb 23 02:15:37 2018] [ 2457]  1000  2457  7795624  3945875   15051      33  3504405             0 julia
[Fri Feb 23 02:15:37 2018] [ 6061]  1000  6061   137759      273     124       3      947             0 unity-panel-ser
[Fri Feb 23 02:15:37 2018] Out of memory: Kill process 2457 (julia) score 922 or sacrifice child
[Fri Feb 23 02:15:37 2018] Killed process 2457 (julia) total-vm:31182496kB, anon-rss:15783500kB, file-rss:0kB
--
[Sun Feb 25 01:38:21 2018] [31248]  1000 31248   178707        0     198       4    14098             0 update-manager
[Sun Feb 25 01:38:21 2018] [31641]  1000 31641   158830    11162     243       2    88624             0 TeamViewer_Desk
[Sun Feb 25 01:38:21 2018] [31842]  1000 31842  7898497  3921800   14943      33  3464435             0 julia
[Sun Feb 25 01:38:21 2018] [ 4229]  1000  4229   137758      257     119       4      950             0 unity-panel-ser
[Sun Feb 25 01:38:21 2018] Out of memory: Kill process 31842 (julia) score 914 or sacrifice child
[Sun Feb 25 01:38:21 2018] Killed process 31842 (julia) total-vm:31593988kB, anon-rss:15686432kB, file-rss:768kB
--
[Mon Feb 26 09:14:39 2018] [ 5779]  1000  5779    90429        0      45       3      269             0 gvfsd-dnssd
[Mon Feb 26 09:14:39 2018] [31248]  1000 31248   178707       68     198       4    13986             0 update-manager
[Mon Feb 26 09:14:39 2018] [14615]  1000 14615  7775460  3937516   15002      32  3464151             0 julia
[Mon Feb 26 09:14:39 2018] [17523]  1000 17523   137742      258     126       3      958             0 unity-panel-ser
[Mon Feb 26 09:14:39 2018] Out of memory: Kill process 14615 (julia) score 916 or sacrifice child
[Mon Feb 26 09:14:39 2018] Killed process 14615 (julia) total-vm:31101840kB, anon-rss:15750064kB, file-rss:0kB
--
[Wed Feb 28 18:26:52 2018] [ 6945]  1000  6945    51965        5      35       3      277             0 oosplash
[Wed Feb 28 18:26:52 2018] [ 6963]  1000  6963   449680     5575     321       5     7391             0 soffice.bin
[Wed Feb 28 18:26:52 2018] [16733]  1000 16733  7751156  3931379   14923      33  3511692             0 julia
[Wed Feb 28 18:26:52 2018] [16745]  1000 16745   137769      255     123       4      964             0 unity-panel-ser
[Wed Feb 28 18:26:52 2018] Out of memory: Kill process 16733 (julia) score 921 or sacrifice child
[Wed Feb 28 18:26:52 2018] Killed process 16733 (julia) total-vm:31004624kB, anon-rss:15725416kB, file-rss:100kB
febbo@febbo-HP-Z220-SFF-Workstation:~/Documents/workspace/PhD/papers/RTPP/results/pp$ 

[Wed Feb 28 18:26:52 2018] [16733]  1000 16733  7751156  3931379   14923      33  3511692             0 julia
[Wed Feb 28 18:26:52 2018] [16745]  1000 16745   137769      255     123       4      964             0 unity-panel-ser
[Wed Feb 28 18:26:52 2018] Out of memory: Kill process 16733 (julia) score 921 or sacrifice child
[Wed Feb 28 18:26:52 2018] Killed process 16733 (julia) total-vm:31004624kB, anon-rss:15725416kB, file-rss:100kB

Is there a way that I can clear the cache in between runs? It works fine if I restart it, but if it has been running for a bit then I need to restart. I have tried:

 clear!(:n)

where n is the object that holds all of the data. My feeling is that KNITRO and IPOPT need to be cleared or reset. Is there a way to do this? I guess that I could open up a new julia after each run…