I’m study why my modules requires a lot of memory > 500MB.
So I decided to study the growth of memory of my modules.
I discovered that only one module using LibPQ let Julia use 213,1 MB. If I compare only Julia REPL with 84,7 MB, I can conclude that LibPQ use 128,4 MB, only with using command! Why this happens?
Julia REPL, I can understand that there is all infrastructure of “Virtual Machine” like Java, but why LibPQ use more memory than Julia “base”?
But it’s not exclusive of LibPQ! When I load using MySQL my memory grow to 250,2 MB of usage (165,5 MB only with MySQL)!
Note: I’m only using the module. I never call any function of module. Only the using command do grow of memory usage.
Is this in the session which precompiled MySQL.jl and dependencies? If so, the memory usage is, I guess, a side-effect of the compilation itself, and, maybe (hopefully) it’s reclaimable by the OS?
Another thing to check is whether you can reproduce this with a C program, without using Julia.
After you can reliably reproduce this, try using the heap snapshot functionality to investigate further.
It isn’t the same session. After add MySQL, I did using MySQL for compile first time. So I closed the REPL, then opened two or three times collecting the memory usage.
At this moment I’m in a computer that I almost installed MySQL and LibPQ. I had same behaviour.
Note: the last version os MySQL is throwing a warning while compiling and recompile all tries. But LibPQ is working well, although a lot of memory.
It’s true @nsajko , I don’t now why, but a new system added old version 1.3.1 (I think). After update to MySQL v1.4.6 the module load only about 20 MB.
So the problem is some modules that I have at my systems. LibPQ for sample, still same the problem leading 212,4 MB after using LibPQ (about 128 MB, only at loading the module). At this case my version is LibPQ v1.18.0.
It’s so weird, when a updated MySQL with comand ] update MySQL upgraded but LibPQ not! When I did add LibPQ@1.18.0, MySQL downgraded. When I run ] update again, LibPQ downgraded and MySQL upgraded.
I’ll try this at virtual enviroment and see the behavior.
At a nutshell the problem are modules, at this case LibPQ is bad optimizated.
(@v1.10) pkg> activate env-a
Activating new project at `C:\Users\ronne\env-a`
(env-a) pkg> add MySQL LibPQ
(env-a) pkg> status
Status `C:\Users\ronne\env-a\Project.toml`
⌃ [194296ae] LibPQ v1.17.1
[39abe10b] MySQL v1.4.6
Case 2:
(env-a) pkg> activate env-b
Activating new project at `C:\Users\ronne\env-b`
(env-b) pkg> add LibPQ
(env-b) pkg> status
Status `C:\Users\ronne\env-b\Project.toml`
[194296ae] LibPQ v1.18.0
Case 3:
(env-b) pkg> activate env-c
Activating new project at `C:\Users\ronne\env-c`
(env-c) pkg> add MySQL
(env-c) pkg> status
Status `C:\Users\ronne\env-c\Project.toml`
[39abe10b] MySQL v1.4.6
So, certainly there is some kind of messy because the last version of these two packages MySQL and LibPQ are imcompactibles. Why when I installed the two packages, LibPQ is getting version 1.17.1 although there is version 1.18.0?
This way, there really is a very big messy, @abraemer . A messy not only at my enviroment, but at this two packages.
Sorry, I didn’t mean to offend you. A weird upgrading/downgrading of packages is just a common consequence of large environments (possibly containing obsolete packages).
I had a brief look and found:
julia> Pkg.activate(;temp=true)
(jl_zwHUIh) pkg> add LibPQ MySQL
# ...
(jl_zwHUIh) pkg> st
Status `/tmp/jl_zwHUIh/Project.toml`
⌃ [194296ae] LibPQ v1.17.1
[39abe10b] MySQL v1.4.6
(jl_zwHUIh) pkg> status --outdated -m
Status `/tmp/jl_zwHUIh/Manifest.toml`
⌅ [fa961155] CEnum v0.4.2 (<v0.5.0): LibPQ
⌃ [194296ae] LibPQ v1.17.1 (<v1.18.0)
⌅ [08be9ffa] LibPQ_jll v14.3.0+1 (<v16.0.0+0): LibPQ
⌅ [aabc7e14] MariaDB_Connector_C_jll v3.1.12+0 (<v3.3.9+0): MySQL
⌅ [458c3c95] OpenSSL_jll v1.1.23+0 (<v3.0.15+0): LibPQ_jll, MariaDB_Connector_C_jll
(jl_zwHUIh) pkg> why OpenSSL_jll
LibPQ → LibPQ_jll → OpenSSL_jll
MySQL → MariaDB_Connector_C_jll → OpenSSL_jll
which tells me that the problem is likely OpenSSL_jll.
Trying to add the latest versions of both gives more info:
(jl_zwHUIh) pkg> add MySQL@1.4.6 LibPQ@1.18.0
Resolving package versions...
ERROR: Unsatisfiable requirements detected for package MariaDB_Connector_C_jll [aabc7e14]:
MariaDB_Connector_C_jll [aabc7e14] log:
├─possible versions are: 3.1.6-3.3.9 or uninstalled
├─restricted by compatibility requirements with MySQL [39abe10b] to versions: 3.1.12
│ └─MySQL [39abe10b] log:
│ ├─possible versions are: 0.6.0-1.4.6 or uninstalled
│ └─restricted to versions 1.4.6 by an explicit requirement, leaving only versions: 1.4.6
└─restricted by compatibility requirements with OpenSSL_jll [458c3c95] to versions: 3.3.9 or uninstalled — no versions left
└─OpenSSL_jll [458c3c95] log:
├─possible versions are: 1.1.1-3.0.15 or uninstalled
└─restricted by compatibility requirements with LibPQ_jll [08be9ffa] to versions: 3.0.8-3.0.15
└─LibPQ_jll [08be9ffa] log:
├─possible versions are: 12.3.0-16.0.0 or uninstalled
└─restricted by compatibility requirements with LibPQ [194296ae] to versions: 16.0.0
└─LibPQ [194296ae] log:
├─possible versions are: 0.4.0-1.18.0 or uninstalled
└─restricted to versions 1.18.0 by an explicit requirement, leaving only versions: 1.18.0
