Memory leak issue on CentOS 7 / perl 5.16
Concerned version
Version: %2.0.6
Platform: Docker/CentOS 7/Nginx/llng-fastcgi-server
Summary
LemonLDAP crashed during the night, it was consuming 1000% of its reserved memory (320MB) which is pretty much the whole host's available memory. By looking at its metrics it seems like its memory consumption was progressively increasing for the past 10 days.
The only way to free its memory was to restart the service. Unfortunately, since its a container, I have no debug informations to analyse what was consuming this much memory... But regarding the differences between our environments, I'm quite sure it's perl-fcgi
processes fault, because each time LLNG is requested, the memory amount of one of the perf-fcgi
processes increase and memory is never released.
Note: it might be related to #1909 (closed) patch to keep LDAP connections persistent
Logs
During the crash we've had various error messages, but I think it's related to the lack of memory and host's processors load increase
[2019/10/29 20:10:48] ERROR: Session cannot be tied: TIMEOUT
[2019/10/29 20:10:50] ERROR: NO token created
...
[2019/10/29 20:26:00] ERROR: Lemonldap::NG::Handler::PSGI::Main: Unable to load configuration: YAML fails to read file: YAML Error: Error in require YAML::Loader - TIMEOUT
Code: Error in require YAML::Loader - TIMEOUT
at /usr/share/perl5/vendor_perl/YAML.pm line 83.
Error: No configuration found in local cache
...
[2019/10/29 20:36:15] ERROR: Session cannot be tied: TIMEOUT, at /usr/share/perl5/vendor_perl/Cache/CacheUtils.pm line 77.