LFD HTTPS messenger used all memory and cpu

This forum is only for reproducible bugs with csf and lfd (i.e. not iptables problems, lack of understanding how to use a feature, etc). Posts must be accompanied with full technical details of the problem and how it can be recreated. Any posts not adhering to this, or not considered bugs, will be moved to the General Discussion (csf) forum.
Post Reply
Dejan
Junior Member
Posts: 13
Joined: 28 Apr 2011, 18:15

LFD HTTPS messenger used all memory and cpu

Post by Dejan »

Helo,

We just had a case that server crashed because of messenger memory usage (160+ GBs or memory)...

Here is the relevan part of the process list...

Code: Select all

Output from ps:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 root     11754  0.2  0.0 280872 49724 ?        Ss   00:00   2:04 lfd - sleeping
 csf      11772  0.0  3.5 4917012 4678416 ?     D    00:00   0:37  \_ lfd HTTPS messenger
 root     14216 25.9  3.5 4915512 4721056 ?     D    16:59   0:27  |   \_ lfd HTTPS messenger client
 root     14234 24.7  3.5 4915512 4721076 ?     D    16:59   0:25  |   \_ lfd HTTPS messenger client
 root     14284 30.2  3.5 4915648 4721108 ?     D    16:59   0:31  |   \_ lfd HTTPS messenger client
 root     14324 30.8  3.5 4915652 4721644 ?     D    17:00   0:30  |   \_ lfd HTTPS messenger client
 root     14375 25.5  3.5 4915652 4721160 ?     D    17:00   0:23  |   \_ lfd HTTPS messenger client
 root     14398 29.3  3.5 4915784 4708904 ?     D    17:00   0:26  |   \_ lfd HTTPS messenger client
 root     14408 25.5  3.5 4915788 4708924 ?     D    17:00   0:22  |   \_ lfd HTTPS messenger client
 root     14434 28.8  3.5 4915788 4711000 ?     D    17:00   0:24  |   \_ lfd HTTPS messenger client
 root     14473 27.4  3.5 4915920 4713080 ?     D    17:00   0:22  |   \_ lfd HTTPS messenger client
 root     14487 25.4  3.5 4915924 4675724 ?     D    17:00   0:20  |   \_ lfd HTTPS messenger client
 root     14497 25.9  3.5 4916056 4690088 ?     D    17:00   0:20  |   \_ lfd HTTPS messenger client
 root     14559 35.3  3.5 4916060 4721348 ?     D    17:00   0:26  |   \_ lfd HTTPS messenger client
 root     14580 23.5  3.5 4916060 4616416 ?     D    17:00   0:17  |   \_ lfd HTTPS messenger client
 root     14590 26.5  3.5 4916192 4636924 ?     D    17:00   0:18  |   \_ lfd HTTPS messenger client
 root     14614  9.4  3.3 4916196 4448564 ?     D    17:00   0:06  |   \_ lfd HTTPS messenger client
 root     14647 30.4  3.4 4916196 4569388 ?     D    17:00   0:19  |   \_ lfd HTTPS messenger client
 root     14724 24.0  3.4 4916328 4575568 ?     D    17:00   0:15  |   \_ lfd HTTPS messenger client
 root     14766 24.2  3.4 4916332 4598120 ?     D    17:00   0:14  |   \_ lfd HTTPS messenger client
 root     14772 33.4  3.4 4916464 4585868 ?     D    17:00   0:20  |   \_ lfd HTTPS messenger client
 root     14934 31.3  3.4 4916468 4579748 ?     D    17:00   0:16  |   \_ lfd HTTPS messenger client
 root     14982 12.1  3.3 4916468 4418020 ?     D    17:00   0:06  |   \_ lfd HTTPS messenger client
 root     15010  4.8  3.3 4916468 4407736 ?     D    17:00   0:02  |   \_ lfd HTTPS messenger client
 root     15075 30.0  3.4 4916600 4530652 ?     D    17:00   0:13  |   \_ lfd HTTPS messenger client
 root     15127 22.7  3.4 4916600 4510172 ?     D    17:00   0:09  |   \_ lfd HTTPS messenger client
 root     15183 31.0  3.4 4916600 4538844 ?     D    17:01   0:12  |   \_ lfd HTTPS messenger client
 root     15199 34.3  3.4 4916600 4522460 ?     D    17:01   0:11  |   \_ lfd HTTPS messenger client
 root     15246 30.5  3.4 4916604 4532724 ?     D    17:01   0:09  |   \_ lfd HTTPS messenger client
 root     15277 29.3  3.4 4916604 4493868 ?     D    17:01   0:08  |   \_ lfd HTTPS messenger client
 root     15325 32.9  3.4 4916736 4538928 ?     D    17:01   0:08  |   \_ lfd HTTPS messenger client
 root     15467 30.9  3.3 4916740 4477612 ?     D    17:01   0:07  |   \_ lfd HTTPS messenger client
 root     15555 11.7  3.3 4916876 4432492 ?     D    17:01   0:02  |   \_ lfd HTTPS messenger client
 root     15588 21.3  3.3 4916876 4440700 ?     D    17:01   0:02  |   \_ lfd HTTPS messenger client
 root     15719 31.6  3.3 4916876 4428456 ?     D    17:01   0:02  |   \_ lfd HTTPS messenger client
 root     15765 47.3  3.3 4917012 4415940 ?     D    17:01   0:02  |   \_ lfd HTTPS messenger client
 root     15778 22.4  3.3 4917012 4411832 ?     D    17:01   0:01  |   \_ lfd HTTPS messenger client
 csf      15902  2.5  3.3 4917012 4404328 ?     D    17:01   0:00  |   \_ lfd HTTPS messenger client
 csf      11773  0.0  0.0 219916 31288 ?        S    00:00   0:01  \_ lfd HTML messenger
 csf      11775  0.0  0.0 219916 31272 ?        S    00:00   0:00  \_ lfd TEXT messenger

Why every child processes use ~4,5 GB? Why every child process use ~25% CPU?

Why most of them are owned by root? They should all run as a separate user?

What can be the cause for such behavior?

This is CSF v10.14

Thanks.
Dejan
Junior Member
Posts: 13
Joined: 28 Apr 2011, 18:15

Re: LFD HTTPS messenger used all memory and cpu

Post by Dejan »

And one more think I would add...

MESSENGER_CHILDREN is set to "10" and here is 36 child processes...
ForumAdmin
Moderator
Posts: 1523
Joined: 01 Oct 2008, 09:24

Re: LFD HTTPS messenger used all memory and cpu

Post by ForumAdmin »

It might help to have some information:

1. What OS, build and kernel version is the server running?

2. What version of Apache is being used?

3. How many domains are being served?

4. Where is MESSENGER_HTTPS_CONF pointing to and if it's a glob, how many files does that equate to?

5. If using a control panel, which one and what version?

6. What associated messages were in /var/log/lfd.log at the time?

7. What is MESSENGER_HTTPS_IN set to?

8. If not the latest version, what version of csf?

9. What version of perl is the server running?
Use the path to perl from the first line in /etc/csf/csf.pl, e.g.:
"/usr/local/cpanel/3rdparty/bin/perl -v"

10. Have you listed the port in MESSENGER_HTTPS in TCP_IN? (you must not do this)

11. Lastly, have you been under a DOS or DDOS against websites?
Dejan
Junior Member
Posts: 13
Joined: 28 Apr 2011, 18:15

Re: LFD HTTPS messenger used all memory and cpu

Post by Dejan »

1. What OS, build and kernel version is the server running?

CloudLinux 7.3, kernel 3.10.0-614.10.2.lve1.4.55.el7.x86_64

2. What version of Apache is being used?

Server version: Apache/2.4.25 (cPanel)
Server built: Apr 14 2017 12:43:52

3. How many domains are being served?

~ 600

4. Where is MESSENGER_HTTPS_CONF pointing to and if it's a glob, how many files does that equate to?

MESSENGER_HTTPS_CONF = "/usr/local/apache/conf/httpd.conf"

5. If using a control panel, which one and what version?

Latest cPanel - Release

6. What associated messages were in /var/log/lfd.log at the time?

Nothing. Only single load message, but that is when server was already swapping, after this issue started...

lfd[14818]: *LOAD* 5 minute load average is 20.85, threshold is 10 - email sent

7. What is MESSENGER_HTTPS_IN set to?

MESSENGER_HTTPS_IN = "443,2083,2087,2096"

8. If not the latest version, what version of csf?

CSF v10.14
Post Reply