LFD create a zombie defunct process

This forum is only for reproducible bugs with csf and lfd (i.e. not iptables problems, lack of understanding how to use a feature, etc). Posts must be accompanied with full technical details of the problem and how it can be recreated. Any posts not adhering to this, or not considered bugs, will be moved to the General Discussion (csf) forum.
12 posts Page 1 of 2
MPaterss
Junior Member
Posts: 4
Joined: 04 Aug 2016, 17:55


Hello,

I've got CSF and LFD csf v9.11 running on my VPS server without problems until a couple of weeks now.
Every couple of hours or sometimes a day LFD create a zombie defunct process on my Directadmin vps server.
When i restart LFD the zombie is gone a couple of hours but then its back again. Is this a bug in LFD?
Is there a way to debug LFD or do you have any idea where to look and to fix the problem?

Thanks and regards
Marcel
ForumAdmin
Moderator
Posts: 1433
Joined: 01 Oct 2008, 09:24


We really would need a bit more info:

1. What type of virtual server?
2. kernel version?
3. OS and version?
4. Amount of memory?
5. Most importantly of all: What was the process name of the zombie process?
MPaterss
Junior Member
Posts: 4
Joined: 04 Aug 2016, 17:55


Hello ForumAdmin,

Thanks for your quick replay.

My server specs are:

Apache 2.4.23
OS = Centos release 6.8 Final
Kernel = 2.6.32-642.3.1.el6.x86_64 on an x86_64
Disc = 150GB SSD
2 cores = 4096 MB

Processor Speed (MHz) 2199.998
Total Memory 3924400 kB
Free Memory 175020 kB
Total Swap Memory 524284 kB
Free Swap Memory 115340 kB

DirectAdmin 1.50.1
Exim 4.84
MySQL 5.6.29
Named 9.8.2rc1 Running
sshd
dovecot 2.2.25
pure-ftpd 1.0.42
Php 5.6.24

When I give this command: [root@server2 ~]# ps aux |grep "defunct"

I get

root 10481 0.0 0.0 103376 860 pts/0 S+ 13:30 0:00 grep defunct
root 13976 0.0 0.0 0 0 ? Z 03:15 0:00 [lfd] <defunct>

The pidfile number (13976) is the latest pidfile used by LFD

Marcel
ForumAdmin
Moderator
Posts: 1433
Joined: 01 Oct 2008, 09:24


Thank you for all the additional information. That is very odd. What are the last few lines of the log that include the pid of the zombie process?
MPaterss
Junior Member
Posts: 4
Joined: 04 Aug 2016, 17:55


Hello Admin,
You ask me:
What are the last few lines of the log that include the pid of the zombie process?
but that's a problem because i can't find the log that include the pid of the zombie process.
Can you tell me where to find that logfile?
Marcel
ForumAdmin
Moderator
Posts: 1433
Joined: 01 Oct 2008, 09:24


/var/log/lfd.log
MPaterss
Junior Member
Posts: 4
Joined: 04 Aug 2016, 17:55


Hello Admin,
Below are the last lines of the log file /var/log/lfd.log. LFD is running with pid file nr: 15214 until now ( Aug 6 12:21) but a new created LFD zombie this morning use pid file nr: 31576
31576 root 20 0 0 0 0 Z 0.0 0.0 0:00.00 [lfd] <defunct>
I can,t find pidnr:31576 is this file.
Aug 6 00:00:01 server2 lfd[24894]: TERM
Aug 6 00:00:01 server2 lfd[24894]: daemon stopped
Aug 6 00:00:02 server2 lfd[15214]: daemon started on server2.m-hosting.com - csf v9.11 (DirectAdmin)
Aug 6 00:00:02 server2 lfd[15214]: Restricting syslog/rsyslog socket acccess to group [mysyslog]...
Aug 6 00:00:02 server2 lfd[15214]: CSF Tracking...
Aug 6 00:00:02 server2 lfd[15214]: IPv6 Enabled...
Aug 6 00:00:02 server2 lfd[15214]: LOAD Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Country Code Filters...
Aug 6 00:00:02 server2 lfd[15214]: Country Code Lookups...
Aug 6 00:00:02 server2 lfd[15214]: System Integrity Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Exploit Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Directory Watching...
Aug 6 00:00:02 server2 lfd[15214]: Email Relay Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Temp to Perm Block Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Process Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Account Tracking...
Aug 6 00:00:02 server2 lfd[15214]: SSH Tracking...
Aug 6 00:00:02 server2 lfd[15214]: SU Tracking...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/messages...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/secure...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/customlog...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/www/html/squirrelmail/data/squirrelmail_access_log...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/exim/mainlog...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/maillog...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/www/html/phpMyAdmin/log/auth.log...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/www/html/roundcube/logs/errors...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/directadmin/login.log...
Aug 6 00:00:02 server2 lfd[15214]: Watching /var/log/httpd/error_log...
Aug 6 08:56:09 server2 lfd[2526]: Directory Watching terminated after 40 seconds
Aug 6 08:56:09 server2 lfd[2526]: LF_DIRWATCH taking 40 seconds, temporarily throttled to run every 900 seconds

Marcel
TommyTheKid
Junior Member
Posts: 5
Joined: 23 Jul 2013, 11:11


We also have an issue where several of our systems are getting defunct lfd processes.


1. Type of Virtual Server: VMWare
2. Kernel Version: 2.6.32-642.3.1.el6.x86_64
3. OS/Version: CentOS 6.8
4. Memory: 16GB
5. Process Name: [lfd]

Details:
Code: Select all
# ps -ef | grep lfd
root      6678     1  0 00:07 ?        00:00:17 lfd - sleeping
root      8237  6678  0 06:12 ?        00:00:00 [lfd] <defunct>
root     19125  6678  0 11:15 ?        00:00:00 [lfd] <defunct>
root     20610  6678  0 11:55 ?        00:00:00 [lfd] <defunct>

6678 (parent process):
Code: Select all
Aug 10 00:07:02 HOSTNAME lfd[6678]: Watching /var/log/maillog...
Aug 10 00:07:02 HOSTNAME lfd[6678]: Watching /var/log/local6.log...
Aug 10 00:07:02 HOSTNAME lfd[6678]: Watching /var/log/messages...
Aug 10 00:07:02 HOSTNAME lfd[6678]: Watching /var/log/secure...
Aug 10 00:07:02 HOSTNAME lfd[6678]: Watching /var/log/local7.log...
Aug 10 14:31:43 HOSTNAME lfd[6678]: TERM
Aug 10 14:31:43 HOSTNAME lfd[6678]: daemon stopped
NOTE: 14:31 is when I restarted, which caused the defunct processes to go away.

None of the child processes (8237, 19125, 20610) have any entries in lfd.log.

This is affecting several (but not all) of our systems. If you would like me to do any debugging on them next time, let me know what to do. These are "production" servers, so we may not have all the tools available.

Tommy
ForumAdmin
Moderator
Posts: 1433
Joined: 01 Oct 2008, 09:24


The best way to help would be if you could set DEBUG to "3" at the end of /etc/csf/csf.conf and then restart csf and then lfd.

This will produced a very detailed /var/log/lfd.log and should indicate which subroutine is being run by the child processes that turn into zombies.

The downside of doing this is that it will produce a very large /var/log/lfd.log so it does need to have an eye kept on it.

If you are able to do this and then check the log for any zombie PIDs it should hopefully narrow down the location of the problem.
tomrowel
Junior Member
Posts: 1
Joined: 11 Sep 2016, 11:20


We got the same problem. The [lfd] <defunct> is returning everytime.
We are monitoring zombie process with Nagios.

Latest in /var/log/lfd.log
Code: Select all
Sep 11 12:13:11 huib lfd[32680]: *User Processing* PID:3330 Kill:0 User:nagios Time:53758 EXE:/usr/local/nagios/bin/nagios CMD:/usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
Sep 11 12:13:11 huib lfd[32680]: *User Processing* PID:3331 Kill:0 User:nagios Time:53758 EXE:/usr/local/nagios/bin/nagios CMD:/usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
12 posts Page 1 of 2