High 5 minute load average alert

Post Reply
adeyjones
Junior Member
Posts: 12
Joined: 21 Dec 2020, 22:12

High 5 minute load average alert

Post by adeyjones »

I have a fairly new 'large' AWS instance which houses around 40 'not very busy' websites, yet I regularly get a High 5 minute load average alert email.

The that I have just received says:
Time: Thu Sep 2 12:35:36 2021 +0000
1 Min Load Avg: 27.97
5 Min Load Avg: 8.12
15 Min Load Avg: 3.15
Running/Total Processes: 68/395

I compare this to an 'x-large' AWS instance I have that houses 100+ websites, a few of which are global and very busy, yet I have never received one of these emails from that instance.

My question is firstly is there anything I should be looking at to find out whats causing the spike in load, perhaps it is a config issue? And secondly, in the vmstat.txt attachment in my email (which I assume is virtual memory), it says that cache is 3074724 (bytes?) compared to 'free' which is 103108. Should I be clearing a cache somewhere to free this up?

Thanks in advance for your help.
adeyjones
Junior Member
Posts: 12
Joined: 21 Dec 2020, 22:12

Re: High 5 minute load average alert

Post by adeyjones »

Keep getting these and wonder if someone could help me decipher the latest one, 2 of my servers went offline overnight Sunday/Monday so I received a lot of complaints Monday morning and would like to understand this stuff a bit better so I can monitor these closer.

Time: Mon Sep 13 13:00:13 2021 +0000
1 Min Load Avg: 38.00
5 Min Load Avg: 11.57
15 Min Load Avg: 4.63
Running/Total Processes: 24/334

ps.txt attachment contains:
Output from ps:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S 08:47 0:00 [kthreadd]
root 4 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S 08:47 0:02 \_ [ksoftirqd/0]
root 7 0.0 0.0 0 0 ? S 08:47 0:01 \_ [migration/0]
root 8 0.0 0.0 0 0 ? S 08:47 0:00 \_ [rcu_bh]
root 9 0.1 0.0 0 0 ? S 08:47 0:25 \_ [rcu_sched]
root 10 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [lru-add-drain]
root 11 0.0 0.0 0 0 ? S 08:47 0:00 \_ [watchdog/0]
root 12 0.0 0.0 0 0 ? S 08:47 0:00 \_ [watchdog/1]
root 13 0.0 0.0 0 0 ? S 08:47 0:01 \_ [migration/1]
root 14 0.0 0.0 0 0 ? S 08:47 0:02 \_ [ksoftirqd/1]
root 16 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kworker/1:0H]
root 18 0.0 0.0 0 0 ? S 08:47 0:00 \_ [kdevtmpfs]
root 19 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [netns]
root 20 0.0 0.0 0 0 ? S 08:47 0:00 \_ [khungtaskd]
root 21 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [writeback]
root 22 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kintegrityd]
root 23 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [bioset]
root 24 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [bioset]
root 25 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [bioset]
root 26 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kblockd]
root 27 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [md]
root 28 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [edac-poller]
root 29 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [watchdogd]
root 35 0.0 0.0 0 0 ? S 08:47 0:00 \_ [kswapd0]
root 36 0.0 0.0 0 0 ? SN 08:47 0:00 \_ [ksmd]
root 37 0.0 0.0 0 0 ? SN 08:47 0:01 \_ [khugepaged]
root 38 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [crypto]
root 46 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kthrotld]
root 48 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kmpath_rdacd]
root 49 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kaluad]
root 51 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kpsmoused]
root 53 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [ipv6_addrconf]
root 66 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [deferwq]
root 107 0.0 0.0 0 0 ? S 08:47 0:00 \_ [kauditd]
root 186 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [rpciod]
root 187 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xprtiod]
root 252 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [nvme-wq]
root 253 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [nvme-reset-wq]
root 254 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [nvme-delete-wq]
root 255 0.0 0.0 0 0 ? S 08:47 0:00 \_ [kworker/u4:2]
root 257 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [ena]
root 279 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [bioset]
root 280 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfsalloc]
root 281 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs_mru_cache]
root 282 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-buf/nvme0n1]
root 283 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-data/nvme0n]
root 284 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-conv/nvme0n]
root 285 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-cil/nvme0n1]
root 286 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-reclaim/nvm]
root 287 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-log/nvme0n1]
root 288 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [xfs-eofblocks/n]
root 289 0.0 0.0 0 0 ? S 08:47 0:05 \_ [xfsaild/nvme0n1]
root 290 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kworker/0:1H]
root 594 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [nfit]
root 606 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [kworker/1:1H]
root 684 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [loop0]
root 689 0.0 0.0 0 0 ? S 08:47 0:00 \_ [jbd2/loop0-8]
root 691 0.0 0.0 0 0 ? S< 08:47 0:00 \_ [ext4-rsv-conver]
root 18898 0.0 0.0 0 0 ? S 10:44 0:00 \_ [kworker/u4:0]
root 28799 0.0 0.0 0 0 ? S 12:15 0:00 \_ [kworker/0:2]
root 29783 0.0 0.0 0 0 ? S 12:24 0:00 \_ [kworker/1:0]
root 30876 0.0 0.0 0 0 ? S 12:35 0:00 \_ [kworker/0:0]
root 32491 0.0 0.0 0 0 ? S 12:50 0:00 \_ [kworker/1:1]
root 661 0.0 0.0 0 0 ? S 12:55 0:00 \_ [kworker/1:2]
root 1538 0.0 0.0 0 0 ? S 13:00 0:00 \_ [kworker/u4:1]
root 1543 0.0 0.0 0 0 ? S 13:00 0:00 \_ [kworker/0:1]
root 1 0.0 0.0 193992 6436 ? Ss 08:47 0:11 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root 401 0.0 0.0 39060 6876 ? Ss 08:47 0:02 /usr/lib/systemd/systemd-journald
root 434 0.0 0.0 48104 3260 ? Ss 08:47 0:00 /usr/lib/systemd/systemd-udevd
root 460 0.0 0.0 55532 856 ? S<sl 08:47 0:00 /sbin/auditd
root 510 0.0 0.0 52792 2704 ? Ss 08:47 0:00 /usr/sbin/smartd -n -q never
rpc 511 0.0 0.0 69256 1000 ? Ss 08:47 0:00 /sbin/rpcbind -w
nscd 515 0.0 0.0 935096 2288 ? Ssl 08:47 0:09 /usr/sbin/nscd
root 517 0.0 0.1 186436 10432 ? Ss 08:47 0:00 queueprocd - waiting up to 60s to process a task
polkitd 534 0.0 0.1 610532 12052 ? Ssl 08:47 0:00 /usr/lib/polkit-1/polkitd --no-debug
root 536 0.0 0.0 195204 1236 ? Ssl 08:47 0:00 /usr/sbin/gssproxy -D
mailnull 581 0.0 0.0 83516 6056 ? Ss 08:47 0:00 /usr/sbin/exim -ps -bd -q15m -oP /var/spool/exim/exim-daemon.pid
dbus 590 0.0 0.0 64612 2392 ? Ssl 08:47 0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 604 0.0 0.0 21540 1236 ? Ss 08:47 0:00 /usr/sbin/irqbalance --foreground
root 608 0.0 0.0 24472 1700 ? Ss 08:47 0:00 /usr/lib/systemd/systemd-logind
root 610 0.0 0.0 152776 1980 ? Ss 08:47 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/local/cpanel/bin/pureauth
chrony 637 0.0 0.0 94568 1320 ? S 08:47 0:00 /usr/sbin/chronyd
root 1120 0.0 0.0 101032 2880 ? Ss 08:48 0:09 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H server eth0
root 1180 0.0 0.2 572372 17228 ? Ssl 08:48 0:01 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
root 1181 0.0 0.0 15104 1812 ? Ss 08:48 0:00 /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
dovenull 1202 0.0 0.0 47096 4632 ? S 08:48 0:00 \_ dovecot/pop3-login
dovenull 1203 0.0 0.0 48176 5764 ? S 08:48 0:00 \_ dovecot/imap-login
dovecot 1204 0.0 0.0 10256 1280 ? S 08:48 0:00 \_ dovecot/anvil
root 1205 0.0 0.0 10388 1492 ? S 08:48 0:00 \_ dovecot/log
dovenull 1206 0.0 0.0 46956 4624 ? S 08:48 0:00 \_ dovecot/pop3-login
dovenull 1207 0.0 0.0 48032 5616 ? S 08:48 0:00 \_ dovecot/imap-login
root 1208 0.0 0.1 23596 11088 ? S 08:48 0:00 \_ dovecot/config
dovecot 1210 0.0 0.0 47648 3252 ? S 08:48 0:00 \_ dovecot/stats
surgery+ 28140 0.0 0.0 37828 3420 ? S 12:09 0:00 \_ dovecot/imap
dovecot 28146 0.0 0.0 10268 1472 ? S 12:09 0:00 \_ dovecot/imap-hibernate
surgery+ 767 0.0 0.0 29036 2640 ? S 12:55 0:00 \_ dovecot/quota-status -p postfix
dovecot 1127 0.0 0.0 70740 4356 ? S 12:58 0:00 \_ dovecot/auth
root 1183 0.0 0.0 153400 3152 ? Ss 08:48 0:00 pure-ftpd (SERVER)
root 1323 0.0 0.3 148328 27524 ? Ss 08:48 0:03 cpsrvd (SSL) - waiting for connections
root 1504 0.0 0.0 0 0 ? Z 12:59 0:00 \_ [webmaild - serv] <defunct>
root 1346 0.0 0.1 224592 10588 ? Ssl 08:48 0:01 /usr/sbin/rsyslogd -n
named 1349 0.0 0.1 663184 11412 ? Ssl 08:48 0:03 /usr/sbin/pdns_server --guardian=no --daemon=no --disable-syslog --log-timestamp=no --write-pid=no
root 1351 0.0 0.0 111452 4300 ? Ss 08:48 0:00 /usr/sbin/sshd -D
root 1365 0.0 0.0 124476 1648 ? Ss 08:48 0:01 /usr/sbin/crond -n
root 1366 0.0 0.0 24208 804 ? Ss 08:48 0:00 /usr/sbin/atd -f
root 1367 0.0 0.0 108292 804 ttyS0 Ss+ 08:48 0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220
root 1368 0.0 0.0 108292 784 tty1 Ss+ 08:48 0:00 /sbin/agetty --noclear tty1 linux
root 1411 0.0 0.1 354260 15076 ? Ss 08:48 0:01 php-fpm: master process (/opt/cpanel/ea-php73/root/etc/php-fpm.conf)
root 1443 0.0 1.5 247004 121856 ? Ss 08:48 0:09 /usr/local/cpanel/3rdparty/perl/532/bin/perl -T -w /usr/local/cpanel/3rdparty/bin/spamd --allowed-ips=127.0.0.1,::1 --max-children=5 --pidfile=/var/run/spamd.pid --listen=5 --listen=6
root 2330 0.1 1.7 272304 141128 ? S 08:49 0:23 \_ spamd child
root 2332 0.0 1.6 263008 131800 ? S 08:49 0:01 \_ spamd child
root 1472 0.0 0.3 454960 31212 ? Ss 08:48 0:00 php-fpm: master process (/opt/cpanel/ea-php70/root/etc/php-fpm.conf)
root 1522 0.0 0.3 175628 28380 ? Ss 08:48 0:05 lfd - sleeping
root 1559 0.0 0.3 175628 26924 ? S 13:00 0:00 \_ lfd - (child) checking load...
root 1560 0.0 0.0 153836 2064 ? R 13:00 0:00 \_ /bin/ps axuf
root 1551 0.0 0.0 183692 6652 ? Ss 08:48 0:01 cpdavd - accepting connections on: 2079, 2080, 2090, 2091, 2077, 2078 (dormant)
root 1555 0.0 0.0 167616 4240 ? Ss 08:48 0:00 dnsadmin - dormant mode
root 1557 0.0 0.0 176004 6136 ? Ss 08:48 0:01 cpgreylistd - processor
root 1558 0.0 0.2 103340 19532 ? Ss 08:48 0:02 tailwatchd
root 1564 0.0 0.1 185628 10708 ? Ss 08:48 0:00 cPhulkd - processor
root 27690 0.0 0.1 151544 10200 ? S 12:05 0:00 \_ cPhulkd - dbprocessor
root 1610 0.0 0.0 26460 2652 ? SN 08:48 0:00 cpanellogd - sleeping for logs
root 1546 0.0 0.0 26592 2208 ? RN 13:00 0:00 \_ cpanellogd - sleeping for logs
root 1621 0.0 0.1 237296 14560 ? Ss 08:48 0:12 /usr/sbin/httpd -k start
root 1625 0.0 0.0 12944 1392 ? S 08:48 0:05 \_ /usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=server.surgeryweb.org.uk --suffix=-bytes_log
root 1626 0.0 0.0 13048 1684 ? S 08:48 0:03 \_ /usr/local/cpanel/bin/splitlogs --dir=/etc/apache2/logs/domlogs --main=server.surgeryweb.org.uk --mainout=/etc/apache2/logs/access_log
root 1636 0.0 0.1 53208 11728 ? S 08:48 0:00 \_ /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
nobody 32480 0.0 0.3 255020 24664 ? S 12:50 0:00 \_ /usr/sbin/httpd -k start
nobody 363 0.0 0.3 255040 24068 ? S 12:53 0:00 \_ /usr/sbin/httpd -k start
nobody 407 0.0 0.3 255052 24400 ? S 12:53 0:00 \_ /usr/sbin/httpd -k start
nobody 414 0.0 0.3 255052 24604 ? S 12:53 0:00 \_ /usr/sbin/httpd -k start
nobody 493 0.0 0.3 255028 24528 ? S 12:54 0:00 \_ /usr/sbin/httpd -k start
nobody 506 0.0 0.3 254940 23888 ? S 12:54 0:00 \_ /usr/sbin/httpd -k start
nobody 551 0.0 0.3 255036 30604 ? S 12:54 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1510 3.8 0.7 257284 62976 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/n30203/index.php
nobody 696 0.0 0.3 254920 30772 ? S 12:55 0:00 \_ /usr/sbin/httpd -k start
nobody 705 0.0 0.3 255020 24452 ? S 12:55 0:00 \_ /usr/sbin/httpd -k start
nobody 1000 0.0 0.3 255028 24020 ? S 12:57 0:00 \_ /usr/sbin/httpd -k start
nobody 1001 0.0 0.3 255028 24028 ? S 12:57 0:00 \_ /usr/sbin/httpd -k start
nobody 1006 0.0 0.3 255060 24116 ? S 12:57 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1539 4.7 0.6 244868 50144 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/87305/index.php
nobody 1008 0.0 0.3 255000 23884 ? S 12:57 0:00 \_ /usr/sbin/httpd -k start
nobody 1038 0.0 0.3 254920 23952 ? S 12:57 0:00 \_ /usr/sbin/httpd -k start
nobody 1129 0.0 0.3 254940 23980 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1501 3.2 0.8 330312 69084 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/c83033/index.php
nobody 1134 0.0 0.3 254940 23984 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1135 0.0 0.2 254920 23700 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1549 6.0 0.3 221504 27156 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/n33004/index.php
nobody 1136 0.0 0.3 254940 23952 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1137 0.0 0.3 254920 24012 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1145 0.0 0.3 255040 24000 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1148 0.0 0.3 255056 24052 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1149 0.0 0.3 255056 23952 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1150 0.0 0.3 255044 23896 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
nobody 1151 0.0 0.3 254920 23972 ? S 12:58 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1553 9.0 0.3 219456 24856 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/m84004/index.php
nobody 1219 0.0 0.3 254920 23860 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1225 0.0 0.3 254872 23728 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1226 0.0 0.3 255040 23792 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1234 0.0 0.3 254920 23776 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1514 3.9 0.7 255172 60688 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/n85617/index.php
nobody 1240 0.0 0.3 254920 23748 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1536 4.7 0.6 246916 51952 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/n30203/index.php
nobody 1245 0.0 0.3 254940 23744 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1246 0.0 0.3 254920 23832 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1247 0.0 0.3 254920 23792 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1253 0.0 0.2 254920 23692 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1254 0.0 0.3 254872 23728 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1256 0.0 0.3 254920 23732 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1531 4.6 0.6 248964 54264 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/j82075/index.php
nobody 1257 0.0 0.3 254920 23728 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1258 0.0 0.2 254672 22360 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1270 0.0 0.3 254904 23772 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1281 0.0 0.2 254920 23668 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1511 3.9 0.8 259460 64832 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/l84055/index.php
nobody 1284 0.0 0.3 255020 23932 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1291 0.0 0.2 254920 23604 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1512 3.8 0.8 259460 65044 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/l84055/index.php
nobody 1292 0.0 0.2 254920 23624 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1525 4.2 0.6 248964 54236 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/f21524/index.php
nobody 1293 0.0 0.2 254796 23660 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1294 0.0 0.2 254920 23652 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1518 4.0 0.7 255172 60412 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/j82075/index.php
nobody 1295 0.0 0.2 254920 23676 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1530 4.5 0.6 248964 54264 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/k84001/index.php
nobody 1296 0.0 0.3 254920 23768 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1524 4.4 0.6 248964 54252 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/m84004/index.php
nobody 1297 0.0 0.3 254920 23800 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1322 0.0 0.2 254788 23624 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1547 6.3 0.4 227648 33276 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/87305/index.php
nobody 1332 0.0 0.2 254924 23712 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1337 0.0 0.3 254920 23740 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1550 5.0 0.3 219456 24852 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/m82043/index.php
nobody 1338 0.0 0.2 254804 22484 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1339 0.0 0.2 254920 23608 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1344 0.0 0.3 254920 23768 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1552 5.0 0.3 219456 24856 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/c83033/wp-cron.php
nobody 1347 0.0 0.2 254788 22700 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1350 0.0 0.2 254920 23608 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1353 0.0 0.2 254920 23592 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1354 0.0 0.3 254904 23780 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1355 0.0 0.2 254804 22480 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1383 0.0 0.2 254920 23504 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1384 0.0 0.2 254796 23264 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1390 0.0 0.0 0 0 ? Z 12:59 0:00 \_ [httpd] <defunct>
nobody 1397 0.0 0.2 254920 23656 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1398 0.0 0.2 254904 23684 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1403 0.0 0.2 254788 23544 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1404 0.0 0.2 254920 23504 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1405 0.0 0.2 255044 23596 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1406 0.0 0.2 254920 23504 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1413 0.0 0.2 254920 23492 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1416 0.0 0.2 254788 23428 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1417 0.0 0.2 254920 23676 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1422 0.0 0.2 254920 23596 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1423 0.0 0.2 254804 22480 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1424 0.0 0.2 254904 23684 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1425 0.0 0.2 254920 23596 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1428 0.0 0.2 254804 22476 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1429 0.0 0.2 254788 23432 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1430 0.0 0.2 254924 23600 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1431 0.0 0.3 254940 23752 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1506 3.8 0.8 261896 67420 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/87305/index.php
nobody 1432 0.0 0.2 254788 23548 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1433 0.0 0.3 254920 23720 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1434 0.0 0.2 254920 23596 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1441 0.0 0.2 254920 23616 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1462 0.0 0.2 254788 23552 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1465 0.0 0.3 254920 23776 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1466 0.0 0.3 254920 23764 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1478 0.0 0.2 254924 23440 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
swclien+ 1513 4.0 0.7 257220 62740 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/m82043/index.php
nobody 1489 0.0 0.2 254940 23452 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
nobody 1490 0.0 0.2 254788 23432 ? S 12:59 0:00 \_ /usr/sbin/httpd -k start
root 1708 0.0 0.0 44916 1056 ? S 08:48 0:00 stunnel /etc/stunnel/stunnel.conf
root 1709 0.0 0.0 44916 720 ? S 08:48 0:00 stunnel /etc/stunnel/stunnel.conf
root 1710 0.0 0.0 44916 720 ? S 08:48 0:00 stunnel /etc/stunnel/stunnel.conf
root 1711 0.0 0.0 44916 720 ? S 08:48 0:00 stunnel /etc/stunnel/stunnel.conf
root 1712 0.0 0.0 44916 720 ? S 08:48 0:00 stunnel /etc/stunnel/stunnel.conf
root 1713 0.0 0.0 176336 5176 ? Ss 08:48 0:00 stunnel /etc/stunnel/stunnel.conf
mysql 1841 4.1 3.9 1501996 312212 ? Sl 08:48 10:21 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

vmstat.txt contains:
Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
24 1 0 2449220 260 2292564 0 0 180 45 797 712 17 3 68 0 11

and netstat.txt contains:
Output from netstat:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:2086 0.0.0.0:* LISTEN 1323/cpsrvd (SSL) -
tcp 0 0 0.0.0.0:2087 0.0.0.0:* LISTEN 1323/cpsrvd (SSL) -
tcp 0 0 0.0.0.0:2090 0.0.0.0:* LISTEN 1551/cpdavd - accep
tcp 0 0 0.0.0.0:2091 0.0.0.0:* LISTEN 1551/cpdavd - accep
tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 581/exim
tcp 0 0 0.0.0.0:110 0.0.0.0:* LISTEN 1181/dovecot
tcp 0 0 0.0.0.0:2095 0.0.0.0:* LISTEN 1323/cpsrvd (SSL) -
tcp 0 0 127.0.0.1:783 0.0.0.0:* LISTEN 1443/perl
tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 1181/dovecot
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 511/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 363/httpd
tcp 0 0 172.31.12.172:80 143.159.153.97:58589 SYN_RECV -
tcp 0 0 172.31.12.172:80 92.13.15.253:52921 SYN_RECV -
tcp 0 0 0.0.0.0:2096 0.0.0.0:* LISTEN 1323/cpsrvd (SSL) -
tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN 581/exim
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 1349/pdns_server
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1183/pure-ftpd (SER
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1351/sshd
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 1349/pdns_server
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 581/exim
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 363/httpd
tcp 0 0 0.0.0.0:2525 0.0.0.0:* LISTEN 1713/stunnel
tcp 0 0 0.0.0.0:2077 0.0.0.0:* LISTEN 1551/cpdavd - accep
tcp 0 0 0.0.0.0:2078 0.0.0.0:* LISTEN 1551/cpdavd - accep
tcp 0 0 0.0.0.0:2079 0.0.0.0:* LISTEN 1551/cpdavd - accep
tcp 0 0 0.0.0.0:2080 0.0.0.0:* LISTEN 1551/cpdavd - accep
tcp 0 0 0.0.0.0:993 0.0.0.0:* LISTEN 1181/dovecot
tcp 0 0 0.0.0.0:2082 0.0.0.0:* LISTEN 1323/cpsrvd (SSL) -
tcp 0 0 127.0.0.1:579 0.0.0.0:* LISTEN 1564/cPhulkd - proc
tcp 0 0 0.0.0.0:2083 0.0.0.0:* LISTEN 1323/cpsrvd (SSL) -
tcp 0 0 0.0.0.0:995 0.0.0.0:* LISTEN 1181/dovecot
tcp 0 0 172.31.12.172:443 192.200.17.247:52711 TIME_WAIT -
tcp 0 0 172.31.12.172:80 143.159.153.97:49784 ESTABLISHED 1295/httpd
tcp 25 0 172.31.12.172:443 209.126.4.156:53614 CLOSE_WAIT 1292/httpd
tcp 0 2898 172.31.12.172:443 83.137.226.61:38229 ESTABLISHED -
tcp 0 0 172.31.12.172:443 86.151.137.208:58037 TIME_WAIT -
tcp 0 0 172.31.12.172:443 90.206.7.34:60992 TIME_WAIT -
tcp 0 0 172.31.12.172:53248 54.73.144.251:443 ESTABLISHED 1514/php-cgi
tcp 0 0 172.31.12.172:443 92.232.86.250:49406 ESTABLISHED 407/httpd
tcp 0 25 172.31.12.172:443 194.207.94.215:62245 FIN_WAIT1 -
tcp 25 0 172.31.12.172:443 54.73.144.251:53240 CLOSE_WAIT 1344/httpd
tcp 0 0 172.31.12.172:443 195.234.243.131:21434 ESTABLISHED 1297/httpd
tcp 0 0 172.31.12.172:443 207.46.13.168:33601 ESTABLISHED 1337/httpd
tcp 0 0 172.31.12.172:443 92.21.27.252:47984 ESTABLISHED 1000/httpd
tcp 0 0 172.31.12.172:443 195.234.243.131:21438 ESTABLISHED 1354/httpd
tcp 0 0 172.31.12.172:80 54.73.144.251:37268 ESTABLISHED 1413/httpd
tcp 0 0 172.31.12.172:443 213.205.192.223:50572 ESTABLISHED 1245/httpd
tcp 0 0 172.31.12.172:443 54.73.144.251:53248 ESTABLISHED 1284/httpd
tcp 0 0 172.31.12.172:443 92.13.15.253:52922 ESTABLISHED 1151/httpd
tcp 0 0 172.31.12.172:443 90.206.7.34:60999 TIME_WAIT -
tcp 0 0 172.31.12.172:443 195.234.243.131:21432 ESTABLISHED 1219/httpd
tcp 0 0 127.0.0.1:54626 127.0.0.1:2095 CLOSE_WAIT 1431/httpd
tcp 0 0 172.31.12.172:80 23.228.109.147:6658 TIME_WAIT -
tcp 0 0 172.31.12.172:443 176.25.154.191:52194 ESTABLISHED 1240/httpd
tcp 0 25 172.31.12.172:443 194.207.94.215:62244 FIN_WAIT1 -
tcp 0 0 172.31.12.172:80 143.159.153.97:57567 FIN_WAIT2 -
tcp 0 0 172.31.12.172:443 213.205.192.223:21902 ESTABLISHED 1253/httpd
tcp 0 0 172.31.12.172:443 90.206.7.34:61000 TIME_WAIT -
tcp 0 0 172.31.12.172:443 38.128.156.207:19447 TIME_WAIT -
tcp 0 0 172.31.12.172:443 90.243.44.14:47238 ESTABLISHED 493/httpd
tcp 25 0 172.31.12.172:443 86.157.91.23:56486 CLOSE_WAIT 1281/httpd
tcp 0 0 172.31.12.172:443 92.118.160.37:41908 ESTABLISHED 1424/httpd
tcp 0 0 172.31.12.172:37268 54.73.144.251:80 ESTABLISHED 1518/php-cgi
tcp 0 0 172.31.12.172:443 176.25.154.191:52196 ESTABLISHED 1270/httpd
tcp 0 262 172.31.12.172:443 83.137.226.61:63109 ESTABLISHED -
tcp 25 0 172.31.12.172:443 86.157.91.23:56487 CLOSE_WAIT 1291/httpd
tcp 0 0 172.31.12.172:443 195.234.243.131:21435 ESTABLISHED 1257/httpd
tcp 0 0 172.31.12.172:443 92.21.27.252:47986 ESTABLISHED 1001/httpd
tcp 1 0 172.31.12.172:443 86.129.243.57:47388 CLOSE_WAIT 1296/httpd
tcp 0 0 172.31.12.172:443 86.30.190.222:37392 ESTABLISHED 1347/httpd
tcp 0 0 127.0.0.1:49692 127.0.0.1:80 TIME_WAIT -
tcp 0 51 172.31.12.172:443 92.118.160.37:43404 ESTABLISHED -
tcp 0 0 172.31.12.172:443 185.58.166.46:1069 ESTABLISHED 1135/httpd
tcp 0 0 172.31.12.172:443 45.5.66.186:46465 ESTABLISHED 1397/httpd
tcp 0 0 172.31.12.172:443 195.234.243.131:21436 ESTABLISHED 1148/httpd
tcp 0 51166 172.31.12.172:443 195.234.243.131:16651 LAST_ACK -
tcp 1 0 172.31.12.172:80 89.240.142.221:51659 CLOSE_WAIT 1294/httpd
tcp 0 0 172.31.12.172:443 195.234.243.131:19202 ESTABLISHED 1247/httpd
tcp 0 0 172.31.12.172:443 104.128.20.232:2937 ESTABLISHED 1006/httpd
tcp 0 300 172.31.12.172:993 62.254.205.3:27489 ESTABLISHED 1207/dovecot/imap-l
tcp 0 0 127.0.0.1:50408 127.0.0.1:2082 CLOSE_WAIT 705/httpd
tcp 0 0 172.31.12.172:443 90.243.44.14:47236 ESTABLISHED 1008/httpd
tcp 0 46 172.31.12.172:993 62.254.205.3:5632 ESTABLISHED 1207/dovecot/imap-l
tcp 0 0 172.31.12.172:443 90.206.7.34:60998 TIME_WAIT -
tcp 1 0 127.0.0.1:579 127.0.0.1:40556 CLOSE_WAIT 27690/cPhulkd - dbp
tcp 0 5178 172.31.12.172:443 49.7.20.71:40814 LAST_ACK -
tcp 0 0 172.31.12.172:443 51.19.196.43:53865 ESTABLISHED 1258/httpd
tcp 0 0 172.31.12.172:80 89.240.142.221:51658 ESTABLISHED 1256/httpd
tcp 1 0 172.31.12.172:443 176.25.154.191:52184 CLOSE_WAIT 551/httpd
tcp 0 0 172.31.12.172:443 185.58.166.46:28078 ESTABLISHED 1398/httpd
tcp 0 0 127.0.0.1:2095 127.0.0.1:54632 TIME_WAIT -
tcp 0 0 127.0.0.1:2095 127.0.0.1:54626 FIN_WAIT2 -
tcp 0 0 172.31.12.172:53240 54.73.144.251:443 FIN_WAIT2 -
tcp 0 0 172.31.12.172:80 86.129.243.57:45654 FIN_WAIT2 -
tcp 0 0 172.31.12.172:443 209.126.4.156:56234 ESTABLISHED -
tcp 0 0 172.31.12.172:993 62.254.205.3:7921 ESTABLISHED 1203/dovecot/imap-l
tcp 0 0 172.31.12.172:443 92.232.86.250:49408 ESTABLISHED 1038/httpd
tcp 0 0 172.31.12.172:443 114.119.141.112:10410 ESTABLISHED 1234/httpd
tcp 0 0 172.31.12.172:443 104.128.20.51:48647 ESTABLISHED 1322/httpd
tcp6 0 0 :::3306 :::* LISTEN 1841/mysqld
tcp6 0 0 :::587 :::* LISTEN 581/exim
tcp6 0 0 :::110 :::* LISTEN 1181/dovecot
tcp6 0 0 ::1:783 :::* LISTEN 1443/perl
tcp6 0 0 :::143 :::* LISTEN 1181/dovecot
tcp6 0 0 :::111 :::* LISTEN 511/rpcbind
tcp6 0 0 :::80 :::* LISTEN 363/httpd
tcp6 0 0 :::465 :::* LISTEN 581/exim
tcp6 0 0 :::53 :::* LISTEN 1349/pdns_server
tcp6 0 0 :::21 :::* LISTEN 1183/pure-ftpd (SER
tcp6 0 0 :::22 :::* LISTEN 1351/sshd
tcp6 0 0 :::25 :::* LISTEN 581/exim
tcp6 0 0 :::443 :::* LISTEN 363/httpd
tcp6 0 0 :::993 :::* LISTEN 1181/dovecot
tcp6 0 0 :::995 :::* LISTEN 1181/dovecot
udp 0 0 0.0.0.0:53 0.0.0.0:* 1349/pdns_server
udp 0 0 0.0.0.0:68 0.0.0.0:* 1120/dhclient
udp 0 0 0.0.0.0:111 0.0.0.0:* 511/rpcbind
udp 0 0 127.0.0.1:323 0.0.0.0:* 637/chronyd
udp 0 0 0.0.0.0:678 0.0.0.0:* 511/rpcbind
udp 0 0 172.31.12.172:54754 5.9.124.53:24441 ESTABLISHED 2330/spamd child
udp 0 0 172.31.12.172:50693 172.31.0.2:53 ESTABLISHED 2330/spamd child
udp 0 0 172.31.12.172:34681 5.9.124.53:24441 ESTABLISHED 2332/spamd child
udp 0 0 172.31.12.172:16096 172.31.0.2:53 ESTABLISHED 2332/spamd child
udp 0 0 172.31.12.172:40902 5.9.124.53:24441 ESTABLISHED 1443/perl
udp6 0 0 :::53 :::* 1349/pdns_server
udp6 0 0 :::111 :::* 511/rpcbind
udp6 0 0 ::1:323 :::* 637/chronyd
udp6 0 0 :::678 :::* 511/rpcbind
Sergio
Junior Member
Posts: 1689
Joined: 12 Dec 2006, 14:56

Re: High 5 minute load average alert

Post by Sergio »

I think you should check why the account with the name "swclients" has so many different processes running using a lot of CPU.
Per example:
swclien+ 1510 3.8 0.7 257284 62976 ? R 12:59 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/n30203/index.php
swclien+ 1539 4.7 0.6 244868 50144 ? R 13:00 0:00 | \_ /opt/cpanel/ea-php74/root/usr/bin/php-cgi /home/swclients/public_html/87305/index.php

and there is your answer, a lot of process from that account consuming a lot of CPU.
adeyjones
Junior Member
Posts: 12
Joined: 21 Dec 2020, 22:12

Re: High 5 minute load average alert

Post by adeyjones »

Hi Sergio

Thanks for the reply. The account swclients contains 55 websites which are all clones of each other for 55 different organisations, but that is all that is on that server. I have another server set up exactly the same way with an account that contains another 55 of these exact same websites, but this serrver also contains around 15 other accounts with other websites for other customers, and that one has never had one of these high load warnings, in fact it performs excellently, so not sure why the server with less sites on is generating significantly more warnings.
adeyjones
Junior Member
Posts: 12
Joined: 21 Dec 2020, 22:12

Re: High 5 minute load average alert

Post by adeyjones »

Could it be anything to do with PHP 7.4? Does that use more resource? Because majority of the sites on the well performing server are still on 7.3
Sergio
Junior Member
Posts: 1689
Joined: 12 Dec 2006, 14:56

Re: High 5 minute load average alert

Post by Sergio »

Don't think has anything to do with PHP any version at all.
If I see one of my customers doing that I will first check what process is causing this.
In your case I see that there a lot of different files doing the same, a few examples:
/home/swclients/public_html/n30203/index.php
/home/swclients/public_html/87305/index.php
/home/swclients/public_html/c83033/index.php
because of the weird directories (for me) inside public_html I will suspect that your site is compromised.

In my case, as I have ImunifyAV+ I will ran an scan on that account to check it.
Or, check your site at https://sitecheck.sucuri.net
adeyjones
Junior Member
Posts: 12
Joined: 21 Dec 2020, 22:12

Re: High 5 minute load average alert

Post by adeyjones »

Hi Sergio

Don't worry about the directory names, they're actually the organisation ID's for the organisation using the website, I provide a template to around 200 organisations and the sites are spread across 4 servers. It is just this one particular server causing so much grief.
Sergio
Junior Member
Posts: 1689
Joined: 12 Dec 2006, 14:56

Re: High 5 minute load average alert

Post by Sergio »

Ok, I really don't know how your server is set, so for me that looked suspicious :)

One more thing is to check your other servers for the csf.pignore file, check in there if you have a line that includes:
/opt/cpanel/ea-php74/root/usr/bin/php-cgi or similar.
Then check if this server has the same line.

It could be that CSF has sent you emails telling about the processes consuming a lot of memory or resources and the emails could contain the line that should be added to csf.pignore in this server.
Post Reply