I’m having a strange problem on one of my servers. Many of my servers are on Vultr, and I’ve get a template setup to easily deploy more servers with a base configuration. The problem I’m having is that one of my servers is significantly slower than the others. I’ve compared CPU usage, and it’s baseline seems to be about 3 times higher than the others. A reload took 38 seconds, while on another server it took 10. We’re having some call quality issues, and I’m not sure that this is the problem, but it’s certainly a variable I’d like to eliminate. The server in question has 1 core. And it is consistently above 1 often chilling in the 2-3 range. The problem is, I can’t figure out what’s using so much CPU. It seems like nothing is particularly taxing. Sometime PHP will pop up for a few seconds doing something processor intensive. From running htop, I’ve seen that fwconsole actions sometimes use a lot of cpu for a while. Here is a top screenshot of normal usage.
top - 10:58:34 up 21:56, 1 user, load average: 2.23, 1.58, 1.45
Tasks: 131 total, 3 running, 128 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.7 us, 1.0 sy, 0.0 ni, 96.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 st
KiB Mem : 1882748 total, 144436 free, 1016372 used, 721940 buff/cache
KiB Swap: 2097148 total, 1968636 free, 128512 used. 514660 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4946 asterisk 20 0 1729228 196300 12600 S 2.0 10.4 56:14.21 asterisk
1103 mongodb 20 0 481800 8800 2004 R 0.7 0.5 8:05.98 mongod
2605 asterisk 20 0 1085704 52136 6120 S 0.3 2.8 0:33.64 node /var/www/h
4243 asterisk 20 0 923508 31980 5552 S 0.3 1.7 1:53.18 PM2 v2.10.5: Go
4760 root 20 0 854216 13940 2272 S 0.3 0.7 3:31.97 fail2ban-server
15742 root 20 0 161988 2308 1580 R 0.3 0.1 0:00.14 top
1 root 20 0 76448 3176 1968 S 0.0 0.2 1:39.02 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:04.72 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 R 0.0 0.0 0:56.99 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.92 watchdog/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
15 root 20 0 0 0 0 S 0.0 0.0 0:00.53 khungtaskd
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.02 writeback
Here is top at one of the more intensive moments:
top - 11:04:03 up 22:01, 1 user, load average: 0.77, 1.29, 1.38
Tasks: 152 total, 11 running, 141 sleeping, 0 stopped, 0 zombie
%Cpu(s): 22.7 us, 19.1 sy, 0.0 ni, 56.7 id, 0.0 wa, 0.0 hi, 0.7 si, 0.7 st
KiB Mem : 1882748 total, 138200 free, 1042516 used, 702032 buff/cache
KiB Swap: 2097148 total, 1968380 free, 128768 used. 487920 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4946 asterisk 20 0 1729228 197004 12604 S 6.6 10.5 56:38.30 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
17825 asterisk 20 0 397316 18764 9592 R 5.6 1.0 0:00.17 php /var/lib/asterisk/bin/schedtc.php
17827 asterisk 20 0 397324 18676 9596 R 5.6 1.0 0:00.17 php /usr/sbin/fwconsole calendar --sync
17821 root 20 0 395296 16044 9236 S 3.6 0.9 0:00.11 /usr/bin/php /convergent/auto/down_monitor.php
17826 asterisk 20 0 394920 15948 9184 S 3.3 0.8 0:00.10 php /var/www/html/admin/modules/dashboard/scheduler.php
895 root 20 0 82140 6692 3356 S 2.3 0.4 6:32.72 /usr/sbin/openvpn --cd /etc/openvpn/ --config sysadmin_server1.conf
1 root 20 0 76448 3176 1968 S 1.0 0.2 1:39.50 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
462 root 20 0 40224 6024 5800 S 0.7 0.3 0:53.30 /usr/lib/systemd/systemd-journald
628 dbus 20 0 58388 1980 1484 S 0.7 0.1 1:01.83 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
905 root 20 0 281912 8788 5744 S 0.7 0.5 0:26.50 /usr/sbin/rsyslogd -n
1103 mongodb 20 0 481800 8800 2004 S 0.7 0.5 8:08.11 /usr/bin/mongod --quiet -f /etc/mongod.conf run
17828 root 20 0 115304 1660 1380 S 0.7 0.1 0:00.02 /bin/bash /usr/bin/check_mk_agent
293 root 0 -20 0 0 0 S 0.3 0.0 0:54.70 [kworker/0:1H]
607 root 20 0 26456 1596 1348 S 0.3 0.1 0:29.81 /usr/lib/systemd/systemd-logind
900 root 20 0 27164 864 744 S 0.3 0.0 0:02.28 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
902 redis 20 0 142952 5484 1252 S 0.3 0.3 2:28.33 /usr/bin/redis-server 127.0.0.1:6379
1313 mysql 20 0 1176312 214488 5832 S 0.3 11.4 14:50.95 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/ma+
2103 root 20 0 395228 108212 2800 S 0.3 5.7 3:52.67 /usr/local/fop2/fop2_server -d -a /var/log/fop2_audit.log
4096 root 20 0 400340 19776 10260 S 0.3 1.1 2:10.41 php /var/www/html/admin/modules/firewall/hooks/voipfirewalld
4243 asterisk 20 0 923508 31980 5552 S 0.3 1.7 1:53.66 PM2 v2.10.5: God Daemon (/home/asterisk/.pm2)
4760 root 20 0 854216 13940 2272 S 0.3 0.7 3:32.77 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x
7300 asterisk 20 0 1140712 110248 4796 S 0.3 5.9 2:05.67 letschat
17360 root 20 0 162016 2380 1660 R 0.3 0.1 0:00.60 top -c
17817 root 20 0 182356 2384 1568 S 0.3 0.1 0:00.01 /usr/sbin/CROND -n
17818 root 20 0 182356 2384 1568 S 0.3 0.1 0:00.01 /usr/sbin/CROND -n
17819 root 20 0 182356 2384 1568 S 0.3 0.1 0:00.01 /usr/sbin/CROND -n
Right after this, calendar -sync jumped up to like 30%, but only briefly. I don’t even I any calendars to sync to on this server.
Any ideas guys?