100% CPU Usage - Nagios - Perl Command

Another user on the forum had an issue with FreePBX being hacked but the command showed “w00t”. I’m still seeing 100% CPU usage even though I killed all processes belonging to the user nagios. I’m seeing the same issue but the command shows up as perl. Any thoughts? Thanks!

Please provide the output of:

ps aux | sort -k 3,3 | head -n 6

I had the same issue with CPU at 100% and the user nagios running 4 processes at 20% each. I looked at the password file to see if there is a user called nagios and find him/her? there. As this PBX is on the internet with a public IP and no other firewall in front, I have added iptables rules to restrict access. monitoring to see if that helps. I will do a fresh install of FreePBX to see if the user nagios is created during install or if a hacker has created it for me.

I did see this pop up even with iptables locked down as best I know how.
1817 nagios 20 0 39364 284 256 S 0.3 0.1 0:32.75 /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d
Looked up NRPE:
The NRPE addon is designed to allow you to execute Nagios plugins on remote Linux/Unix machines

Follow the instructions here: http://schmoozestatus.tumblr.com/. It’s an opsview-agent vulnerability and anyone who has an Internet facing system should block it off as it’s stupidly easy for anyone to gain access to your system and execute code.

This thread should probably be in the Distro Specific forum, as it is unlikely that you would have this problem if you don’t use it.

Thread moved.

Thanks so much for clearing that up… I’m looking forward to a day when the distro is locked down during install and there is a firewall GUI to allow only IPs we trust and ports we accept. (I saw one that may be coming mentioned in another post). In the mean-time, how do we lowly users get notified of issues we need to fix?

Thanks again.

That’s a little tricky as apparently there is nowhere to do that in the opensource FreePBX, it seems now to be under the domain of the sysadmin commercial module, which will require zend and won’t work everywhere even you don’t mind being no longer open source.

It used to be set in amportal.conf but if you

mysql -D asterisk -ppassword -e “select * from admin”

you will get an idea , so:-

mysql -D asterisk -ppassword -e “update admin set value=‘[email protected]’ where variable=‘email’”

might fix that for you.

(Perhaps a daily cron job of

amportal a ma showannounce|mail -s “FreePBX Updates” [email protected]

might suffice.)

There you go making more stuff up Dicko. You really need to stop just making things up to attack us on.

FreePBX modules updates have nothing to do with the Distro or Sysadmin module. You set your email address to be notified on updates to modules in module admin under Tango Shield at the top. Been that way for 2 years I think now. Use to be in General Settings module that we got rid of.

well there you go, i looked everywhere for a fiele to fill in. I should have looked at the pictures, sorry about that.

Seems that the easier way to see if there are important updates to the system that need to be applied is to click on the link at top right of the forum… (check for issues/outages) and to visit:
http://wiki.freepbx.org/display/FD/Updating+FreePBX+Official+Distro
and update the system (CLI or PuTTy or SSH)

I have been chastised quite rightly for thinking the only way to set that up in FreePBX per-se was with sysadmin.

I might suggest that for us mere mortals replacing the only anomalous instance in the whole of the FreePBX gui where an important variable is hidden behind an an image (undocumented as far as I have discovered) with a obvious text box on the module admin page might be more logical, go figure why they did it that way :slight_smile:

Hello,

We have 15 - 20 systems hosted in a data center that are all set up the same way. I have now seen two systems with these results from running ‘top’ in the server’s SSH session:

top - 09:54:00 up 33 days, 5:44, 1 user, load average: 3.23, 3.24, 3.23
Tasks: 106 total, 5 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 39.4%us, 60.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.2%st
Mem: 2040896k total, 1999432k used, 41464k free, 42992k buffers
Swap: 2097148k total, 94536k used, 2002612k free, 1644472k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21995 nagios 20 0 129m 872 440 R 89.4 0.0 11801:14 perl
21852 nagios 20 0 103m 456 320 S 17.0 0.0 391:44.77 sh
11762 nagios 20 0 103m 456 320 R 16.3 0.0 397:33.05 sh
3 root 20 0 0 0 0 R 0.3 0.0 1:45.31 ksoftirqd/0
10 root RT 0 0 0 0 S 0.3 0.0 1:03.11 migration/1
2538 asterisk 20 0 1813m 26m 4652 S 0.3 1.3 246:23.60 asterisk
1 root 20 0 19412 320 136 S 0.0 0.0 0:01.52 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 6:21.83 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root RT 0 0 0 0 S 0.0 0.0 0:58.53 migration/0
11 root 20 0 0 0 0 S 0.0 0.0 1:36.24 ksoftirqd/1
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns

I try to press ‘k’ and type the pid of the processes running as the user ‘nagios’ and kill the process normally, but nothing happens. I then try to type ‘killall perl’ and nothing is stopped. I have already typed ‘chkconfig opsview-agent off’ to prevent the process from running in the future, but how can I stop this process right NOW? This system is on a direct IP connection with iptables configured to whitelist every port except 5060 and 10000-20000 udp. We whitelist extensions at the extension level in Asterisk (allow=1.2.3.4/255.255.255.255).

Just to reiterate, what can be done to stop this process now without having to do a full restart?

UPDATE:

This was able to stop the processes now without having to restart:

skill -STOP -u nagios
skill -KILL -u nagios

Those processes then stopped and now I can have the system restart after hours.

You need to upgrade your PBX which will patch the issue.

Tony,

Thanks! When you say you need to upgrade your PBX, which are you referring to? Upgrade the Asterisk binaries, upgrade the modules, run a yum update -y or all three?

Jon

Upgrade your FreePBX DIstro using the Distro upgrade scripts.