100% CPU usage with asterisk

Continuing the discussion from Seeking help on 100% asterisk thread on FreePBX:

Turned out I have removed the G723 729 codec but the system is still now using 100% cpu after 2 days of running.

You may want to post the output of top. Please use the code tags so it is readable.

If you are just going by the CPU meter on FreePBX status you need to dig deeper.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2006 asterisk  20   0 1810m  56m  15m S 100.8  3.0 164:35.58 asterisk
 6876 root      20   0 98280 3932 2980 S  0.3  0.2   0:04.30 sshd
 6896 root      20   0 15028 1316 1000 R  0.3  0.1   0:14.57 top

After tracing the process, the following information is obtained.
And the box is set up inside a ESXi hypervisor if it does matter.

Thanks a lot for your help

Thread 1 (process 2185):
0  0x00007f284b69f767 in bind () from /lib64/libc.so.6
1  0x000000000052d350 in ast_bind ()
2  0x00007f27e02a5f9f in ?? () from /usr/lib64/asterisk/modules/res_rtp_asterisk.so
3  0x0000000000554e64 in ast_rtp_instance_new ()
4  0x00007f27f7115da2 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
5  0x00007f27f715b724 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
6  0x00007f27f717e359 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
7  0x00007f27f718b207 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
8  0x00007f27f718bda7 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
9  0x00007f27f718b98c in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
10 0x0000000000500cd3 in ast_io_wait ()
11 0x00007f27f718d96d in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
12 0x000000000059da0a in ?? ()
13 0x00007f284a0d59d1 in start_thread () from /lib64/libpthread.so.0
14 0x00007f284b69eb6d in clone () from /lib64/libc.so.6

How many SIP channels are open asterisk -rx ‘sip show channels’

Is your machine have SIP open to the world?

I have quite a number of guest channel established.
I have no idea of where they from.

Since I was using SIP over the internet, my SIP port is open to the world. And rely on fail2ban for basic protection.

    Peer             User/ANR         Call ID          Format           Hold     Las                                    t Message    Expiry     Peer
62.75.207.126    287              1310541868       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    309              3304013697       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    404              3290771888       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    348              2791364027       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    391              3816120939       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    347              1677947199       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    328              3235717283       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    288              823296523        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    366              429792599        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    400              316782731        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    306              2232237812       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    302              2547798169       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    353              3179090622       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    380              1365197744       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    369              3011805493       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    336              3316448257       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    399              2883457018       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    350              3069842786       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    382              853792100        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    392              1399683265       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    300              2468791927       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    411              2813038794       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    303              576418681        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    312              3141368927       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    338              1718331605       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    351              2219804349       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    395              4233589968       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    321              3060653634       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    317              4022653640       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    285              1643222037       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    325              3285131374       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    290              3946864064       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    335              3442006732       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    412              121988854        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    386              2509350788       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    376              1648446546       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    403              2796129429       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    334              892808382        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    364              1051177867       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    406              1893240002       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    367              2217142126       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    377              3806353518       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    361              2298178031       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    304              1936362900       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    307              290689503        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    355              4091542805       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    372              3628688074       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    311              3200139109       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    356              3184864913       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    397              2865028749       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    294              348857314        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    370              2150864579       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    388              756613265        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    383              415299013        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    352              2661323184       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    320              3955893024       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    408              2272659420       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    308              892905256        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    322              1289737368       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    291              3396098134       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    331              1375080097       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    332              186959883        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    344              388574510        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    315              1047103201       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    413              3151335807       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    374              4241786132       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    393              4208477193       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    365              3645095145       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    280              2034875496       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    384              3573871605       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    341              2476038903       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    327              3323773478       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    337              3751056278       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    373              3039522181       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    314              2823545744       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    357              2427096253       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    296              400212541        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    349              3378595270       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    345              2432440104       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    305              208591747        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    394              3963281329       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    410              2741230571       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    346              3773938718       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    379              1018087686       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    289              3352141968       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    318              254649575        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    354              1320652976       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    329              968028970        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    368              1836129787       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    409              189300866        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    360              6965234          (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    279              3848962084       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    342              3188067917       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    324              2322753107       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    387              3320158442       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    396              2814328345       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    298              2439103273       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    390              4198810652       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    284              852028177        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    301              3932862890       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    381              1223675630       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    402              1806947379       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    286              2500178993       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    283              3800505406       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    293              3053870339       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    316              2439496464       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    362              276680201        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    323              3453335830       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    359              3292226909       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    310              3600840648       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    297              3477013638       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    281              1026021888       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    340              3756960743       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    405              2205127023       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    414              874679140        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    363              1771981690       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    330              4209114125       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    295              2405001236       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    299              1168032886       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    358              771196822        (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    371              3595721507       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    343              2244516615       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    375              4231406038       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    385              71507148         (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    378              1549506699       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    333              3148530423       (nothing)        No       Rx:                                     INVITE                 <guest>
62.75.207.126    401              319451645        (nothing)        No       Rx:                                     INVITE                 <guest>

I also enabled Intrusion Detection function in the panel.
But it doesnt capture any banned IP.

from bash:-

whois 62.75.207.126

A German cloud hosting service. You need to get your installation of fail2ban working.

For immediate relief add 62.75.128.0/17 to your firewall’s banned network list.

I remembered that my fail2ban used to work after my initial set up.
And I just adjusted the ban time only.
and it is now just stopping. Thanks for the information.

Probably because it is not working. You are using the FreePBX distro? Did you change anything in the configs that might have effected your jails?

You may want to puruse the fail2ban documentation so you understand what the gears are doing behind the pretty GUI.

http://www.fail2ban.org/wiki/index.php/Main_Page

That fail2ban thing is more difficult than I expected.

[[email protected] action.d]# fail2ban-client status
Status
|- Number of jail:      5
`- Jail list:           apache-badbots, apache-tcpwrapper, ssh-iptables, asterisk-iptables, vsftpd-iptables

I have these jail list, does it seems right?

For a little light hearted relief, back in 1955 IBM reputedly had a sign up in their visitor center that said:-

ACHTUNG!
ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN! ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN MIT SPITZENSPARKEN.
IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS.
ZO RELAXEN UND WATSCHEN DER BLINKENLICHTEN.

And I have now manually injected the ban IP.

[[email protected] action.d]# fail2ban-client set  asterisk-iptables banip 62.75.128.0/17
62.75.128.0/17
[[email protected] action.d]# fail2ban-client status asterisk-iptables
Status for the jail: asterisk-iptables
|- filter
|  |- File list:        /var/log/asterisk/fail2ban
|  |- Currently failed: 0
|  `- Total failed:     3
`- action
   |- Currently banned: 1
   |  `- IP list:       62.75.128.0/17
   `- Total banned:     1

that would totally depend on your " apache-badbots, apache-tcpwrapper, ssh-iptables, asterisk-iptables, vsftpd-iptables" jails and there included regexes and what log files are they following , (defined in /etc/fail2ban/jail.conf)

You got that guy I guesws, but probably only temporary for your bantime until you get the fail2ban process working

I guest this is where I should start looking.
The service is started properly and I receive a lot of email of the status change regarding these services where I reboot my server.

Thanks.

Problem is the fail2ban is looking at the right log file, but the file is not exist.

I have no idea on why the file:-

/var/log/asterisk/messages

is now changed to

/var/log/asterisk/fail2ban

I have not set anything to change that setting, where can I change it back please?

-rw-rw-r--. 1 asterisk asterisk  1850084 Jun 27 09:35 freepbx.log
-rw-rw----  1 asterisk asterisk   406307 Jun 27 09:55 full
-rw-rw-r--. 1 asterisk asterisk 88998772 Jun 27 10:20 freepbx_dbug
-rw-rw----  1 asterisk asterisk   111968 Jun 27 10:20 fail2ban

You can rely on a prepackaged version and hope for the best or perhaps go to

http://www.fail2ban.org/wiki/index.php/Downloads

and get the latest and greatest, which if you follow the recipes there work fine for Asterisk,ssh and your webserver, webmin and pretty well anything else you might have added,( well, probably not the latest and greatest but at least the best one for redhat based distros)

If you go the “roll your own” route, you will have a much better understanding of how it all works. check your work with fail2ban-regex.

I will just try to point the fail2ban to the correct path of asterisk log.
Just wait and see…
I guess the /var/log/asterisk/fail2ban is just the default log file generated from freepbx for this purpose, right?

I can’t help you there, I don’t use RPM’s.