Greetings,
I’ve been battling an issue that I cannot quite put my thumb on. On my most recent install for a customer, they started complaining of their phones “locking up”. As I have looked into it, i have found that the PBX is sending a ridiculous amount of SIP notify packets to the endpoints, which causes their CPU’s to spike to 100% and become pretty much unresponsive. I have 50MB wireshark dumps rotating on eth0, and when it starts ramping up, it will fill a 50MB capture file about once a minute… approximately 70k SIP NOTIFY packets.
I have ruled out loop onsite, replaced all of their switches, phones, and firewall. I have a phone registered to it remotely over a site-to-site VPN and that phone also experiences this, so it doesnt appear to be caused by anything on the customer site. I have even setup another brand new image and reconfigured everything from scratch for them and the same thing happens. I may even try using an old distro ISO to see if the issue follows me there…
The PBX exists in our “cloud”, which is a hyperV failover cluster. None of the other images have this issue that i know of. I am currently firing up another image with the latest CentOS7.6 GA that was just released to see if it happens there as well… i know more about that later.
Doing a fwconsole restart, will temporarily stop it, but then it starts back up again soon after.
I am using Polycom 410’s, I have tried several different firmwares and it happens on all of them, so I dont believe it’s the phones that are causing it. I have other semi-recent installs for other customers that do not experience this issue, so I have to think that it is something to do with a recent module update and/or asterisk 13.22… but I have no specific proof… just a hunch.
I have updated to the edge track for Core and Framework to see if that makes a difference, so far no help there.
Here is my environment:
FPBX Distro
12.7.6-1904-1.sng7
FreePBX 14.0.12.10
Asterisk: 13.22
[root@localhost~]# fwconsole ma list
No repos specified, using: [standard,extended,commercial] from last GUI settings
+----------------------+------------+---------+------------+
| Module | Version | Status | License |
+----------------------+------------+---------+------------+
| accountcodepreserve | 13.0.2.2 | Enabled | GPLv2 |
| announcement | 13.0.7.7 | Enabled | GPLv3+ |
| arimanager | 13.0.5.2 | Enabled | GPLv3+ |
| asterisk-cli | 14.0.1 | Enabled | GPLv3+ |
| asteriskinfo | 13.0.7.1 | Enabled | GPLv3+ |
| backup | 14.0.10.3 | Enabled | GPLv3+ |
| blacklist | 14.0.2 | Enabled | GPLv3+ |
| builtin | | Enabled | |
| bulkhandler | 13.0.14.8 | Enabled | GPLv3+ |
| calendar | 14.0.2.16 | Enabled | GPLv3+ |
| callback | 13.0.5.4 | Enabled | GPLv3+ |
| callforward | 14.0.1.3 | Enabled | AGPLv3+ |
| callrecording | 14.0.14 | Enabled | AGPLv3+ |
| callwaiting | 14.0.1.1 | Enabled | GPLv3+ |
| cdr | 14.0.5.19 | Enabled | GPLv3+ |
| cel | 14.0.2.12 | Enabled | GPLv3+ |
| certman | 14.0.3.2 | Enabled | AGPLv3+ |
| cidlookup | 14.0.1.8 | Enabled | GPLv3+ |
| conferences | 13.0.23.15 | Enabled | GPLv3+ |
| configedit | 13.0.7.1 | Enabled | AGPLv3+ |
| contactmanager | 14.0.5.4 | Enabled | GPLv3+ |
| core | 14.0.28.9 | Enabled | GPLv3+ |
| customappsreg | 13.0.5.7 | Enabled | GPLv3+ |
| dahdiconfig | 14.0.1.4 | Enabled | GPLv3+ |
| dashboard | 14.0.6.2 | Enabled | AGPLv3+ |
| daynight | 14.0.1 | Enabled | GPLv3+ |
| dictate | 13.0.5 | Enabled | GPLv3+ |
| digiumaddoninstaller | 13.0.1.1 | Enabled | GPLv2 |
| directory | 13.0.19.12 | Enabled | GPLv3+ |
| disa | 13.0.6.12 | Enabled | AGPLv3+ |
| donotdisturb | 14.0.1.1 | Enabled | GPLv3+ |
| dundicheck | 2.11.0.3 | Enabled | GPLv3+ |
| endpoint | 14.0.2.188 | Enabled | Commercial |
| extensionroutes | 13.0.10.7 | Enabled | Commercial |
| extensionsettings | 13.0.4 | Enabled | GPLv3+ |
| fax | 14.0.2.7 | Enabled | GPLv3+ |
| featurecodeadmin | 13.0.6.4 | Enabled | GPLv3+ |
| findmefollow | 14.0.1.23 | Enabled | GPLv3+ |
| firewall | 13.0.57.1 | Enabled | AGPLv3+ |
| framework | 14.0.12.10 | Enabled | GPLv2+ |
| fw_langpacks | 14.0.1 | Enabled | GPLv3+ |
| hotelwakeup | 14.0.1.6 | Enabled | GPLv2 |
| iaxsettings | 14.0.1.4 | Enabled | AGPLv3 |
| infoservices | 13.0.1.4 | Enabled | GPLv2+ |
| ivr | 14.0.4 | Enabled | GPLv3+ |
| languages | 14.0.1.4 | Enabled | GPLv3+ |
| logfiles | 13.0.10.5 | Enabled | GPLv3+ |
| manager | 13.0.2.5 | Enabled | GPLv2+ |
| miscapps | 13.0.3.1 | Enabled | GPLv3+ |
| miscdests | 13.0.7 | Enabled | GPLv3+ |
| music | 13.0.22.7 | Enabled | GPLv3+ |
| outroutemsg | 14.0.1 | Enabled | GPLv3+ |
| paging | 14.0.12 | Enabled | GPLv3+ |
| parking | 13.0.19.11 | Enabled | GPLv3+ |
| pbdirectory | 2.11.0.6 | Enabled | GPLv3+ |
| phonebook | 13.0.6.4 | Enabled | GPLv3+ |
| phpinfo | 13.0.2 | Enabled | GPLv2+ |
| pinsets | 13.0.13 | Enabled | GPLv3+ |
| pm2 | 13.0.7.1 | Enabled | AGPLv3+ |
| presencestate | 14.0.1.7 | Enabled | GPLv3+ |
| printextensions | 13.0.3.2 | Enabled | GPLv3+ |
| queuemetrics | 2.11.0.3 | Enabled | GPLv3+ |
| queueprio | 13.0.6 | Enabled | GPLv3+ |
| queues | 14.0.2.25 | Enabled | GPLv2+ |
| recordings | 13.0.30.13 | Enabled | GPLv3+ |
| restapi | 13.0.21.2 | Enabled | AGPLv3 |
| ringgroups | 14.0.1.8 | Enabled | GPLv3+ |
| setcid | 13.0.6.3 | Enabled | GPLv3+ |
| sipsettings | 14.0.27.12 | Enabled | AGPLv3+ |
| soundlang | 14.0.7 | Enabled | GPLv3+ |
| speeddial | 2.11.0.4 | Enabled | GPLv3+ |
| superfecta | 14.0.18 | Enabled | GPLv2+ |
| sysadmin | 14.0.33 | Enabled | Commercial |
| timeconditions | 14.0.2.17 | Enabled | GPLv3+ |
| ucp | 14.0.3.3 | Enabled | AGPLv3+ |
| userman | 14.0.3.49 | Enabled | AGPLv3+ |
| vmblast | 13.0.11 | Enabled | GPLv3+ |
| voicemail | 14.0.6.5 | Enabled | GPLv3+ |
| weakpasswords | 13.0.2 | Enabled | GPLv3+ |
+----------------------+------------+---------+------------+
The phones have a pretty basic template. One Line Key for one SIP account, 2 Parking BLFs, and 4 BLF-XFER BLF’s for the 4 other extensions on the PBX (5 total PJSIP endpoints). Nothing crazy at all.
Here is a screenshot of a packet capture, I can provide the full dumpfile if needed. The capture filled with these NOTIFY packets, to each of the 5 registered endpoints.
All of the SIP Notify packets have the same Message Body, which leaves me to believe it has something to do with the voicemail module or MWI in asterisk…
Messages-Waiting: no\r\n
Voice-Message: 0/0 (0/0)\r\n
I would appreciate any additional insight as to why this is happening, I am running out of ideas here.
thanks in advance!
