I downgraded to Asterisk 22 certified version and I did not have to reboot any PBXs today. Swap space is barely being used as well compared to before.
We have probably 95% of our 100+ desk phones on newer Yealink models as well. We donāt see the memory utilization creeping up. Iād say after 5+ days of uptime the memory utilization for asterisk is still sitting at a steady 1.4% of the 16 GB total.
We are on FreePBX Distro v12.7.8-2408-1.sng12, FreePBX Framework 17.0.33.2, Asterisk 22.11.0. I know this doesnāt help, but a frame of reference. We donāt heavily rely on BLF or a lot of subscriptions in that regard.
@gregarican It still helps because it is a data point that helps further isolate things.
When I ran some quick checks just now, here is what mine came back with.
root@DC-PBX03:~# grep -E 'RssAnon|VmSwap' /proc/$(pidof asterisk)/status
RssAnon: 177004 kB
VmSwap: 0 kB
root@DC-PBX03:~# asterisk -rx "pjsip show subscriptions inbound" | tail -1
17 active subscriptions
root@DC-PBX03:~# asterisk -rx "core show taskprocessors" | grep -c '^[a-z]'
348
root@DC-PBX03:~#
iām not sure if this is helpful, and admittedly i donāt know a lot of the architecture changes in asterisk 22, but in case this is useful information: we have a small handful of asterisk/freepbx instances. 3 of them are currently on freepbx 17.0.33.2. of those 3, 1 is using asterisk 21.12.3 and 2 of them are using 22.11.0 ā these 2 have experienced oom-kills since updating to these versions.
what i found a little strange was that the 22.11.0 instances appear to have a very widely expanding list of entries in ācore show taskprocessorsā, especially for example lines such as these:
stasis/p:extension_state:extension/1503@ext-local-000000cb 311 0 1 450 500 0 40
stasis/p:extension_state:extension/1503@ext-local-000000d4 310 0 1 450 500 0 28
stasis/p:extension_state:extension/1503@ext-local-000000d5 309 0 1 450 500 0 9
stasis/p:extension_state:extension/1503@ext-local-000000d7 308 0 1 450 500 0 48
stasis/p:extension_state:extension/1503@ext-local-00000108 306 0 1 450 500 0 36
stasis/p:extension_state:extension/1503@ext-local-00000110 305 0 1 450 500 0 12
stasis/p:extension_state:extension/1503@ext-local-00000120 304 0 1 450 500 0 7
stasis/p:extension_state:extension/1503@ext-local-00000146 303 0 1 450 500 0 8
often hundreds for the same extension, which appeared very different from what counterparts might be on our asterisk 21 instance.
on our asterisk 21 instance, we have 39 pjsip contacts and after 4 days (of very little use), the line count on āsudo asterisk -rx ācore show taskprocessorsā | wc -lā is 268. on an asterisk 22.11.0 instance which was last rebooted at the same time, has 19 pjsip contacts and actually no call use since then, the line count was 8700+. after a fwconsole restart, it dropped to 365. another 22.11.0 instance was rebooted 3 hours ago, has 50 pjsip endpoints and has had 13 calls in those 3 hours, its line count for core show taskprocessors is 3743.
i hope this is helpful information. thanks!
@tcl Are you also on Yealink?
@jcolp almost exclusively Yealink for our hardware phones, yes. likely a few others in small numbers, and some software phone use at times as well.
Understood! For anyone who might be wondering why Iāve asked - knowing the phone helps us reproduce the subscription usage with the same content type. Different phones ask for subscription notifications in different formats so knowing the primary one people experiencing the issue are using helps us better replicate it.
Yes, around 100% Yealink
For us, I just took a peek. We have roughly 50 Yealink T54W (mostly F/W 96.86.0.75) and 50 Yealink T29G (F/W 46.83.0.120). Like I said, we donāt have a lot of BLF type of subscriptions in place. Just the standard MWI for the desk phones and basic stuff. Pretty steadily sitting at ~350 active subscriptions the past 5 or so days, when we did switch to Asterisk 22.11.0.
@jcolp Another data point that may help isolate this. On our box the leak is limited to a single extension, the trigger looks like asterisk/asterisk#2179 from the 20.21.0 / 22.11.0 / 23.5.0 release thread, and the box crashed twice when many phones resubscribed at once.
How we got here: This is a FreePBX 16 system on SNG7. On 10-25 I applied the PHP2Shell update with yum upgrade (PHP 7.4.16 ā 7.4.33), updated the FreePBX modules, and rebooted. The yum upgrade also took Asterisk from 20.17.0 to 20.21.0, the same release as 22.11.0.
Setup: Small VM, 2 vCPU and 1.8 GB RAM. About 25 Yealink SIP-T57W phones on firmware 96.86.0.70, mostly connecting over TCP through the sysadmin OpenVPN. Every phone has BLF keys for most extensions and the park slots.
The leak, about 3 hours after an Asterisk restart:
$ asterisk -rx "pjsip list subscriptions inbound" | grep -oE '[^ ]+/(dialog|message-summary|presence)' | sort | uniq -c | sort -rn | head -5
952 200/dialog
24 303/dialog
24 208/dialog
24 204/dialog
24 203/dialog
$ asterisk -rx "core show taskprocessors" | grep -c 'extension_state:extension/200@'
952
$ asterisk -rx "core show taskprocessors" | grep -c 'extension_state:'
1546
$ asterisk -rx "pjsip list subscriptions inbound" | tail -1
1544 active subscriptions
Every other extension and park hint has about 24 subscriptions, one per phone. Extension 200 has 952, because each phone holds 41 to 45 separate subscriptions to it, each with a different Call-ID and all showing a live expiry. The extension_state taskprocessors match those subscriptions 1:1. Total subscriptions went from 1466 to 1535 in about 15 minutes, nearly all of it on 200. An hour later 200 was at 1,141, about 50 per phone.
Whatās different about 200: it was the only watched hint with no presence value. database show CustomPresence had an entry for every extension except 200.
200@ext-local : PJSIP/200&Custom:DND200,CustomPresence:200 State:Idle Presence: Watchers 1315
203@ext-local : PJSIP/203&Custom:DND203,CustomPresence:203 State:Idle Presence:available Watchers 25
Test: I ran presencestate change CustomPresence:200 AVAILABLE. Watchers rose briefly to 1,343 while the phones finished their retries, then held flat at 1,343 for 10+ minutes. Before the change it had been growing by about 6 a minute. It looks like the missing initial NOTIFY makes the Yealinks resubscribe over and over, and Asterisk never cleans up the old subscriptions. The leaked subscriptions also havenāt expired since the fix, so they seem to stay until a restart.
Two other 20.21.0 boxes of ours have had no duplicate subscriptions in 37 hours. The only hints there with no presence value are extensions that no phone has a BLF key for.
@tcl it may be worth checking database show CustomPresence for the extensions with all the duplicate extension_state entries.
Memory: Asterisk RssAnon+VmSwap was 344 MB about 3 hours after the restart and 395 MB an hour later. Our other 20.21.0 b sit at 90 to 150 MB.
Crashes: Twice in about 19 hours on 20.21.0. Each came right after many phones resubscribed at once, while extension 200 had the duplicate subscriptions.
- Crash 1: Several phonesā TCP connections reset within 7 seconds of each other (they re-registered from new source ports). Asserts began about 20 seconds later, and Asterisk crashed 20 seconds after that.
- Crash 2: Assert bursts began 2 to 3 minutes after each BLF key push from Endpoint Manager, when about 25 Yealinks re-provisioned and resubscribed. The crash came 4 minutes after one of those pushes.
ERROR[...] pjproject: ../src/pj/lock.c .Assert failed: cnt > 0
ERROR[...] pjproject: ../src/pj/os_core_unix .Assert failed: mutex
The same box logged zero asserts in a week on 20.17.0, including a similar burst of TCP reconnects that caused no problems. Our two other 20.21.0 boxes have no asserts. I havenāt yet tested whether a bulk resubscribe on 20.21.0 triggers the asserts without the duplicates present.
Debugging and forum post creation assisted by Claude Opus 5.5
Yes, we already suspected that earlier and have been working down that road already.
I am seeing this on a couple systems exactly as described above. Yealink endpoints, same versions Asterisk and FreePBX. No additional data points, just āme too.ā
Scrubbed SIP debug logs in our https://pastebin.freepbx.org/ pastebin might help further efforts.
Following this thread, since we should be in the same boat due to our environment. But arenāt. Similar Yealink models deployed. Most all of our users are configured to e-mail voice mail messages and delete them from the voice mailboxes. So Iād say that we only have a small number of MWI subscribers. By the same token, I believe we only have maybe 2-3 users who have BLF features on their desk phones. All out of 100+ Yealinks, on the same FreePBX and Asterisk versions.
Am I to assume that the apparent bug is subscriber-related?
Yes, it appears to be - specifically extension state BLFs so MWI subscriptions would not trigger it. The number of subscriptions would be a factor, as well as the state of the CustomPresence that the hint references.
Makes sense, each of our extensions has between 5-10 BLFs configured. This was a legacy config and not necessary. Going to work through each site template and switch to speed dials.
I havenāt really had to look in detail at this area, so I ran this command below. Whatās odd is that only x839 is defined in User Management. x820 through x823 are new extensions I created a little while ago. But they arenāt in User Management, so I assume they arenāt associated with UCP or anything. These arenāt even SIP registered, as they will be new desk phones for a new site that isnāt ready yet.
root@DC-PBX03:~# asterisk -rx "database show CustomPresence"
/CustomPresence/820 : available,,
/CustomPresence/821 : available,,
/CustomPresence/822 : available,,
/CustomPresence/823 : available,,
/CustomPresence/839 : available,,
5 results found.
We are also seeing this across multiple systems. Grandstream phones. I have all of the logs uploaded in a Sangoma Case: 02265599
Another data point on this, with a workaround that has worked so far on our system.
Environment: FreePBX 17, Asterisk 22.11.0-1.sng12 (upgraded from 22.10.1 on Oct 5), Debian 12, about 25 PJSIP extensions with BLF keys on the desk phones.
Symptom: Asterisk was OOM-killed on Oct 28 at ~3 GB anon-rss. After the restart it sat at 2.3 GB RSS after under 3 hours. At that point pjsip show subscriptions inbound listed 653 subscriptions, of which 581 had Expiry: 0 and never went away (the count kept growing). Only ~70 were live.
What we found: core show hints showed that every extension hint contains CustomPresence:<ext>, but the AstDB family CustomPresence had no entry for our real extensions (only a few unused ones had one). The Presence column was empty for those. That looks like the condition described in asterisk#2179 (dialog subscriptions when a hint references CustomPresence without an AstDB entry, listed as a regression in 22.11.0 and fixed in 20.22.0-rc2 on the 20 branch). I havenāt confirmed that the fix has reached the 22 branch.
Workaround we applied: one entry per extension, then a restart:
asterisk -rx "database put CustomPresence XXX available,,"
(repeated for each extension)
fwconsole restart
Result so far (uptime [X]): 106 subscriptions, RSS 114 MB right after startup and MB now (about +0.3 MB/min in the first 30 minutes, versus roughly 14 MB/min before).
Caveats:
- This is correlation. We have hours of data and havenāt proven that #2179 is what drives the memory growth, only that adding the entries coincides with it stopping.
- Even with the workaround, 38
Expiry: 0entries appeared right after the restart and have stayed constant since. Each one has a live twin for the same endpoint/resource pair, so it looks like phones re-subscribed during the post-restart storm and the old dialogs werenāt released. They have not grown over hours. - We donāt know why the AstDB entries were missing in the first place.
@matthewljensen: if your leak is limited to a single extension, it might be worth checking whether that extensionās hint references CustomPresence with no matching entry:
asterisk -rx "core show hints" | grep -i presence
asterisk -rx "database show CustomPresence"
Happy to share more logs if useful.