@jcolp Another data point that may help isolate this. On our box the leak is limited to a single extension, the trigger looks like asterisk/asterisk#2179 from the 20.21.0 / 22.11.0 / 23.5.0 release thread, and the box crashed twice when many phones resubscribed at once.
How we got here: This is a FreePBX 16 system on SNG7. On 10-25 I applied the PHP2Shell update with yum upgrade (PHP 7.4.16 → 7.4.33), updated the FreePBX modules, and rebooted. The yum upgrade also took Asterisk from 20.17.0 to 20.21.0, the same release as 22.11.0.
Setup: Small VM, 2 vCPU and 1.8 GB RAM. About 25 Yealink SIP-T57W phones on firmware 96.86.0.70, mostly connecting over TCP through the sysadmin OpenVPN. Every phone has BLF keys for most extensions and the park slots.
The leak, about 3 hours after an Asterisk restart:
$ asterisk -rx "pjsip list subscriptions inbound" | grep -oE '[^ ]+/(dialog|message-summary|presence)' | sort | uniq -c | sort -rn | head -5
952 200/dialog
24 303/dialog
24 208/dialog
24 204/dialog
24 203/dialog
$ asterisk -rx "core show taskprocessors" | grep -c 'extension_state:extension/200@'
952
$ asterisk -rx "core show taskprocessors" | grep -c 'extension_state:'
1546
$ asterisk -rx "pjsip list subscriptions inbound" | tail -1
1544 active subscriptions
Every other extension and park hint has about 24 subscriptions, one per phone. Extension 200 has 952, because each phone holds 41 to 45 separate subscriptions to it, each with a different Call-ID and all showing a live expiry. The extension_state taskprocessors match those subscriptions 1:1. Total subscriptions went from 1466 to 1535 in about 15 minutes, nearly all of it on 200. An hour later 200 was at 1,141, about 50 per phone.
What’s different about 200: it was the only watched hint with no presence value. database show CustomPresence had an entry for every extension except 200.
200@ext-local : PJSIP/200&Custom:DND200,CustomPresence:200 State:Idle Presence: Watchers 1315
203@ext-local : PJSIP/203&Custom:DND203,CustomPresence:203 State:Idle Presence:available Watchers 25
Test: I ran presencestate change CustomPresence:200 AVAILABLE. Watchers rose briefly to 1,343 while the phones finished their retries, then held flat at 1,343 for 10+ minutes. Before the change it had been growing by about 6 a minute. It looks like the missing initial NOTIFY makes the Yealinks resubscribe over and over, and Asterisk never cleans up the old subscriptions. The leaked subscriptions also haven’t expired since the fix, so they seem to stay until a restart.
Two other 20.21.0 boxes of ours have had no duplicate subscriptions in 37 hours. The only hints there with no presence value are extensions that no phone has a BLF key for.
@tcl it may be worth checking database show CustomPresence for the extensions with all the duplicate extension_state entries.
Memory: Asterisk RssAnon+VmSwap was 344 MB about 3 hours after the restart and 395 MB an hour later. Our other 20.21.0 b sit at 90 to 150 MB.
Crashes: Twice in about 19 hours on 20.21.0. Each came right after many phones resubscribed at once, while extension 200 had the duplicate subscriptions.
- Crash 1: Several phones’ TCP connections reset within 7 seconds of each other (they re-registered from new source ports). Asserts began about 20 seconds later, and Asterisk crashed 20 seconds after that.
- Crash 2: Assert bursts began 2 to 3 minutes after each BLF key push from Endpoint Manager, when about 25 Yealinks re-provisioned and resubscribed. The crash came 4 minutes after one of those pushes.
ERROR[...] pjproject: ../src/pj/lock.c .Assert failed: cnt > 0
ERROR[...] pjproject: ../src/pj/os_core_unix .Assert failed: mutex
The same box logged zero asserts in a week on 20.17.0, including a similar burst of TCP reconnects that caused no problems. Our two other 20.21.0 boxes have no asserts. I haven’t yet tested whether a bulk resubscribe on 20.21.0 triggers the asserts without the duplicates present.
Debugging and forum post creation assisted by Claude Opus 5.5