Sudden Increase in Inbound Call Volume Crashes System

[swalker2001]

We produce two syndicated radio morning shows as well as other shows, and use FreePBX for all our studio phones. Request lines, contest lines etc. We have never had a problem with call volume on FreePBX before, but last week they decided to give away $1,000 every morning for three weeks. When they give out the number on the air hundreds, maybe thousands of people try to call in at once. Here is what happens:

First, the calls that are already on hold have no audio in either direction when they are picked up. Once all those lines are cleared, no calls can come into the system for a period of time. Up to an hour. Then everything is fine again until the next morning when they try this again.

Things we have tried:

1. Having the SIP provider decrease the Burst Rate on the trunk from 105 to 50. This seemed to help somewhat. The number of concurrent call paths is also 50. That didn’t change.
2. Asking the toll-free provider to limit the number of call paths from the 866 number to the DID. AT&T claims not to be able to do that.

Things we are thinking of trying:

1. Decreasing both the CCP and Burst Rate numbers even lower to say, 20.
2. Can’t think of anything for #2.

I don’t believe that Asterisk is actually crashing anymore but last week I was getting a lot of “Cannot allocate memory” and a few “FRACK! Failed assertion Excessive refcount” errors. The change in burst rate may have mitigated that somewhat.

Not sure what else to search for in the logs. I do see tons of “Everyone is busy” messages which I would expect to see with hundreds of calls trying to hit the system.

Any suggestions would be welcome.

Steve

[Stewart1 (Stewart)]

There was a good solution 60 years ago:

Choke exchange

Although they still exist, it isn’t a good solution today. If you port such a number to VoIP, the LRN will point to a new NXX and it won’t choke. An RCF (remote call forward) would work, but that’s expensive.

I hope that your business numbers and listener numbers are separate. Limiting the listener call paths to 3 or 4 should not be an issue. If occasionally all the initial callers are ineligible, people will be madly retrying busy signals and as soon as you end the calls, new ones will quickly appear.

@swalker2001

Asking the toll-free provider to limit the number of call paths from the 866 number to the DID.

Unless you have a special requirement (please explain), you should get the toll-free directly from your trunking provider, without an intermediate DID. This improves reliability, is usually less expensive, and in case of trouble, avoids the possibility of the providers pointing fingers at each other.

Regarding call paths, how many users can field calls? It usually doesn’t make sense to have more than two or three times that many, i.e. no more than two on hold or in queue for each one talking. Although you would like that no one gets a busy signal, being in queue for an hour isn’t fun, either. If you regularly (without a contest) get most callers queued, set up the queue to offer callback and/or voicemail after a few minutes waiting.

If “Burst Rate” refers to calls per second, 50 seems way high, unless you have a really large organization. On my system, it’s set at the provider to 3, and I’ve had only one instance ever where a caller got a busy signal because 4 calls arrived within the same second. Of course, I don’t do contests, but IMO you shouldn’t care if a few contesters get a gratuitous busy signal. If Burst Rate is per minute and 50 calls arrive all at once (with none in the following seconds), I don’t see why that should cause Asterisk to choke.

How big is your PBX (memory, CPU cores, clock rate, etc.)? Virtual or physical? Does it do anything requiring a lot of horsepower (transcoding, encryption, DTMF decoding, etc.)? I assume that you are recording all calls, but is that possibly a bottleneck (recording to a network drive with competing traffic, or a local drive shared with other VMs)?