Phone system all-stop

Running on FreePBX 2.9.0.7, installed from freepbx distro cd.

I got a message today from the office that they could not place calls anymore - says ‘all circuits are busy’. If you try to call in, you get a busy signal.

This is completely out of the blue. No configuration changes have been made in weeks, no handset additions/changes, etc. I have checked our internet service - nothing has changed, not even IP. Trunk registration shows green, 6 channels, the phones attached to the PBX no problem, and it’s been up for weeks without issue. The billing cycle hasn’t even completed yet and I know the card used is good. What else could it be that’s doing this? I even rebooted the asterisk server to see if that would somehow clear it up and it came back and seems happy as ever - except that the situation stays the same, calls out say circuits are busy and calls in get a busy signal.

Some recent log messages from /var/log/asterisk/full are as follows, and looks like the problem may exist on the provider end of things. Is there anything I can do?

– Executing [s@macro-dialout-trunk:20] Dial(“SIP/333-00000009”, “SIP/fpbx-2-f81014a1/12102795544,300,”) in new stack
[2011-09-29 16:03:00] VERBOSE[4641] netsock2.c: == Using SIP RTP TOS bits 184
[2011-09-29 16:03:00] VERBOSE[4641] netsock2.c: == Using SIP RTP CoS mark 5
[2011-09-29 16:03:00] VERBOSE[4641] app_dial.c: – Called fpbx-2-f81014a1/12102795544
[2011-09-29 16:03:00] VERBOSE[3087] chan_sip.c: – Got SIP response 480 “Temporarily Unavailable” back from 50.56.59.168:5060
[2011-09-29 16:03:00] VERBOSE[4641] app_dial.c: – SIP/fpbx-2-f81014a1-0000000b is circuit-busy
[2011-09-29 16:03:00] VERBOSE[4641] app_dial.c: == Everyone is busy/congested at this time (1:0/1/0)
[2011-09-29 16:03:00] VERBOSE[4641] pbx.c: – Executing [s@macro-dialout-trunk:21] NoOp(“SIP/333-00000009”, “Dial failed for some reason with DIALSTATUS = CONGESTION and HANGUPCAUSE = 19”) in new stack
[2011-09-29 16:03:00] VERBOSE[4641] pbx.c: – Executing [s@macro-dialout-trunk:22] Goto(“SIP/333-00000009”, “s-CONGESTION,1”) in new stack

Have you contacted your provider?

BF

The only contact method they provide is via email (regardless of support need - tech vs sales, etc), which I have used and given them the same log excerpt used here. The auto-responder indicates an SLA of 24hrs, since it arrived a couple minutes after 5pm EST.

Your logs indicate a provider problem.

I would not be happy with that SLA at all!

BF

That’s what I was afraid of (provider problem).

I’m not happy about it at all. If this doesn’t clear itself up soon there may be some strong words headed their way.


There server is up, it is actually sending back a temp unavailable message.

They are not much of a provider, just running a server at rackspace (I did a trace route on the IP).

I would consider a more responsive provider.

There server is up, it is actually sending back a temp unavailable message.

They are not much of a provider, just running a server at rackspace (I did a trace route on the IP).

I would consider a more responsive provider.

I didn’t think I could really go wrong with the provider that’s built into freepbx’s distribution, guess I was mistaken.

Any recommendations on a good one?

Built into FreePBX? That’s not a server for bandwidth.com that provides sip-station. Who are you talking about?

Nothing works. inbound goes straight to busy signal, outbound gets the ‘all circuits are busy’ message, or I saw one recently go by from one of the night staff saying that call from extension 75 was rejected because it could not be found in the context ‘from internal’.

So if this is some sub-contracted provider, how do I figure out who my trunks actually go through and get OFF them so I can get this system working again?

The only provider we have ever used is the one that goes through the SIPSTATION module on the FreePBX setup section of the web-admin pages. The account info goes to this URL: https://store.freepbx.com/store/index.php/myaccount and at the very top of the page the first thing you see is ‘SIPSTATION - powered by bandwidth.com’, the footer at the bottom of the page is ©Bandwidth.com, and if I remember right the guy who pays the bills asked who bandwidth.com was the first time the phone system hit the company card.

Color me confused if our provider isn’t bandwidth.com in the end.

That’s a slicehost account at rackspace (the IP you provided) unless I made a type.

I would find it hard to believe that bandwidth servers are on shared servers at Rackspace. Maybe this is a downstream local provide. Are all calls failing? Does inbound work?


Host                                                                                                                             Loss%  Last   Avg  Best  Wrst StDev
 1. 10.222.1.1                                                                                                                     2.0%  25.6 332.5  23.1 2146. 667.9
 2. 192.168.11.4                                                                                                                   2.0%  23.9 331.8  23.1 2150. 666.8
 3. 204.xx.xx.225                                                                                                                  0.1%  30.6  24.1  23.5 803.2   7.3
 4. 208.44.228.161                                                                                                                 0.1%  26.8  24.2  23.7 1022.  11.5
 5. ge-6-19.car1.Cleveland1.Level3.net                                                                                             0.0%  26.0  33.2  24.3 1021.  36.3
 6. ae-2-4.bar1.Cleveland1.Level3.net                                                                                              0.5%  26.6  25.3  24.1 1012.  17.6
 7. ae-0-11.bar2.Cleveland1.Level3.net                                                                                             0.0%  31.8  25.3  24.1 944.2  16.6
 8. ae-9-9.ebr2.Chicago1.Level3.net                                                                                                0.1%  40.1  34.1  33.4 884.9  11.4
 9. ae-5-5.ebr2.Chicago2.Level3.net                                                                                                0.2%  35.8  34.1  33.3 817.6  10.0
10. ae-2-52.edge1.Chicago2.Level3.net                                                                                              0.1%  66.3  34.2  33.5 1003.  13.0
11. RACKSPACE-M.edge1.Chicago2.Level3.net                                                                                          0.0%  37.1  34.7  34.0 936.0  10.6
12. 184.106.126.134                                                                                                                0.1%  40.4  38.5  37.9 872.3   9.9
13. 184.106.126.129                                                                                                                0.1%  39.3  38.6  37.9 804.3   8.8
14. core1-aggr301a-9.ord1.rackspace.net                                                                                            0.1%  39.3  38.5  37.7 1088.  11.3
15. 50-56-59-168.static.cloud-ips.com                                                                                              0.1%  38.8  37.7  37.2 1019.  10.0

If you have calls coming in to the from-internal context something is very wrong.

Bandwidth is a very large company, I am surprised at the rackspace deal. Beyond that I do not know what to say.

No - that was a call from an internal extension (device 75 is a desk in the sleep lab) trying to call out and getting rejected because it was not part of the ‘from internal’ context. I have never messed with the contexts, steered well clear and didn’t even install the ‘custom context’ module.

It’s possible that part may be mucked up because I tried to purge/recreate my trunk/route configurations in hopes that it might kick something loose, but still no dice and I will have to look at it in the morning. For now we are dead in the water and the problem started around 4pm out of the blue killing off both dial-in and dial-out ability, and the logs were spitting out that ‘all circuits busy’ message.

I don’t know, maybe that is not the right IP? Is the IP rejecting the call the same IP you have in the host field of your trunk.

The trunks are funneled through gateways - ‘trunk1.phonebooth.net’ and ‘trunk2.phonebooth.net’ - the SIPSTATION module only registers the two gateways but acknowledges six ‘channels’ (which normally are individual trunks, if I understand right).

When asterisk starts up, it registers with the two gateways I mentioned above and creates two trunks using a username-hash that is auto-generated and configured by the sipstation module using a provided key that came from the store/account-info URL I mentioned above. These two trunks have - up until now - worked with up to the 6 channel limit without issue (except for overriding in-progress calls that I was asking for help about before).

It really did go all quite well when it was first activated and put together, but seems to have gone belly up today. I’ve tried to report a problem to them but the only method possible is via ‘[email protected]’ - the only way to reach a person is via a phone number provided for service cancellation, or via their ‘professional services’ group for $150/hr and the problem there is that it’s not my problem, it’s outside me.

Man that’s frustrating, hopefully this will get resolved soon. You could get outbound up and running fairly quickly with one of the PAYG (pay-as-you-go) providers. Voip.ms, Callcentric, Voxbeam come to mind, but there are plenty of others. Yes it will cost a few bucks, but many have low minimums plus having a secondary provider is a good idea. And PAYG means no recurring charges and you can sit on whatever balance you have.

This won’t help your inbound, although does Sipstation provide an online dashboard with an option to (at least temporarily) forward inbound until this gets worked out?

If they have an inbound forwarding option, that would be great if I could get outbound up on an alternate provider. The office runs primarily off of one number (it’s all IVR and extensions when someone calls) so that would be great.

I don’t use Sipstation so I don’t know what ability they have to manage your account online. Forwarding to someone in the office’s cell would be the easiest option in this situation. Some of the PAYG providers can have you up and making outbound calls in minutes after signing up, although most will restrict the number of channels until the account is verified. $25 (or less) should be about the most you should spend initially.

One thought- does Sipstation use user/password authentication (are there username, secret fields in the trunk peer details?) or IP authentication? If the ISP changed the IP & Sipstation uses IP auth, that would cause failures.