Calls shot up, crashed server

General Help
1

Hi all,

We have a server that’s been in production for a couple years with about 160 extensions.

FreePBX 13.0.167 with Asterisk 13.10.0

Yesterday, within about a 10 minute period, we had calls shoot up from the average of around 15, to 87. I found this out via the Dashboard after we forced a restart, as we couldn’t make calls, and the GUI was unresponsive.

After the restart, the Dashboard showed that the 87 calls had flat-lined for a 30 mn period or so. I’m assuming this is what crashed the server.

Has anyone seen anything like this before? I know there are a ton of questions about firewalls and security, but I don’t believe that has anything to do with what happened.

Also, are inbound calls are capped at 25 at the VSP, so the majority of the calls must have been outbound, or the server crashed for some other reason and falsely reported the 87 calls.

2

Why would you think that?

cat /var/log/asterisk/full*|grep -Ei "@from-trunk:1|answered|dial\("|sort -n

might give you insight.

1 Like
3

Hi dicko,

I think that because of how the client’s server is locked down, and how no other servers were affected.

I ran the cat. What am I looking for?

4

suspicious increase in calls at “that time” the calls both in and out will be grepped, if as you say 30 mins than add |hangup to the regex, but if the calls are real you will see them and the difference between dial and hangup, the rest is kind of just detective work.

5

Thanks.

So I found where it happened, as there is a 39 mn gap after, and a weird block of log that extends up the the last second before that 39 mn gap. Here it is. Is there anything actually odd about this, or does it give any insight as to what might have happened?

[2017-03-10 13:20:25] VERBOSE[62877][C-00013e39] pbx.c: Executing [s@macro-dial:17] Dial("SIP/anveo_out-00044d6f", "SIP/405&SIP/408&SIP/401&SIP/404&SIP/406&SIP/407&SIP/402,50,trM(auto-blkvm)b(func-apply-sipheaders^s^1),") in new stack
[2017-03-10 13:23:11] VERBOSE[63131][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144bb;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:11] VERBOSE[63132][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144bc;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:11] VERBOSE[63133][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144bd;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:31] VERBOSE[63136][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144be;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:31] VERBOSE[63137][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144bf;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:31] VERBOSE[63138][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144c0;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:51] VERBOSE[63157][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144c1;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:51] VERBOSE[63158][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144c2;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:23:51] VERBOSE[63159][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144c3;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:11] VERBOSE[63202][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144c4;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:11] VERBOSE[63203][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144c5;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:11] VERBOSE[63204][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144c6;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:31] VERBOSE[63227][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144c7;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:31] VERBOSE[63242][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144c8;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:31] VERBOSE[63243][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144c9;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:51] VERBOSE[63261][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144ca;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:51] VERBOSE[63262][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144cb;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:24:51] VERBOSE[63263][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144cc;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:11] VERBOSE[63296][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144cd;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:11] VERBOSE[63297][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144ce;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:11] VERBOSE[63298][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144cf;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:31] VERBOSE[63338][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144d0;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:31] VERBOSE[63339][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144d1;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:31] VERBOSE[63340][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144d2;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:51] VERBOSE[63342][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144d3;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:51] VERBOSE[63343][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144d4;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:25:51] VERBOSE[63344][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144d5;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:11] VERBOSE[63388][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144d6;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:11] VERBOSE[63389][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144d7;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:11] VERBOSE[63390][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144d8;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:31] VERBOSE[63429][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144d9;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:31] VERBOSE[63430][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144da;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:31] VERBOSE[63434][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144db;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:51] VERBOSE[63436][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144dc;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:51] VERBOSE[63437][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144dd;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:26:51] VERBOSE[63438][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144de;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:11] VERBOSE[63481][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144df;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:11] VERBOSE[63482][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144e0;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:11] VERBOSE[63483][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144e1;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:31] VERBOSE[63505][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144e2;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:31] VERBOSE[63506][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144e3;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:31] VERBOSE[63507][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144e4;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:51] VERBOSE[63526][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144e5;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:51] VERBOSE[63527][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144e6;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:27:51] VERBOSE[63528][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144e7;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:11] VERBOSE[63558][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144e8;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:11] VERBOSE[63559][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144e9;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:11] VERBOSE[63560][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144ea;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:31] VERBOSE[63663][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144eb;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:31] VERBOSE[63664][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144ec;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:31] VERBOSE[63665][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144ed;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:51] VERBOSE[63684][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144ee;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:51] VERBOSE[63685][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144ef;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:28:51] VERBOSE[63686][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144f0;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:11] VERBOSE[63724][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144f1;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:11] VERBOSE[63725][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144f2;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:11] VERBOSE[63726][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144f3;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:31] VERBOSE[63747][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144f4;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:31] VERBOSE[63748][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144f5;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:31] VERBOSE[63749][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144f6;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:51] VERBOSE[63754][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/601@from-queue-000144f7;2", "SIP/601,,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:51] VERBOSE[63755][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/602@from-queue-000144f8;2", "SIP/602,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:29:51] VERBOSE[63756][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("Local/604@from-queue-000144f9;2", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
[2017-03-10 13:30:11] VERBOSE[62841][C-00013e35] pbx.c: Executing [s@macro-dial-one:48] Dial("SIP/anveo_out-00044d63", "SIP/604,23,trM(auto-blkvm)Ib(func-apply-sipheaders^s^1)") in new stack
6

I would check your queue members for a call-forward that might be re-entrant, not necessari[y in FreePBX

1 Like
7

Meaning possibly a call forward on their phone itself? Re-entrant meaning creating a loop somehow?

8

Yes, but it’s only a guess.

9

Ya, I noticed there are a few blocks like that over the past few days, just not as big as the one posted. I’ve got some digging to do. Thanks for your input once again dicko.

1 Like