ALL Feature codes are going out the trunk

edlentz · July 22, 2016, 3:07pm

The system is version 2.210.62-6 with asterisk 11.3. All of a sudden the Features codes do not work. They are sent out the trunks (SIP). Dialing *98 gets the login for the sip trunk provider Voicemail system. There are no apparent Abnormal Dial rules in the Outgoing routes. I even deleted them and rebuilt the dialing rules. Disabling the feature code in Feature Code Admin has no affect. I connected a Zoiper client to rule out the phones, it gets the same results. Earlier in the week we had an issue where we apparently had someone into the system and apparently installed something maloicious. It would dial or try to dial an international call every minute. Updating the system and having International calling turned off at the Trunk provider appears to have stopped that.

Here is the log for a call to *98

  -- Executing [*[email protected]:1] Macro("SIP/107-000000e6", "user-callerid,LIMIT,EXTERNAL,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:1] Macro("SIP/107-000000e6", "user-callerid,LIMIT,EXTERNAL,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] Set("SIP/107-000000e6", "TOUCH_MONITOR=1469199092.230") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] Set("SIP/107-000000e6", "TOUCH_MONITOR=1469199092.230") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:4] ExecIf("SIP/107-000000e6", "1?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:4] ExecIf("SIP/107-000000e6", "1?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:5] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:5] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:6] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:6] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:7] Set("SIP/107-000000e6", "AMPUSERCIDNAME=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:7] Set("SIP/107-000000e6", "AMPUSERCIDNAME=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:8] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:8] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:9] Set("SIP/107-000000e6", "AMPUSERCID=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:9] Set("SIP/107-000000e6", "AMPUSERCID=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:10] Set("SIP/107-000000e6", "__DIAL_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:10] Set("SIP/107-000000e6", "__DIAL_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:11] Set("SIP/107-000000e6", "CALLERID(all)="107" <107>") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:11] Set("SIP/107-000000e6", "CALLERID(all)="107" <107>") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:12] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:12] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:13] ExecIf("SIP/107-000000e6", "1?Set(GROUP(concurrency_limit)=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:13] ExecIf("SIP/107-000000e6", "1?Set(GROUP(concurrency_limit)=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:14] GosubIf("SIP/107-000000e6", "7?sub-ccss,s,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:14] GosubIf("SIP/107-000000e6", "7?sub-ccss,s,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] ExecIf("SIP/107-000000e6", "0?Return()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] ExecIf("SIP/107-000000e6", "0?Return()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] Set("SIP/107-000000e6", "CCSS_SETUP=TRUE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] Set("SIP/107-000000e6", "CCSS_SETUP=TRUE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GosubIf("SIP/107-000000e6", "0?monitor_config,1(from-internal,*98):monitor_default,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GosubIf("SIP/107-000000e6", "0?monitor_config,1(from-internal,*98):monitor_default,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] GotoIf("SIP/107-000000e6", "0?is_exten") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] GotoIf("SIP/107-000000e6", "0?is_exten") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] StackPop("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] StackPop("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] Return("SIP/107-000000e6", "FALSE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] Return("SIP/107-000000e6", "FALSE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:15] ExecIf("SIP/107-000000e6", "0?Set(CHANNEL(language)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:15] ExecIf("SIP/107-000000e6", "0?Set(CHANNEL(language)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:16] GotoIf("SIP/107-000000e6", "1?continue") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:16] GotoIf("SIP/107-000000e6", "1?continue") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-user-callerid,s,29)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-user-callerid,s,29)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:29] Set("SIP/107-000000e6", "CALLERID(number)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:29] Set("SIP/107-000000e6", "CALLERID(number)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:30] Set("SIP/107-000000e6", "CALLERID(name)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:30] Set("SIP/107-000000e6", "CALLERID(name)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:31] Set("SIP/107-000000e6", "CDR(cnum)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:31] Set("SIP/107-000000e6", "CDR(cnum)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:32] Set("SIP/107-000000e6", "CDR(cnam)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s[email protected]:32] Set("SIP/107-000000e6", "CDR(cnam)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:33] Set("SIP/107-000000e6", "CHANNEL(language)=en") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:33] Set("SIP/107-000000e6", "CHANNEL(language)=en") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:2] Set("SIP/107-000000e6", "MOHCLASS=default") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:2] Set("SIP/107-000000e6", "MOHCLASS=default") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:3] Set("SIP/107-000000e6", "_NODEST=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:3] Set("SIP/107-000000e6", "_NODEST=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:4] Macro("SIP/107-000000e6", "dialout-trunk,1,*98,,on") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*[email protected]:4] Macro("SIP/107-000000e6", "dialout-trunk,1,*98,,on") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] Set("SIP/107-000000e6", "DIAL_TRUNK=1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] Set("SIP/107-000000e6", "DIAL_TRUNK=1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] GosubIf("SIP/107-000000e6", "0?sub-pincheck,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] GosubIf("SIP/107-000000e6", "0?sub-pincheck,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GotoIf("SIP/107-000000e6", "0?disabletrunk,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GotoIf("SIP/107-000000e6", "0?disabletrunk,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:4] Set("SIP/107-000000e6", "DIAL_NUMBER=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:4] Set("SIP/107-000000e6", "DIAL_NUMBER=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:5] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:5] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:6] Set("SIP/107-000000e6", "OUTBOUND_GROUP=OUT_1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:6] Set("SIP/107-000000e6", "OUTBOUND_GROUP=OUT_1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:7] GotoIf("SIP/107-000000e6", "0?nomax") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:7] GotoIf("SIP/107-000000e6", "0?nomax") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:8] GotoIf("SIP/107-000000e6", "0?chanfull") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:8] GotoIf("SIP/107-000000e6", "0?chanfull") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:9] GotoIf("SIP/107-000000e6", "0?skipoutcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:9] GotoIf("SIP/107-000000e6", "0?skipoutcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:10] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:10] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:11] Macro("SIP/107-000000e6", "outbound-callerid,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:11] Macro("SIP/107-000000e6", "outbound-callerid,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] ExecIf("SIP/107-000000e6", "0?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] ExecIf("SIP/107-000000e6", "0?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GotoIf("SIP/107-000000e6", "1?normcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:3] GotoIf("SIP/107-000000e6", "1?normcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,6)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,6)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:6] Set("SIP/107-000000e6", "USEROUTCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:6] Set("SIP/107-000000e6", "USEROUTCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:7] Set("SIP/107-000000e6", "EMERGENCYCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:7] Set("SIP/107-000000e6", "EMERGENCYCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:8] Set("SIP/107-000000e6", "TRUNKOUTCID=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:8] Set("SIP/107-000000e6", "TRUNKOUTCID=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:9] GotoIf("SIP/107-000000e6", "1?trunkcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:9] GotoIf("SIP/107-000000e6", "1?trunkcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,14)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,14)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:14] ExecIf("SIP/107-000000e6", "1?Set(CALLERID(all)=9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:14] ExecIf("SIP/107-000000e6", "1?Set(CALLERID(all)=9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:15] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:15] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:16] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:16] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:17] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=prohib_passed_screen)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:17] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=prohib_passed_screen)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:18] Set("SIP/107-000000e6", "CDR(outbound_cnum)=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:18] Set("SIP/107-000000e6", "CDR(outbound_cnum)=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:19] Set("SIP/107-000000e6", "CDR(outbound_cnam)=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:19] Set("SIP/107-000000e6", "CDR(outbound_cnam)=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:12] GosubIf("SIP/107-000000e6", "0?sub-flp-1,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:12] GosubIf("SIP/107-000000e6", "0?sub-flp-1,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:13] Set("SIP/107-000000e6", "OUTNUM=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:13] Set("SIP/107-000000e6", "OUTNUM=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:14] Set("SIP/107-000000e6", "custom=SIP/Clearrate") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:14] Set("SIP/107-000000e6", "custom=SIP/Clearrate") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:15] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(setmusic^default))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:15] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(setmusic^default))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:16] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(confirm))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:16] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(confirm))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:17] Macro("SIP/107-000000e6", "dialout-trunk-predial-hook,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:17] Macro("SIP/107-000000e6", "dialout-trunk-predial-hook,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] MacroExit("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] MacroExit("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:18] GotoIf("SIP/107-000000e6", "0?bypass,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:18] GotoIf("SIP/107-000000e6", "0?bypass,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:19] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(num,i)=*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:19] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(num,i)=*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:20] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(name,i)=CID:9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:20] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(name,i)=CID:9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:21] GotoIf("SIP/107-000000e6", "0?customtrunk") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:21] GotoIf("SIP/107-000000e6", "0?customtrunk") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:22] Dial("SIP/107-000000e6", "SIP/Clearrate/*98,300,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:22] Dial("SIP/107-000000e6", "SIP/Clearrate/*98,300,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP TOS bits 184
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP TOS bits 184
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP CoS mark 5
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP CoS mark 5
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- Called SIP/Clearrate/*98
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- Called SIP/Clearrate/*98
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- SIP/Clearrate-000000e7 answered SIP/107-000000e6
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- SIP/Clearrate-000000e7 answered SIP/107-000000e6
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] Macro("SIP/107-000000e6", "hangupcall,") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] Macro("SIP/107-000000e6", "hangupcall,") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] ExecIf("SIP/107-000000e6", "0?Set(CDR(recordingfile)=.)") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:1] ExecIf("SIP/107-000000e6", "0?Set(CDR(recordingfile)=.)") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] GotoIf("SIP/107-000000e6", "1?theend") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:2] GotoIf("SIP/107-000000e6", "1?theend") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-hangupcall,s,4)
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-hangupcall,s,4)
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:4] Hangup("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [[email protected]:4] Hangup("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] app_macro.c:   == Spawn extension (macro-hangupcall, s, 4) exited non-zero on 'SIP/107-000000e6' in macro 'hangupcall'
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] app_macro.c:   == Spawn extension (macro-hangupcall, s, 4) exited non-zero on 'SIP/107-000000e6' in macro 'hangupcall'
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:   == Spawn extension (macro-dialout-trunk, h, 1) exited non-zero on 'SIP/107-000000e6'

Can anyone suggest a starting point to get this resolved?

Thanks

edlentz · July 22, 2016, 3:15pm

FYI, In the error notification on the Dashboard I have this Cronmanager error:

The following commands failed with the listed error
/usr/bin/cp /tmp/.vivo.php /var/www/html/.marvels.php; (127)

They appear to be gone. From a command line logged in as root I do not see them.

cynjut · July 22, 2016, 3:25pm

If it was my system, I’d tear it down to the metal this weekend and start over.

lgaetz · July 22, 2016, 4:37pm

This is indicative of an exploit of some sort, this PBX needs to be removed from service asap

edlentz · July 22, 2016, 6:57pm

We will probably be replacing this system this afternoon. We might have a couple more of them. Not sure yet.

dicko · July 23, 2016, 3:39am

Well I came across this post,

You HAVE been compromised as Lorne suggested, It is probably too late to fix it now but please consider the value of a previous post of mine:-

It has saved me in the past from similar intrusions that get past the standard signature checking etc. currently in FreePBX which operates at a higher level but misses the sneaky bastards, it will catch ANY changes made in /etc/asterisk ( your problem) or /var/www/html/ yes you will get an email soon after you press reload, that would be expected, but if you get an email and you had not done that, then it’s time to get your detective boots on. if necessary repairing the damage is as trivial as an rsync from the snapshot directory.

Another suggestion, never have your /tmp filesystem on a hard disk, it’s just begging for trouble and the damage remains even after a reboot.

edlentz · July 23, 2016, 12:06pm

Thanks for the info dicko. I did replace that system. We are investigating, but I think I may have others. It seems to be older systems that are affected. So rsnapshot basically sets up a diff file for the target and checks it and if there is a change it notes it and emails an alert?

dicko · July 23, 2016, 1:04pm

Not quite but a few minutes at https://rsnapshot.org wil clue you in.

edlentz · July 23, 2016, 1:42pm

I went there and did some reading. We have discussed using rsync to update some custom files we put on the systems, this looks way better! Thanks for the tip.

dicko · July 23, 2016, 6:44pm

Another suggestion is to also use something like rkhunter, it also notices lots of untoward activity.

edlentz · August 10, 2016, 2:04pm

Thanks dicko