ALL Feature codes are going out the trunk

[edlentz]

The system is version 2.210.62-6 with asterisk 11.3. All of a sudden the Features codes do not work. They are sent out the trunks (SIP). Dialing *98 gets the login for the sip trunk provider Voicemail system. There are no apparent Abnormal Dial rules in the Outgoing routes. I even deleted them and rebuilt the dialing rules. Disabling the feature code in Feature Code Admin has no affect. I connected a Zoiper client to rule out the phones, it gets the same results. Earlier in the week we had an issue where we apparently had someone into the system and apparently installed something maloicious. It would dial or try to dial an international call every minute. Updating the system and having International calling turned off at the Trunk provider appears to have stopped that.

Here is the log for a call to *98

  -- Executing [*98@from-internal:1] Macro("SIP/107-000000e6", "user-callerid,LIMIT,EXTERNAL,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:1] Macro("SIP/107-000000e6", "user-callerid,LIMIT,EXTERNAL,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:1] Set("SIP/107-000000e6", "TOUCH_MONITOR=1469199092.230") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:1] Set("SIP/107-000000e6", "TOUCH_MONITOR=1469199092.230") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:2] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:2] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:3] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:3] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:4] ExecIf("SIP/107-000000e6", "1?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:4] ExecIf("SIP/107-000000e6", "1?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:5] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:5] Set("SIP/107-000000e6", "AMPUSER=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:6] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:6] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:7] Set("SIP/107-000000e6", "AMPUSERCIDNAME=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:7] Set("SIP/107-000000e6", "AMPUSERCIDNAME=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:8] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:8] GotoIf("SIP/107-000000e6", "0?report") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:9] Set("SIP/107-000000e6", "AMPUSERCID=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:9] Set("SIP/107-000000e6", "AMPUSERCID=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:10] Set("SIP/107-000000e6", "__DIAL_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:10] Set("SIP/107-000000e6", "__DIAL_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:11] Set("SIP/107-000000e6", "CALLERID(all)="107" <107>") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:11] Set("SIP/107-000000e6", "CALLERID(all)="107" <107>") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:12] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:12] GotoIf("SIP/107-000000e6", "0?limit") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:13] ExecIf("SIP/107-000000e6", "1?Set(GROUP(concurrency_limit)=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:13] ExecIf("SIP/107-000000e6", "1?Set(GROUP(concurrency_limit)=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:14] GosubIf("SIP/107-000000e6", "7?sub-ccss,s,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:14] GosubIf("SIP/107-000000e6", "7?sub-ccss,s,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@sub-ccss:1] ExecIf("SIP/107-000000e6", "0?Return()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@sub-ccss:1] ExecIf("SIP/107-000000e6", "0?Return()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@sub-ccss:2] Set("SIP/107-000000e6", "CCSS_SETUP=TRUE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@sub-ccss:2] Set("SIP/107-000000e6", "CCSS_SETUP=TRUE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@sub-ccss:3] GosubIf("SIP/107-000000e6", "0?monitor_config,1(from-internal,*98):monitor_default,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@sub-ccss:3] GosubIf("SIP/107-000000e6", "0?monitor_config,1(from-internal,*98):monitor_default,1(from-internal,*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [monitor_default@sub-ccss:1] GotoIf("SIP/107-000000e6", "0?is_exten") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [monitor_default@sub-ccss:1] GotoIf("SIP/107-000000e6", "0?is_exten") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [monitor_default@sub-ccss:2] StackPop("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [monitor_default@sub-ccss:2] StackPop("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [monitor_default@sub-ccss:3] Return("SIP/107-000000e6", "FALSE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [monitor_default@sub-ccss:3] Return("SIP/107-000000e6", "FALSE") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:15] ExecIf("SIP/107-000000e6", "0?Set(CHANNEL(language)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:15] ExecIf("SIP/107-000000e6", "0?Set(CHANNEL(language)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:16] GotoIf("SIP/107-000000e6", "1?continue") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:16] GotoIf("SIP/107-000000e6", "1?continue") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-user-callerid,s,29)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-user-callerid,s,29)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:29] Set("SIP/107-000000e6", "CALLERID(number)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:29] Set("SIP/107-000000e6", "CALLERID(number)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:30] Set("SIP/107-000000e6", "CALLERID(name)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:30] Set("SIP/107-000000e6", "CALLERID(name)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:31] Set("SIP/107-000000e6", "CDR(cnum)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:31] Set("SIP/107-000000e6", "CDR(cnum)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:32] Set("SIP/107-000000e6", "CDR(cnam)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:32] Set("SIP/107-000000e6", "CDR(cnam)=107") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:33] Set("SIP/107-000000e6", "CHANNEL(language)=en") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-user-callerid:33] Set("SIP/107-000000e6", "CHANNEL(language)=en") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:2] Set("SIP/107-000000e6", "MOHCLASS=default") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:2] Set("SIP/107-000000e6", "MOHCLASS=default") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:3] Set("SIP/107-000000e6", "_NODEST=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:3] Set("SIP/107-000000e6", "_NODEST=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:4] Macro("SIP/107-000000e6", "dialout-trunk,1,*98,,on") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [*98@from-internal:4] Macro("SIP/107-000000e6", "dialout-trunk,1,*98,,on") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:1] Set("SIP/107-000000e6", "DIAL_TRUNK=1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:1] Set("SIP/107-000000e6", "DIAL_TRUNK=1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:2] GosubIf("SIP/107-000000e6", "0?sub-pincheck,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:2] GosubIf("SIP/107-000000e6", "0?sub-pincheck,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:3] GotoIf("SIP/107-000000e6", "0?disabletrunk,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:3] GotoIf("SIP/107-000000e6", "0?disabletrunk,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:4] Set("SIP/107-000000e6", "DIAL_NUMBER=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:4] Set("SIP/107-000000e6", "DIAL_NUMBER=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:5] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:5] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=tr") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:6] Set("SIP/107-000000e6", "OUTBOUND_GROUP=OUT_1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:6] Set("SIP/107-000000e6", "OUTBOUND_GROUP=OUT_1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:7] GotoIf("SIP/107-000000e6", "0?nomax") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:7] GotoIf("SIP/107-000000e6", "0?nomax") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:8] GotoIf("SIP/107-000000e6", "0?chanfull") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:8] GotoIf("SIP/107-000000e6", "0?chanfull") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:9] GotoIf("SIP/107-000000e6", "0?skipoutcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:9] GotoIf("SIP/107-000000e6", "0?skipoutcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:10] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:10] Set("SIP/107-000000e6", "DIAL_TRUNK_OPTIONS=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:11] Macro("SIP/107-000000e6", "outbound-callerid,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:11] Macro("SIP/107-000000e6", "outbound-callerid,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:1] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:1] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:2] ExecIf("SIP/107-000000e6", "0?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:2] ExecIf("SIP/107-000000e6", "0?Set(REALCALLERIDNUM=107)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:3] GotoIf("SIP/107-000000e6", "1?normcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:3] GotoIf("SIP/107-000000e6", "1?normcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,6)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,6)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:6] Set("SIP/107-000000e6", "USEROUTCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:6] Set("SIP/107-000000e6", "USEROUTCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:7] Set("SIP/107-000000e6", "EMERGENCYCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:7] Set("SIP/107-000000e6", "EMERGENCYCID=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:8] Set("SIP/107-000000e6", "TRUNKOUTCID=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:8] Set("SIP/107-000000e6", "TRUNKOUTCID=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:9] GotoIf("SIP/107-000000e6", "1?trunkcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:9] GotoIf("SIP/107-000000e6", "1?trunkcid") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,14)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-outbound-callerid,s,14)
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:14] ExecIf("SIP/107-000000e6", "1?Set(CALLERID(all)=9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:14] ExecIf("SIP/107-000000e6", "1?Set(CALLERID(all)=9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:15] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:15] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:16] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:16] ExecIf("SIP/107-000000e6", "0?Set(CALLERID(all)=)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:17] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=prohib_passed_screen)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:17] ExecIf("SIP/107-000000e6", "0?Set(CALLERPRES()=prohib_passed_screen)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:18] Set("SIP/107-000000e6", "CDR(outbound_cnum)=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:18] Set("SIP/107-000000e6", "CDR(outbound_cnum)=9898354796") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:19] Set("SIP/107-000000e6", "CDR(outbound_cnam)=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-outbound-callerid:19] Set("SIP/107-000000e6", "CDR(outbound_cnam)=") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:12] GosubIf("SIP/107-000000e6", "0?sub-flp-1,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:12] GosubIf("SIP/107-000000e6", "0?sub-flp-1,s,1()") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:13] Set("SIP/107-000000e6", "OUTNUM=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:13] Set("SIP/107-000000e6", "OUTNUM=*98") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:14] Set("SIP/107-000000e6", "custom=SIP/Clearrate") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:14] Set("SIP/107-000000e6", "custom=SIP/Clearrate") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:15] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(setmusic^default))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:15] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(setmusic^default))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:16] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(confirm))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:16] ExecIf("SIP/107-000000e6", "0?Set(DIAL_TRUNK_OPTIONS=M(confirm))") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:17] Macro("SIP/107-000000e6", "dialout-trunk-predial-hook,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:17] Macro("SIP/107-000000e6", "dialout-trunk-predial-hook,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk-predial-hook:1] MacroExit("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk-predial-hook:1] MacroExit("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:18] GotoIf("SIP/107-000000e6", "0?bypass,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:18] GotoIf("SIP/107-000000e6", "0?bypass,1") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:19] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(num,i)=*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:19] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(num,i)=*98)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:20] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(name,i)=CID:9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:20] ExecIf("SIP/107-000000e6", "1?Set(CONNECTEDLINE(name,i)=CID:9898354796)") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:21] GotoIf("SIP/107-000000e6", "0?customtrunk") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:21] GotoIf("SIP/107-000000e6", "0?customtrunk") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:22] Dial("SIP/107-000000e6", "SIP/Clearrate/*98,300,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-dialout-trunk:22] Dial("SIP/107-000000e6", "SIP/Clearrate/*98,300,") in new stack
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP TOS bits 184
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP TOS bits 184
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP CoS mark 5
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] netsock2.c:   == Using SIP RTP CoS mark 5
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- Called SIP/Clearrate/*98
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- Called SIP/Clearrate/*98
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- SIP/Clearrate-000000e7 answered SIP/107-000000e6
[2016-07-22 10:51:32] VERBOSE[28871][C-00000164] app_dial.c:     -- SIP/Clearrate-000000e7 answered SIP/107-000000e6
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [h@macro-dialout-trunk:1] Macro("SIP/107-000000e6", "hangupcall,") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [h@macro-dialout-trunk:1] Macro("SIP/107-000000e6", "hangupcall,") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-hangupcall:1] ExecIf("SIP/107-000000e6", "0?Set(CDR(recordingfile)=.)") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-hangupcall:1] ExecIf("SIP/107-000000e6", "0?Set(CDR(recordingfile)=.)") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-hangupcall:2] GotoIf("SIP/107-000000e6", "1?theend") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-hangupcall:2] GotoIf("SIP/107-000000e6", "1?theend") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-hangupcall,s,4)
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Goto (macro-hangupcall,s,4)
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-hangupcall:4] Hangup("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:     -- Executing [s@macro-hangupcall:4] Hangup("SIP/107-000000e6", "") in new stack
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] app_macro.c:   == Spawn extension (macro-hangupcall, s, 4) exited non-zero on 'SIP/107-000000e6' in macro 'hangupcall'
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] app_macro.c:   == Spawn extension (macro-hangupcall, s, 4) exited non-zero on 'SIP/107-000000e6' in macro 'hangupcall'
[2016-07-22 10:51:36] VERBOSE[28871][C-00000164] pbx.c:   == Spawn extension (macro-dialout-trunk, h, 1) exited non-zero on 'SIP/107-000000e6'

Can anyone suggest a starting point to get this resolved?

Thanks

[edlentz]

FYI, In the error notification on the Dashboard I have this Cronmanager error:

The following commands failed with the listed error
/usr/bin/cp /tmp/.vivo.php /var/www/html/.marvels.php; (127)

They appear to be gone. From a command line logged in as root I do not see them.

[cynjut (Dave Burgess)]

If it was my system, I’d tear it down to the metal this weekend and start over.

[lgaetz (Lorne Gaetz)]

@edlentz

/usr/bin/cp /tmp/.vivo.php /var/www/html/.marvels.php;

This is indicative of an exploit of some sort, this PBX needs to be removed from service asap

[edlentz]

We will probably be replacing this system this afternoon. We might have a couple more of them. Not sure yet.

[dicko]

Well I came across this post,

You HAVE been compromised as Lorne suggested, It is probably too late to fix it now but please consider the value of a previous post of mine:-

Any way to disable module signature checking for an individual module?

This is perhaps off topic, but I will share my recipe, it doesn’t use sexy pgp signing but has certain advantages when it comes to recovery and is just good old bash and perl scripting and some general linux utilities already installed on every machine. I have used it successfully for several years and it has occasionally protected me from myself and a couple of old invasions of /recordings Install rnapshot yum|apt-get install rsnapshot with a schedule to suit, I use a thirty minute…

It has saved me in the past from similar intrusions that get past the standard signature checking etc. currently in FreePBX which operates at a higher level but misses the sneaky bastards, it will catch ANY changes made in /etc/asterisk ( your problem) or /var/www/html/ yes you will get an email soon after you press reload, that would be expected, but if you get an email and you had not done that, then it’s time to get your detective boots on. if necessary repairing the damage is as trivial as an rsync from the snapshot directory.

Another suggestion, never have your /tmp filesystem on a hard disk, it’s just begging for trouble and the damage remains even after a reboot.

[edlentz]

Thanks for the info dicko. I did replace that system. We are investigating, but I think I may have others. It seems to be older systems that are affected. So rsnapshot basically sets up a diff file for the target and checks it and if there is a change it notes it and emails an alert?

[dicko]

Not quite but a few minutes at https://rsnapshot.org wil clue you in.

[edlentz]

I went there and did some reading. We have discussed using rsync to update some custom files we put on the systems, this looks way better! Thanks for the tip.

[dicko]

Another suggestion is to also use something like rkhunter, it also notices lots of untoward activity.

[edlentz]

Thanks dicko