IAX Truck One way does not work

I have two boxes setup at two locations with a iax trunk between then. today one side just stopped working.

When dialing out i get all circuits are busy

asterisk -rvvvv
Asterisk 13.9.1, Copyright © 1999 - 2014, Digium, Inc. and others.
Created by Mark Spencer [email protected]
Asterisk comes with ABSOLUTELY NO WARRANTY; type ‘core show warranty’ for detail s.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type ‘core show license’ for details.

Connected to Asterisk 13.9.1 currently running on apdpbx (pid = 21513)
[2016-09-22 15:34:33] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 689
[2016-09-22 15:34:33] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 649
[2016-09-22 15:34:34] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 689
[2016-09-22 15:34:34] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 649
– Channel SIP/621-0000b68a left ‘simple_bridge’ basic-bridge
– Channel IAX2/freepbx-130 left ‘simple_bridge’ basic-bridge
== Extension Changed 621[ext-local] new state Idle for Notify User 678
== Extension Changed 621[ext-local] new state Idle for Notify User 639
== Extension Changed 621[ext-local] new state Idle for Notify User 668
== Extension Changed 621[ext-local] new state Idle for Notify User 622
== Extension Changed 621[ext-local] new state Idle for Notify User 642
== Spawn extension (macro-dial, s, 17) exited non-zero on ‘IAX2/freepbx-130’ in macro ‘dial’
== Extension Changed 621[ext-local] new state Idle for Notify User 683
== Spawn extension (followme-sub, 621, 38) exited non-zero on ‘IAX2/freepbx-130’
== Extension Changed 621[ext-local] new state Idle for Notify User 660
== Extension Changed 621[ext-local] new state Idle for Notify User 646
– Hungup ‘IAX2/freepbx-130’
== Extension Changed 621[ext-local] new state Idle for Notify User 643
== Extension Changed 621[ext-local] new state Idle for Notify User 664
== MixMonitor close filestream (mixed)
== End MixMonitor Recording IAX2/freepbx-130
== Extension Changed 621[ext-local] new state Idle for Notify User 623
== Extension Changed 621[ext-local] new state Idle for Notify User 624
== Extension Changed 621[ext-local] new state Idle for Notify User 662
== Extension Changed 621[ext-local] new state Idle for Notify User 663
== Extension Changed 621[ext-local] new state Idle for Notify User 621
== Extension Changed 621[ext-local] new state Idle for Notify User 634
[2016-09-22 15:34:40] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 692
[2016-09-22 15:34:41] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 692
== Using SIP RTP TOS bits 184
== Using SIP RTP CoS mark 5
== Extension Changed 621[ext-local] new state InUse for Notify User 678
== Extension Changed 621[ext-local] new state InUse for Notify User 639
== Extension Changed 621[ext-local] new state InUse for Notify User 668
== Extension Changed 621[ext-local] new state InUse for Notify User 622
== Extension Changed 621[ext-local] new state InUse for Notify User 642
– Executing [[email protected]:1] Macro(“SIP/621-0000b68b”, “user-callerid,LIMIT,EXTERNAL,”) in new stack
== Extension Changed 621[ext-local] new state InUse for Notify User 683
== Extension Changed 621[ext-local] new state InUse for Notify User 660
== Extension Changed 621[ext-local] new state InUse for Notify User 646
== Extension Changed 621[ext-local] new state InUse for Notify User 643
– Executing [[email protected]:1] Set(“SIP/621-0000b68b”, “TOUCH_MONITOR=1474576481.543350”) in new stack
== Extension Changed 621[ext-local] new state InUse for Notify User 664
== Extension Changed 621[ext-local] new state InUse for Notify User 623
– Executing [[email protected]:2] Set(“SIP/621-0000b68b”, “AMPUSER=621”) in new stack
== Extension Changed 621[ext-local] new state InUse for Notify User 624
– Executing [[email protected]:3] GotoIf(“SIP/621-0000b68b”, “0?report”) in new stack
== Extension Changed 621[ext-local] new state InUse for Notify User 662
– Executing [[email protected]:4] ExecIf(“SIP/621-0000b68b”, “1?Set(REALCALLERIDNUM=621)”) in new stack
== Extension Changed 621[ext-local] new state InUse for Notify User 663
== Extension Changed 621[ext-local] new state InUse for Notify User 621
– Executing [[email protected]:5] Set(“SIP/621-0000b68b”, “AMPUSER=621”) in new stack
== Extension Changed 621[ext-local] new state InUse for Notify User 634
– Executing [[email protected]:6] GotoIf(“SIP/621-0000b68b”, “0?limit”) in new stack
– Executing [[email protected]:7] Set(“SIP/621-0000b68b”, “AMPUSERCIDNAME=J Lando”) in new stack
– Executing [[email protected]:8] GotoIf(“SIP/621-0000b68b”, “0?report”) in new stack
– Executing [[email protected]:9] Set(“SIP/621-0000b68b”, “AMPUSERCID=621”) in new stack
– Executing [[email protected]:10] Set(“SIP/621-0000b68b”, “__DIAL_OPTIONS=Ttr”) in new stack
– Executing [[email protected]:11] Set(“SIP/621-0000b68b”, “CALLERID(all)=“J Lando” <621>”) in new stack
– Executing [[email protected]:12] GotoIf(“SIP/621-0000b68b”, “0?limit”) in new stack
– Executing [[email protected]:13] ExecIf(“SIP/621-0000b68b”, “1?Set(GROUP(concurrency_limit)=621)”) in new stack
– Executing [[email protected]:14] ExecIf(“SIP/621-0000b68b”, “0?Set(CHANNEL(language)=)”) in new stack
– Executing [[email protected]:15] GotoIf(“SIP/621-0000b68b”, “1?continue”) in new stack
– Goto (macro-user-callerid,s,29)
– Executing [[email protected]:29] Set(“SIP/621-0000b68b”, “CALLERID(number)=621”) in new stack
– Executing [[email protected]:30] Set(“SIP/621-0000b68b”, “CALLERID(name)=J Lando”) in new stack
– Executing [[email protected]:31] GotoIf(“SIP/621-0000b68b”, “0?cnum”) in new stack
– Executing [[email protected]:32] Set(“SIP/621-0000b68b”, “CDR(cnam)=J Lando”) in new stack
– Executing [[email protected]:33] Set(“SIP/621-0000b68b”, “CDR(cnum)=621”) in new stack
– Executing [[email protected]:34] Set(“SIP/621-0000b68b”, “CHANNEL(language)=en”) in new stack
– Executing [[email protected]:2] Gosub(“SIP/621-0000b68b”, “sub-record-check,s,1(out,203,force)”) in new stack
– Executing [[email protected]:1] GotoIf(“SIP/621-0000b68b”, “0?initialized”) in new stack
– Executing [[email protected]:2] Set(“SIP/621-0000b68b”, “__REC_STATUS=INITIALIZED”) in new stack
– Executing [[email protected]:3] Set(“SIP/621-0000b68b”, “NOW=1474576481”) in new stack
– Executing [[email protected]:4] Set(“SIP/621-0000b68b”, “__DAY=22”) in new stack
– Executing [[email protected]:5] Set(“SIP/621-0000b68b”, “__MONTH=09”) in new stack
– Executing [[email protected]:6] Set(“SIP/621-0000b68b”, “__YEAR=2016”) in new stack
– Executing [[email protected]:7] Set(“SIP/621-0000b68b”, “__TIMESTR=20160922-153441”) in new stack
– Executing [[email protected]:8] Set(“SIP/621-0000b68b”, “__FROMEXTEN=621”) in new stack
– Executing [[email protected]:9] Set(“SIP/621-0000b68b”, “__MON_FMT=wav”) in new stack
– Executing [[email protected]:10] NoOp(“SIP/621-0000b68b”, “Recordings initialized”) in new stack
– Executing [[email protected]:11] ExecIf(“SIP/621-0000b68b”, “0?Set(ARG3=dontcare)”) in new stack
– Executing [[email protected]:12] Set(“SIP/621-0000b68b”, “REC_POLICY_MODE_SAVE=”) in new stack
– Executing [[email protected]:13] ExecIf(“SIP/621-0000b68b”, “0?Set(REC_STATUS=NO)”) in new stack
– Executing [[email protected]:14] GotoIf(“SIP/621-0000b68b”, “3?checkaction”) in new stack
– Goto (sub-record-check,s,17)
– Executing [[email protected]:17] GotoIf(“SIP/621-0000b68b”, “1?sub-record-check,out,1”) in new stack
– Goto (sub-record-check,out,1)
– Executing [[email protected]:1] NoOp(“SIP/621-0000b68b”, “Outbound Recording Check from 621 to 203”) in new stack
– Executing [[email protected]:2] Set(“SIP/621-0000b68b”, “RECMODE=force”) in new stack
– Executing [[email protected]:3] ExecIf(“SIP/621-0000b68b”, “0?Goto(routewins)”) in new stack
– Executing [[email protected]:4] ExecIf(“SIP/621-0000b68b”, “1?Goto(routewins)”) in new stack
– Goto (sub-record-check,out,7)
– Executing [out[email protected]:7] Gosub(“SIP/621-0000b68b”, “recordcheck,1(force,out,203)”) in new stack
– Executing [[email protected]:1] NoOp(“SIP/621-0000b68b”, “Starting recording check against force”) in new stack
– Executing [[email protected]:2] Goto(“SIP/621-0000b68b”, “force”) in new stack
– Goto (sub-record-check,recordcheck,5)
– Executing [[email protected]:5] Set(“SIP/621-0000b68b”, “__REC_POLICY_MODE=FORCE”) in new stack
– Executing [[email protected]:6] GotoIf(“SIP/621-0000b68b”, “1?startrec”) in new stack
– Goto (sub-record-check,recordcheck,16)
– Executing [[email protected]:16] NoOp(“SIP/621-0000b68b”, “Starting recording: out, 203”) in new stack
– Executing [[email protected]:17] Set(“SIP/621-0000b68b”, “AUDIOHOOK_INHERIT(MixMonitor)=yes”) in new stack
– Executing [[email protected]:18] Set(“SIP/621-0000b68b”, “__CALLFILENAME=out-203-621-20160922-153441-1474576481.543350”) in new stack
– Executing [[email protected]:19] MixMonitor(“SIP/621-0000b68b”, “2016/09/22/out-203-621-20160922-153441-1474576481.543350.wav,ai(LOCAL_MIXMON_ID),”) in new stack
== Begin MixMonitor Recording SIP/621-0000b68b
– Executing [[email protected]:20] Set(“SIP/621-0000b68b”, “__MIXMON_ID=0x7ff474aeb380”) in new stack
– Executing [[email protected]:21] Set(“SIP/621-0000b68b”, “__RECORD_ID=SIP/621-0000b68b”) in new stack
– Executing [[email protected]:22] Set(“SIP/621-0000b68b”, “__REC_STATUS=RECORDING”) in new stack
– Executing [[email protected]:23] Set(“SIP/621-0000b68b”, “CDR(recordingfile)=out-203-621-20160922-153441-1474576481.543350.wav”) in new stack
– Executing [[email protected]:24] Return(“SIP/621-0000b68b”, “”) in new stack
– Executing [[email protected]:8] Return(“SIP/621-0000b68b”, “”) in new stack
– Executing [[email protected]:3] ExecIf(“SIP/621-0000b68b”, “0 ?Set(CDR(accountcode)=)”) in new stack
– Executing [[email protected]:4] Set(“SIP/621-0000b68b”, “INTRACOMPANYROUTE=YES”) in new stack
– Executing [[email protected]:5] Set(“SIP/621-0000b68b”, “MOHCLASS=default”) in new stack
– Executing [[email protected]:6] Set(“SIP/621-0000b68b”, “_NODEST=”) in new stack
– Executing [[email protected]:7] Macro(“SIP/621-0000b68b”, “dialout-trunk,2,203,off”) in new stack
– Executing [[email protected]:1] Set(“SIP/621-0000b68b”, “DIAL_TRUNK=2”) in new stack
– Executing [[email protected]:2] GosubIf(“SIP/621-0000b68b”, “0?sub-pincheck,s,1()”) in new stack
– Executing [[email protected]:3] GotoIf(“SIP/621-0000b68b”, “0?disabletrunk,1”) in new stack
– Executing [[email protected]:4] Set(“SIP/621-0000b68b”, “DIAL_NUMBER=203”) in new stack
– Executing [[email protected]:5] Set(“SIP/621-0000b68b”, “DIAL_TRUNK_OPTIONS=Ttr”) in new stack
– Executing [[email protected]:6] Set(“SIP/621-0000b68b”, “OUTBOUND_GROUP=OUT_2”) in new stack
– Executing [[email protected]:7] GotoIf(“SIP/621-0000b68b”, “1?nomax”) in new stack
– Goto (macro-dialout-trunk,s,9)
– Executing [[email protected]:9] GotoIf(“SIP/621-0000b68b”, “1?skipoutcid”) in new stack
– Goto (macro-dialout-trunk,s,12)
– Executing [[email protected]:12] GosubIf(“SIP/621-0000b68b”, “1?sub-flp-2,s,1()”) in new stack
– Executing [[email protected]:1] ExecIf(“SIP/621-0000b68b”, “1?Return()”) in new stack
– Executing [[email protected]:13] Set(“SIP/621-0000b68b”, “OUTNUM=203”) in new stack
– Executing [[email protected]:14] Set(“SIP/621-0000b68b”, “custom=IAX2/ToCityHall”) in new stack
– Executing [[email protected]:15] ExecIf(“SIP/621-0000b68b”, “0?Set(DIAL_TRUNK_OPTIONS=M(setmusic^default)Ttr)”) in new stack
– Executing [[email protected]:16] ExecIf(“SIP/621-0000b68b”, “0?Set(DIAL_TRUNK_OPTIONS=TtrM(confirm))”) in new stack
– Executing [[email protected]:17] Macro(“SIP/621-0000b68b”, “dialout-trunk-predial-hook,”) in new stack
– Executing [[email protected]:1] MacroExit(“SIP/621-0000b68b”, “”) in new stack
– Executing [[email protected]:18] GotoIf(“SIP/621-0000b68b”, “0?bypass,1”) in new stack
– Executing [[email protected]:19] ExecIf(“SIP/621-0000b68b”, “1?Set(CONNECTEDLINE(num,i)=203)”) in new stack
– Executing [[email protected]:20] ExecIf(“SIP/621-0000b68b”, “1?Set(CONNECTEDLINE(name,i)=CID:621)”) in new stack
– Executing [[email protected]:21] ExecIf(“SIP/621-0000b68b”, “0?Set(CONNECTEDLINE(name,i)=CID:(Hidden)621)”) in new stack
– Executing [[email protected]:22] GotoIf(“SIP/621-0000b68b”, “0?customtrunk”) in new stack
– Executing [[email protected]:23] Dial(“SIP/621-0000b68b”, “IAX2/ToCityHall/203,300,Ttr”) in new stack
– Called IAX2/ToCityHall/203
– Hungup ‘IAX2/ToCityHall-16994’
== Everyone is busy/congested at this time (1:0/0/1)
– Executing [[email protected]:24] NoOp(“SIP/621-0000b68b”, “Dial failed for some reason with DIALSTATUS = CHANUNAVAIL and HANGUPCAUSE = 50”) in new stack
– Executing [[email protected]:25] GotoIf(“SIP/621-0000b68b”, “0?continue,1:s-CHANUNAVAIL,1”) in new stack
– Goto (macro-dialout-trunk,s-CHANUNAVAIL,1)
– Executing [[email protected]:1] Set(“SIP/621-0000b68b”, “RC=50”) in new stack
– Executing [[email protected]:2] Goto(“SIP/621-0000b68b”, “50,1”) in new stack
– Goto (macro-dialout-trunk,50,1)
– Executing [[email protected]:1] Goto(“SIP/621-0000b68b”, “continue,1”) in new stack
– Goto (macro-dialout-trunk,continue,1)
– Executing [[email protected]:1] NoOp(“SIP/621-0000b68b”, “TRUNK Dial failed due to CHANUNAVAIL HANGUPCAUSE: 50 - failing through to other trunks”) in new stack
– Executing [[email protected]:2] ExecIf(“SIP/621-0000b68b”, “1?Set(CALLERID(number)=621)”) in new stack
– Executing [[email protected]:8] Macro(“SIP/621-0000b68b”, “outisbusy,”) in new stack
– Executing [[email protected]:1] Progress(“SIP/621-0000b68b”, “”) in new stack
– Executing [[email protected]:2] GotoIf(“SIP/621-0000b68b”, “0?emergency,1”) in new stack
– Executing [[email protected]:3] GotoIf(“SIP/621-0000b68b”, “1?intracompany,1”) in new stack
– Goto (macro-outisbusy,intracompany,1)
– Executing [[email protected]:1] Playback(“SIP/621-0000b68b”, “all-circuits-busy-now&pls-try-call-later, noanswer”) in new stack
– <SIP/621-0000b68b> Playing ‘all-circuits-busy-now.ulaw’ (language ‘en’)
> 0x7ff48efdc5d0 – Probation passed - setting RTP source address to 10.50.20.33:38080
[2016-09-22 15:34:43] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 683
– <SIP/621-0000b68b> Playing ‘pls-try-call-later.ulaw’ (language ‘en’)
[2016-09-22 15:34:44] NOTICE[21561]: chan_sip.c:28223 handle_request_subscribe: Received SIP subscribe for peer without mailbox: 683
– Executing [[email protected]:2] Congestion(“SIP/621-0000b68b”, “20”) in new stack
[2016-09-22 15:34:46] WARNING[2063][C-0000561f]: channel.c:4883 ast_prod: Prodding channel ‘SIP/621-0000b68b’ failed
== Spawn extension (macro-outisbusy, intracompany, 2) exited non-zero on ‘SIP/621-0000b68b’ in macro ‘outisbusy’
== Spawn extension (from-internal, 203, 8) exited non-zero on ‘SIP/621-0000b68b’
– Executing [[email protected]:1] Macro(“SIP/621-0000b68b”, “hangupcall”) in new stack
– Executing [[email protected]:1] GotoIf(“SIP/621-0000b68b”, “1?theend”) in new stack
– Goto (macro-hangupcall,s,3)
– Executing [[email protected]:3] ExecIf(“SIP/621-0000b68b”, “0?Set(CDR(recordingfile)=)”) in new stack
– Executing [[email protected]:4] Hangup(“SIP/621-0000b68b”, “”) in new stack
== Spawn extension (macro-hangupcall, s, 4) exited non-zero on ‘SIP/621-0000b68b’ in macro ‘hangupcall’
== Spawn extension (from-internal, h, 1) exited non-zero on ‘SIP/621-0000b68b’
== Extension Changed 621[ext-local] new state Idle for Notify User 678
== Extension Changed 621[ext-local] new state Idle for Notify User 639
== MixMonitor close filestream (mixed)
== End MixMonitor Recording SIP/621-0000b68b
== Extension Changed 621[ext-local] new state Idle for Notify User 668
== Extension Changed 621[ext-local] new state Idle for Notify User 622
== Extension Changed 621[ext-local] new state Idle for Notify User 642
== Extension Changed 621[ext-local] new state Idle for Notify User 683
== Extension Changed 621[ext-local] new state Idle for Notify User 660
== Extension Changed 621[ext-local] new state Idle for Notify User 646
== Extension Changed 621[ext-local] new state Idle for Notify User 643
== Extension Changed 621[ext-local] new state Idle for Notify User 664
== Extension Changed 621[ext-local] new state Idle for Notify User 623
== Extension Changed 621[ext-local] new state Idle for Notify User 624
== Extension Changed 621[ext-local] new state Idle for Notify User 662
== Extension Changed 621[ext-local] new state Idle for Notify User 663
== Extension Changed 621[ext-local] new state Idle for Notify User 621
== Extension Changed 621[ext-local] new state Idle for Notify User 634

all passwords on trunks are correct all settings are the same.

I have check outbound routes and trunk routes with no luck

I figured it out it was the incoming user settings was not matching the remote end