Incoming calls dropping after about 10 seconds

I’m having an issue with incoming calls dropping after about 10 seconds. When I run debug for one of the phone in Asterisk’s console I get these 2 warrnings.

[2015-04-21 16:04:43] WARNING[1999]: chan_sip.c:4164 retrans_pkt: Retransmission timeout reached on transmission [email protected] for seqno 501 (Critical Response)
Packet timed out after 6401ms with no response
[2015-04-21 16:04:43] WARNING[1999]: chan_sip.c:4193 retrans_pkt: Hanging up call [email protected]

I’ve tried looking this up with Warning 1999 and seqno 501 but I’m not finding any references to these specific messages. So I’m not sure if I should be looking at the Asterisk or our Router config. Any help with this would be appreciated.