Some extensions can't register after IP change

FreePBX: 6.12.65-22 from AsteriskNOW distro

I’ve run into an extremely strange issue with one of my new servers. We are working on implementing 2 phone systems. 1 of them is 100% implemented and has zero issues. The other one was absolutely perfect until this last week.

The problem system was recently moved into its own brand new network behind a firewall. It is going to be going into a brand new office that has a PRI and new fiber internet connection. We will be moving into a temporary office space for a month or two while construction is complete in our new place. This is where the problems begin. Our network is the 10.0.1.0/24 network, and the new network for the phone system is 10.0.3.0/24. Our working office is the 10.0.2.0/24 network. My home network is 192.168.1.0/24. They are connected together via VPN connections via our firewalls.

The problem all started when I got a new firewall for the 10.0.3.0 network. I got the VPN set up, then moved the phone system into that network. I was able to get our two asterisk systems connected via IAX, all extensions were connected on the LAN, and I even tested a few extensions from the 10.0.2.0 network. Everything worked great. Then, after the move, I came home and noticed only 1 of my two extensions on my home desk phone was properly provisioned. After literally hours of troubleshooting, only extension 114 would register, but 144 would not. However, even though the 114 extension was registered, when I would do *43 for an echo test, the system would answer, but it would not hear me at all. It could not hear DTMF nor my voice. I also noticed that the phone system would not receive the SIP disconnect when I would hang up the phone, which would cause further log entries. I swore I had a routing/VPN issue somewhere, so I started tearing things apart in the firewalls. I got to a point where I had phones on the 10.0.1.0 network no longer registering. I also noticed our IAX trunk would no longer work in one direction. Calls from the working system would come in, but calls from this broken system would not go out to the working system. I got fed up and started moving phones directly onto the 10.0.3.0 LAN to try and eliminate the firewall as a problem. This is where things get weird.

When I started putting phones on the 10.0.3.0 LAN, they would register, but when I would make calls, the phone would say “NO RESPONSE” (Grandstream 2130 & 2160) immediately after I would hit the send key on the phone. I also noticed that several extensions seemed to not know that they had been registered by the system, because in the asterisk/full log I would see this:

[2015-01-11 22:42:06] VERBOSE[3460] res_pjsip_registrar.c: Added contact ‘sip:[email protected]:5060’ to AOR ‘112’ with expiration of 3600 seconds
[2015-01-11 22:42:08] VERBOSE[2393] res_pjsip_registrar.c: Added contact ‘sip:[email protected]:5060’ to AOR ‘112’ with expiration of 3600 seconds
[2015-01-11 22:42:10] VERBOSE[2393] res_pjsip_registrar.c: Added contact ‘sip:[email protected]:5060’ to AOR ‘112’ with expiration of 3600 seconds
[2015-01-11 22:42:12] VERBOSE[2393] res_pjsip_registrar.c: Added contact ‘sip:[email protected]:5060’ to AOR ‘112’ with expiration of 3600 seconds
[2015-01-11 22:42:14] VERBOSE[2393] res_pjsip_registrar.c: Added contact ‘sip:[email protected]:5060’ to AOR ‘112’ with expiration of 3600 seconds
[2015-01-11 22:42:16] VERBOSE[2393] res_pjsip_registrar.c: Added contact ‘sip:[email protected]:5060’ to AOR ‘112’ with expiration of 3600 seconds

As you can see, every 2 seconds these extensions with issues are trying to register, but it’s as if the phone doesn’t know it, and tries again 2 seconds later. These phones will no longer show that they have registered, and therefore cannot make calls.

Here’s what else I have tried to fix this issue:
-Stopped fail2ban and iptables
-Checked for static routes on the linux machine itself
-Double-checked the Asterisk SIP NAT settings (External Address and local networks)

Finally, I decided to start from scratch. I wiped and reinstalled the entire phone system to see if this was something specific to Asterisk config. Sure enough, the problem was fixed. I registered a phone on the 10.0.3.0 LAN and it worked along with a phone on each VPN, which also worked. Didn’t notice any issues. IAX connection also worked between systems. So, now I KNOW it’s something to do with Asterisk/FreePBX config.

Next I decided to restore the full backup I had from before I wiped clean. That brought the issues back, but they are totally different. Now I can register from my home and make calls, but my home phone repeats the “Added contact to AOR” message in the log. Phones on the 10.0.3.0 LAN absolutely refuse to register.

Next I took a full backup from a clean system I set up, and restored it. It did NOT fix the issue.

I’m prepared to just wipe and start over, as I am putting this system into production starting on Jan 19th. HOWEVER, I’m not content with the idea that I don’t know what is going on here. I’m missing something critical, and I don’t know what.

The only thing I can think is that there’s some sort of blocker, router, etc. that is causing traffic to be routed in a strange way, and it has something to do with the change of the system IP from 10.0.1.10 to 10.0.3.10 when I moved it to its own LAN. This 10.0.3.0 LAN is only temporary until we move into our new building. I will then switch its IP back from 10.0.3.10 to 10.0.1.10. I’ve grepped to try and find files with 10.0.1.10, but I find nothing except old log entries.

I’m at a total loss and am ready to pay for support, or just move on with a clean install. But I’m afraid that when it comes time to switch the IP, this problem will pop up again, and I’ll be stuck with a production system with this issue, and I simply can’t have that.

Sorry for the novel, but I like to be very thorough. Anyone have any idea WTH I’m doing wrong?

I just wiped the machine and started over from scratch… again. I restored ONLY the mysql database, and that brought the problem back again, so whatever is happening is definitely in the asterisk db.

Just restored the mysql db to what it was before I restored my backup db. Did not fix the issue, even with a system reboot.

Looking at the db, the only table that has a reference to 10.0.1.10 is kvstore. Not sure what that table is for.

I will try wiping the machine again, and restoring only necessary tables (extensions, routes, trunks, ivr, etc.), so we’ll see how that goes.

The solution here was relatively simple.

I had our second phone system suddenly start doing this. Decided to call support, and after a bit of troubleshooting it was determined that we had a couple settings wrong.

First, in Settings > Asterisk SIP Settings, you must make sure that Local Networks are set correctly for any traffic that should not go back through the server’s default gateway. I did have these correct, in my case.

Second, in Settings > Asterisk SIP Settings > CHAN PJSIP, make sure that Local Network is set to the subnet of the server running asterisk.

Then, for good measure, running “amportal restart”. After double-checking all this, it has solved registration problems on both servers. It also solved all one-way audio issues.