Clicking on apply causes pjsip/sip-tls endpoints to drop

tonyg · July 18, 2017, 12:20pm

hi, I am currently running FreePBX 14.0.1.1 and have been for about a month. initially when I did the v14 install, I did not have this issue, but at some point about 2 weeks ago, I noticed that when I made changes (any changes, it does not matter what) and then hit the apply button, not only did asterisk reload (as expected) but all my tls connected endpoints dropped. I am using pjsip with sip-tls and srtp. I do have a couple of endpoints (pjsip) that do not support encryption and they do NOT disconnect (UDP). When the endpoints drop, it takes a while for them to re-connect, sometimes as long as 30 minutes, so I now have to batch any changes for nightime (really inconvenient).
To narrow down the issue, rather than clicking on apply, I tried using the ‘core reload’ command in asterisk. I noticed that core reload does NOT result in my endpoints dropping. I was wondering if the ‘apply’ button does more than just a ‘core reload’? is it possible that it is restarting the firewall? or something else network related that might be messing with the tcp connections?
I would really appreciate any help with this, this is a real problem for me.

i have attached logs of a core reload that does not drop connections, an apply that does drop connections and a modules version list
thanks
reload issue.tgz (1.3 MB)

tm1000 · July 18, 2017, 1:46pm

Apply changes only does a core reload.

tonyg · July 18, 2017, 2:16pm

andrew, can you think of a away that core reload on the asterisk command line would differ from the core reload that you do when i hit apply? btw, this is not sporadic, it happens on every change no mater what the change is.

cynjut · July 18, 2017, 2:17pm

On the other hand, we have seen a report of something like this from a few weeks ago. The problem he was having was that Asterisk was restarting on the reload because of an error in one of his config files IIRC.

Check your logs and see if your system is actually restarting on the reload - that will help you get to the bottom of the problem.

tonyg · July 18, 2017, 2:26pm

Dave, thanks for your input on this. i have looked through the logs and to be honest, i am not sure what i am looking for…but i did see this. i think it might be important because it references the tls interface i am using…what do you think? also, how can i tell from the log if asterisk reloaded or restarted

[2017-07-17 22:06:11] ERROR[20786] res_pjsip/config_transport.c: Transport ‘172.30.200.1-tls’ could not be started: Object with the same type exists (PJSIP_ETYPEEXISTS)
[2017-07-17 22:06:11] ERROR[20786] res_sorcery_config.c: Could not create an object of type ‘transport’ with id ‘172.30.200.1-tls’ from configuration file ‘pjsip.conf’

tonyg · July 18, 2017, 11:20pm

fyi, i have to apply changes tonight, is there anything that i could gather while i do this to further narrow down the issue?

thanks for all your help!

tm1000 · July 18, 2017, 11:28pm

Looks like an asterisk bug but I would have to see your pjsip settings in sip settings to know for sure.

tonyg · July 18, 2017, 11:33pm

Andrew, i opened a ticket there and they closed it. they say that if ‘core reload’ from the command line does not cause the issue, it is not their problem.

i would be happy to share my pjsip config, what do you need?

tm1000 · July 18, 2017, 11:35pm

we do the same thing we run asterisk -rx ‘core reload’

Same. Thing.

github.com

FreePBX/framework/blob/release/13.0/amp_conf/htdocs/admin/libraries/php-asmanager.php#L1706


      
          			$keyUsed="/".str_replace(" ","/",$family)."/".str_replace(" ","/",$key);
          			if (isset($this->memAstDB[$keyUsed])){
          				return $this->memAstDB[$keyUsed];
          			}
          		} else {
          			$r = $this->command("database get ".str_replace(" ","/",$family)." ".str_replace(" ","/",$key));
          			$data = strpos($r["data"],"Value:");
          			if ($data !== false) {
          				return trim(substr($r["data"],6+$data));
          			}
          		}
          		return false;
          	}
          
          
          	/** Delete an entry from the asterisk database
          	 * @param string $family	The family name to use
          	 * @param string $key		The key name to use
          	 * @return bool True if successful
          	 */
          	function database_del($family, $key) {

The difference is that configs are written out. So maybe you should compare configs before a reload and after a reload.

tonyg · July 18, 2017, 11:39pm

i don’t understand it. the only difference is i go int asterisk -vr and they type core reload there…

is there a file for the pjsip config, dave seem to indicated that someone else had a similar issue that was due to issues in the config files

tm1000 · July 18, 2017, 11:40pm

You should diff /etc/asterisk before and after a reload.

tonyg · July 18, 2017, 11:56pm

andrew, thanks, i just tried it again, first i made a copy of /etc/asterisk. then i did a asterisk -xr ‘core reload’ and that did not drop any connections. then i did an apply button and that did cause connections to drop.

i did a diff and there were differences, but i don’t know what that means, here they are, is this significant?

diff /etc/asterisk/ari_additional.conf /tmp/etcasterisk/ari_additional.conf
> 12c12
> < password=$6$xwjGh0FNguMJfGi.$FzScmFLIbLcZeRkV8F2yPIuKbVLGgb1YkQvGX8S0hufo2WOAVj9TQZrsRhhCAN/GMHc50AYbZva5.kV/9BZIl1
> ---
> > password=$6$pHw3ZmdNNvSnctC0$Q/XW/lYEqclIfcbQoM73kFjZBAINJj9E4cQVLqm5DfloC/Qvj091lh1KFZgqU6EMMLu0A7T9A.OwM/C389Bd31
> diff /etc/asterisk/extensions_additional.conf /tmp/etcasterisk/extensions_additional.conf
> 3901a3902,3903
> > exten => 5385,1,GotoIf($[${DB_EXISTS(AMPUSER/${EXTEN}/followme/ddial)} != 1 | "${DB(AMPUSER/${EXTEN}/followme/ddial)}" = "EXTENSION"]?ext-local,${EXTEN},1:followme-check,${EXTEN},1)
> >
> 3930,3931d3931
> < exten => 5385,1,GotoIf($[${DB_EXISTS(AMPUSER/${EXTEN}/followme/ddial)} != 1 | "${DB(AMPUSER/${EXTEN}/followme/ddial)}" = "EXTENSION"]?ext-local,${EXTEN},1:followme-check,${EXTEN},1)
> <
> 4096a4097,4098
> > exten => _RG-5385.,1+1,Macro(dial,${DB(AMPUSER/5385/followme/grptime)},${DIAL_OPTIONS}M(confirm^^^5385),${EXTEN:8})
> >
> 4124,4125d4125
> <
> < exten => _RG-5385.,1+1,Macro(dial,${DB(AMPUSER/5385/followme/grptime)},${DIAL_OPTIONS}M(confirm^^^5385),${EXTEN:8})
> Common subdirectories: /etc/asterisk/keys and /tmp/etcasterisk/keys
> diff /etc/asterisk/pjsip.aor.conf /tmp/etcasterisk/pjsip.aor.conf
> 597a598,604
> > [5385]
> > type=aor
> > mailboxes=5385@device
> > max_contacts=1
> > remove_existing=yes
> > qualify_frequency=60
> >
> 702,710d708
> < maximum_expiration=7200
> < minimum_expiration=60
> < qualify_frequency=60
> <
> < [5385]
> < type=aor
> < mailboxes=5385@device
> < max_contacts=1
> < remove_existing=yes
> diff /etc/asterisk/pjsip.auth.conf /tmp/etcasterisk/pjsip.auth.conf
> 496a497,502
> > [5385-auth]
> > type=auth
> > auth_type=userpass
> > password=b504eacca8059168148f862c69b73063
> > username=5385
> >
> 580,585d585
> <
> < [5385-auth]
> < type=auth
> < auth_type=userpass
> < password=b504eacca8059168148f862c69b73063
> < username=5385
> diff /etc/asterisk/pjsip.endpoint.conf /tmp/etcasterisk/pjsip.endpoint.conf
> 2200a2201,2227
> > [5385]
> > type=endpoint
> > aors=5385
> > auth=5385-auth
> > allow=ulaw,alaw,gsm,g726
> > context=from-internal
> > callerid=Cafeteria Conference <5385>
> > dtmf_mode=rfc4733
> > mailboxes=5385@device
> > mwi_subscribe_replaces_unsolicited=yes
> > transport=172.30.2.1-tls
> > aggregate_mwi=yes
> > use_avpf=no
> > rtcp_mux=no
> > ice_support=no
> > media_use_received_transport=no
> > trust_id_inbound=yes
> > media_encryption=sdes
> > timers=yes
> > media_encryption_optimistic=yes
> > rtp_symmetric=yes
> > rewrite_contact=yes
> > force_rport=yes
> > media_address=
> > bind_rtp_to_media_address=yes
> > language=en
> >
> 2574,2601d2600
> < timers=yes
> < media_encryption_optimistic=yes
> < send_pai=yes
> < rtp_symmetric=yes
> < rewrite_contact=yes
> < force_rport=yes
> < media_address=
> < bind_rtp_to_media_address=yes
> < language=en
> <
> < [5385]
> < type=endpoint
> < aors=5385
> < auth=5385-auth
> < allow=ulaw,alaw,gsm,g726
> < context=from-internal
> < callerid=Cafeteria Conference <5385>
> < dtmf_mode=rfc4733
> < mailboxes=5385@device
> < mwi_subscribe_replaces_unsolicited=yes
> < transport=172.30.2.1-udp
> < aggregate_mwi=yes
> < use_avpf=no
> < rtcp_mux=no
> < ice_support=no
> < media_use_received_transport=no
> < trust_id_inbound=yes
> < media_encryption=no
> diff /etc/asterisk/pjsip.identify.conf /tmp/etcasterisk/pjsip.identify.conf
> 334a335,338
> > [5385-identify]
> > type=identify
> > endpoint=5385
> >
> 390,393d393
> <
> < [5385-identify]
> < type=identify
> < endpoint=5385

tm1000 · July 18, 2017, 11:59pm

You added an extension between tries “5385”

That makes it really difficult to see whats going on.

does fwconsole reload make it crash? if so then do the diff again but please dont change anything but use fwconsole reload.

tonyg · July 19, 2017, 12:03am

andrew , fwconsole reload DOES drop the connections. i made no changes and this was the diff

diff /etc/asterisk/ari_additional.conf /tmp/etcasterisk2/ari_additional.conf
12c12
< password=$6$iqaXyMFT3kTtxCwY$8G/Vy7nAvxKlcU1iAAsvdvG73jxBHc0nt10HikRPMpLFroub7kbRxb/mOiCrE0Gt5LR2yBz7u0NADkgqREi.y/

password=$6$xwjGh0FNguMJfGi.$FzScmFLIbLcZeRkV8F2yPIuKbVLGgb1YkQvGX8S0hufo2WOAVj9TQZrsRhhCAN/GMHc50AYbZva5.kV/9BZIl1
Common subdirectories: /etc/asterisk/keys and /tmp/etcasterisk2/keys

tm1000 · July 19, 2017, 12:07am

Nothing is changing from FreePBX (you are seeing only a change in ARI for generating ARI passwords)

I can’t say this is freepbx at all.

you can also try

/var/lib/asterisk/bin/retrieve_conf
asterisk -rx 'core reload'

tonyg · July 19, 2017, 12:11am

edit delete-no dropped connections

[root@uepbx1 asterisk]# /var/lib/asterisk/bin/retrieve_conf
Please update your modules and reload Asterisk by browsing to your server.
[root@uepbx1 asterisk]# asterisk -rx ‘core reload’
[root@uepbx1 asterisk]#

edit: wait, sorry, i jumped the gun
i am dropping connections

tm1000 · July 19, 2017, 12:33am

Are you dropping connections with just: asterisk -rx ‘core reload’

if the answer is no then do fwconsole ma disable arimanager

then reload and you will notice no files are changing.

tonyg · July 19, 2017, 1:26am

andrew, i asterisk -rx ‘core reload’ does not drop connections, i just verified again.

i disabled arimanager and then did a fwconsole reload, and connections still dropped

[root@uepbx1 ~]# fwconsole ma disable arimanager
Module arimanager successfully disabled
Updating Hooks…Done
[root@uepbx1 ~]# fwconsole reload

tm1000 · July 19, 2017, 1:28am

Are any files changing in /etc/asterisk ?

tonyg · July 19, 2017, 1:55am

i just recreated the test again, fwconsole reload caused the connections to drop.

/etc/asterisk had no changes

Clicking on apply causes pjsip/sip-tls endpoints to drop

diff /etc/asterisk/ari_additional.conf /tmp/etcasterisk2/ari_additional.conf 12c12 < password=$6$iqaXyMFT3kTtxCwY$8G/Vy7nAvxKlcU1iAAsvdvG73jxBHc0nt10HikRPMpLFroub7kbRxb/mOiCrE0Gt5LR2yBz7u0NADkgqREi.y/

diff /etc/asterisk/ari_additional.conf /tmp/etcasterisk2/ari_additional.conf
12c12
< password=$6$iqaXyMFT3kTtxCwY$8G/Vy7nAvxKlcU1iAAsvdvG73jxBHc0nt10HikRPMpLFroub7kbRxb/mOiCrE0Gt5LR2yBz7u0NADkgqREi.y/