Clicking on apply causes pjsip/sip-tls endpoints to drop

hi, I am currently running FreePBX 14.0.1.1 and have been for about a month. initially when I did the v14 install, I did not have this issue, but at some point about 2 weeks ago, I noticed that when I made changes (any changes, it does not matter what) and then hit the apply button, not only did asterisk reload (as expected) but all my tls connected endpoints dropped. I am using pjsip with sip-tls and srtp. I do have a couple of endpoints (pjsip) that do not support encryption and they do NOT disconnect (UDP). When the endpoints drop, it takes a while for them to re-connect, sometimes as long as 30 minutes, so I now have to batch any changes for nightime (really inconvenient).
To narrow down the issue, rather than clicking on apply, I tried using the ‘core reload’ command in asterisk. I noticed that core reload does NOT result in my endpoints dropping. I was wondering if the ‘apply’ button does more than just a ‘core reload’? is it possible that it is restarting the firewall? or something else network related that might be messing with the tcp connections?
I would really appreciate any help with this, this is a real problem for me.

i have attached logs of a core reload that does not drop connections, an apply that does drop connections and a modules version list
thanks
reload issue.tgz (1.3 MB)

Apply changes only does a core reload.

andrew, can you think of a away that core reload on the asterisk command line would differ from the core reload that you do when i hit apply? btw, this is not sporadic, it happens on every change no mater what the change is.

On the other hand, we have seen a report of something like this from a few weeks ago. The problem he was having was that Asterisk was restarting on the reload because of an error in one of his config files IIRC.

Check your logs and see if your system is actually restarting on the reload - that will help you get to the bottom of the problem.

Dave, thanks for your input on this. i have looked through the logs and to be honest, i am not sure what i am looking for…but i did see this. i think it might be important because it references the tls interface i am using…what do you think? also, how can i tell from the log if asterisk reloaded or restarted

[2017-07-17 22:06:11] ERROR[20786] res_pjsip/config_transport.c: Transport ‘172.30.200.1-tls’ could not be started: Object with the same type exists (PJSIP_ETYPEEXISTS)
[2017-07-17 22:06:11] ERROR[20786] res_sorcery_config.c: Could not create an object of type ‘transport’ with id ‘172.30.200.1-tls’ from configuration file ‘pjsip.conf’

fyi, i have to apply changes tonight, is there anything that i could gather while i do this to further narrow down the issue?

thanks for all your help!

Looks like an asterisk bug but I would have to see your pjsip settings in sip settings to know for sure.

Andrew, i opened a ticket there and they closed it. they say that if ‘core reload’ from the command line does not cause the issue, it is not their problem.

i would be happy to share my pjsip config, what do you need?

we do the same thing we run asterisk -rx ‘core reload’

Same. Thing.

The difference is that configs are written out. So maybe you should compare configs before a reload and after a reload.

i don’t understand it. the only difference is i go int asterisk -vr and they type core reload there…

is there a file for the pjsip config, dave seem to indicated that someone else had a similar issue that was due to issues in the config files

You should diff /etc/asterisk before and after a reload.

andrew, thanks, i just tried it again, first i made a copy of /etc/asterisk. then i did a asterisk -xr ‘core reload’ and that did not drop any connections. then i did an apply button and that did cause connections to drop.

i did a diff and there were differences, but i don’t know what that means, here they are, is this significant?

diff /etc/asterisk/ari_additional.conf /tmp/etcasterisk/ari_additional.conf
> 12c12
> < password=$6$xwjGh0FNguMJfGi.$FzScmFLIbLcZeRkV8F2yPIuKbVLGgb1YkQvGX8S0hufo2WOAVj9TQZrsRhhCAN/GMHc50AYbZva5.kV/9BZIl1
> ---
> > password=$6$pHw3ZmdNNvSnctC0$Q/XW/lYEqclIfcbQoM73kFjZBAINJj9E4cQVLqm5DfloC/Qvj091lh1KFZgqU6EMMLu0A7T9A.OwM/C389Bd31
> diff /etc/asterisk/extensions_additional.conf /tmp/etcasterisk/extensions_additional.conf
> 3901a3902,3903
> > exten => 5385,1,GotoIf($[${DB_EXISTS(AMPUSER/${EXTEN}/followme/ddial)} != 1 | "${DB(AMPUSER/${EXTEN}/followme/ddial)}" = "EXTENSION"]?ext-local,${EXTEN},1:followme-check,${EXTEN},1)
> >
> 3930,3931d3931
> < exten => 5385,1,GotoIf($[${DB_EXISTS(AMPUSER/${EXTEN}/followme/ddial)} != 1 | "${DB(AMPUSER/${EXTEN}/followme/ddial)}" = "EXTENSION"]?ext-local,${EXTEN},1:followme-check,${EXTEN},1)
> <
> 4096a4097,4098
> > exten => _RG-5385.,1+1,Macro(dial,${DB(AMPUSER/5385/followme/grptime)},${DIAL_OPTIONS}M(confirm^^^5385),${EXTEN:8})
> >
> 4124,4125d4125
> <
> < exten => _RG-5385.,1+1,Macro(dial,${DB(AMPUSER/5385/followme/grptime)},${DIAL_OPTIONS}M(confirm^^^5385),${EXTEN:8})
> Common subdirectories: /etc/asterisk/keys and /tmp/etcasterisk/keys
> diff /etc/asterisk/pjsip.aor.conf /tmp/etcasterisk/pjsip.aor.conf
> 597a598,604
> > [5385]
> > type=aor
> > [email protected]
> > max_contacts=1
> > remove_existing=yes
> > qualify_frequency=60
> >
> 702,710d708
> < maximum_expiration=7200
> < minimum_expiration=60
> < qualify_frequency=60
> <
> < [5385]
> < type=aor
> < [email protected]
> < max_contacts=1
> < remove_existing=yes
> diff /etc/asterisk/pjsip.auth.conf /tmp/etcasterisk/pjsip.auth.conf
> 496a497,502
> > [5385-auth]
> > type=auth
> > auth_type=userpass
> > password=b504eacca8059168148f862c69b73063
> > username=5385
> >
> 580,585d585
> <
> < [5385-auth]
> < type=auth
> < auth_type=userpass
> < password=b504eacca8059168148f862c69b73063
> < username=5385
> diff /etc/asterisk/pjsip.endpoint.conf /tmp/etcasterisk/pjsip.endpoint.conf
> 2200a2201,2227
> > [5385]
> > type=endpoint
> > aors=5385
> > auth=5385-auth
> > allow=ulaw,alaw,gsm,g726
> > context=from-internal
> > callerid=Cafeteria Conference <5385>
> > dtmf_mode=rfc4733
> > [email protected]
> > mwi_subscribe_replaces_unsolicited=yes
> > transport=172.30.2.1-tls
> > aggregate_mwi=yes
> > use_avpf=no
> > rtcp_mux=no
> > ice_support=no
> > media_use_received_transport=no
> > trust_id_inbound=yes
> > media_encryption=sdes
> > timers=yes
> > media_encryption_optimistic=yes
> > rtp_symmetric=yes
> > rewrite_contact=yes
> > force_rport=yes
> > media_address=
> > bind_rtp_to_media_address=yes
> > language=en
> >
> 2574,2601d2600
> < timers=yes
> < media_encryption_optimistic=yes
> < send_pai=yes
> < rtp_symmetric=yes
> < rewrite_contact=yes
> < force_rport=yes
> < media_address=
> < bind_rtp_to_media_address=yes
> < language=en
> <
> < [5385]
> < type=endpoint
> < aors=5385
> < auth=5385-auth
> < allow=ulaw,alaw,gsm,g726
> < context=from-internal
> < callerid=Cafeteria Conference <5385>
> < dtmf_mode=rfc4733
> < [email protected]
> < mwi_subscribe_replaces_unsolicited=yes
> < transport=172.30.2.1-udp
> < aggregate_mwi=yes
> < use_avpf=no
> < rtcp_mux=no
> < ice_support=no
> < media_use_received_transport=no
> < trust_id_inbound=yes
> < media_encryption=no
> diff /etc/asterisk/pjsip.identify.conf /tmp/etcasterisk/pjsip.identify.conf
> 334a335,338
> > [5385-identify]
> > type=identify
> > endpoint=5385
> >
> 390,393d393
> <
> < [5385-identify]
> < type=identify
> < endpoint=5385

You added an extension between tries “5385”

That makes it really difficult to see whats going on.

does fwconsole reload make it crash? if so then do the diff again but please dont change anything but use fwconsole reload.

andrew , fwconsole reload DOES drop the connections. i made no changes and this was the diff

diff /etc/asterisk/ari_additional.conf /tmp/etcasterisk2/ari_additional.conf
12c12
< password=$6$iqaXyMFT3kTtxCwY$8G/Vy7nAvxKlcU1iAAsvdvG73jxBHc0nt10HikRPMpLFroub7kbRxb/mOiCrE0Gt5LR2yBz7u0NADkgqREi.y/

password=$6$xwjGh0FNguMJfGi.$FzScmFLIbLcZeRkV8F2yPIuKbVLGgb1YkQvGX8S0hufo2WOAVj9TQZrsRhhCAN/GMHc50AYbZva5.kV/9BZIl1
Common subdirectories: /etc/asterisk/keys and /tmp/etcasterisk2/keys

Nothing is changing from FreePBX (you are seeing only a change in ARI for generating ARI passwords)

I can’t say this is freepbx at all.

you can also try

/var/lib/asterisk/bin/retrieve_conf
asterisk -rx 'core reload'

edit delete-no dropped connections

[[email protected] asterisk]# /var/lib/asterisk/bin/retrieve_conf
Please update your modules and reload Asterisk by browsing to your server.
[[email protected] asterisk]# asterisk -rx ‘core reload’
[[email protected] asterisk]#

edit: wait, sorry, i jumped the gun
i am dropping connections

Are you dropping connections with just: asterisk -rx ‘core reload’

if the answer is no then do fwconsole ma disable arimanager

then reload and you will notice no files are changing.

andrew, i asterisk -rx ‘core reload’ does not drop connections, i just verified again.

i disabled arimanager and then did a fwconsole reload, and connections still dropped

[[email protected] ~]# fwconsole ma disable arimanager
Module arimanager successfully disabled
Updating Hooks…Done
[[email protected] ~]# fwconsole reload

Are any files changing in /etc/asterisk ?

i just recreated the test again, fwconsole reload caused the connections to drop.

/etc/asterisk had no changes