APACHE fails to run after every power cycle

We’ve run into an error every time we power cycle our pbxact server. It reboots properly but apache fails to run every time. As a result, we can’t access the web gui.

The reason for the failure appears to be that “haproxy” is running on the same port as “sangoma phone desktop client service.” In order to get apache to run, I have to SSH into the server as root and run “sudo systemctl stop haproxy” then “sudo systemctl restart httpd.” That works, but haproxy is no longer running.

What is odd to me is that after this happened the first time, I went in and changed the port on the Sangoma phone desktop client service from 6443 to 2443. I figured that would resolve the conflict. However, the next time I power cycled it, the same issue reoccurred, except now when I look at the haproxy.cfg file it has somehow updated to 2443 on it’s own instead of 6443. (see below).

Is there a way to permanently resolve this conflict on bootup?

I’m tempted to change the port in the haproxy.cfg file but I’m not sure the consequences of making a manual change to that.

Automatically Generated File - 1703369695

Do NOT edit this file as it is auto-generated

#---------------------------------------------------------------------
#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global

to have these messages end up in /var/log/haproxy.log you will

need to:

1) configure syslog to accept network log events. This is done

by adding the ‘-r’ option to the SYSLOGD_OPTIONS in

/etc/sysconfig/syslog

2) configure local2 events to go to the /var/log/haproxy.log

file. A line like the following can be added to

/etc/sysconfig/syslog

local2.* /var/log/haproxy.log

tune.ssl.default-dh-param 2048
log /dev/log local0 debug #127.0.0.1 local2

chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 25000
user haproxy
group haproxy
daemon
nbproc 1
nbthread 2
cpu-map auto:1/1-2 0-1
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11

turn on stats unix socket

stats socket /var/lib/haproxy/stats mode 777

#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout tunnel 1h
timeout check 10s
maxconn 25000
option log-health-checks

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------

frontend wss_frontend
bind *:2443 ssl crt /etc/httpd/pki/webserver.pem
http-request add-header X-Real-IP %[src]
maxconn 25000

acl main_path path /main && query -i -m sub showPasswordForm=true
acl is_sangoma hdr_sub(user-agent) -i Sangoma

http-request deny unless is_sangoma

http-request replace-header Host ([^:]+)  if main_path
http-request redirect code 301 location https://%[hdr(host)]:3443/?action=reset if main_path

use_backend chat if { path /chatul } || { path_beg /chatul/ }
use_backend scd_downloads if { path /scd_downloads } || { path_beg /scd_downloads/ }
use_backend scd_contacts if { path /scd_contacts } || { path_beg /scd_contacts/ }
use_backend scd_queues if { path /scd_queues } || { path_beg /scd_queues/ }
use_backend rtapiws if { path /rtapiws } || { path_beg /rtapiws/ }
use_backend sipws if { path /sipws } || { path_beg /sipws/ }

acl main_path path /main && query -i -m sub showPasswordForm=true
acl is_scd_downloads path_beg /scd_downloads
acl is_scd_contacts path_beg /scd_contacts
acl is_scd_queues path_beg /scd_queues
acl is_rtapiws path_beg /rtapiws
acl is_sipws path_beg /sipws
acl is_main path_beg /main

backend chat
balance source
server chat 127.0.0.1:6086 check

backend scd_downloads
balance source
server scd_downloads 127.0.0.1:6002 check

backend scd_contacts
balance source
server scd_contacts 127.0.0.1:6002 check

backend scd_queues
balance source
server scd_queues 127.0.0.1:6002 check

backend rtapiws
balance source
server rtapiws 127.0.0.1:6006 check

backend sipws
balance source
server sipws 127.0.0.1:6006 check

You can have only one server running on any one port, and
haproxy is an excellent choice to service http, https, or any tls as a proxy . How did you get so ‘half assed’ here?
haproxy takes more than a passing understanding but if well implemented increases your ‘security’ by orders of magnitude , look to implementing ‘strict sni’ and map service to url , your webserver will then need to serve (without tls) each webroot on each internal port, map them to any frontend you care to register, certs should be handled by haproxy :slight_smile:

Thank you for reviewing this, dicko. It’s weird, because I seemingly haven’t configured a single port manually. The only change I make that seems to change the apache + haproxy port is in system admin > port management where I set the “HTTPS port”. Whenever I make a change in system admin > portmanagement, it changes the port for both apache and haproxy to the same port, and that is what causes the conflict during bootup where only haproxy runs and apache is unable to.

Is there a module in pbxact that is dictating this? the haproxy.cfg file does say it is auto generated but something in the gui here is changing it…

EDIT: I should mention I used “Let’sEncrypt” to create the cert using tcp port 80.

Did you also change the listening address of the backend services ?

No, I haven’t changed anything yet, since I don’t want to create a bigger mess than I already have. I have the Sangoma Phone Desktop Client Service pointed at 6443 and haproxy is not running.

Do you recommend I change the listening address on the haproxy.cfg file to an available port? Or should I change it in the httpd? I guess I could try each to see if it works, but I have a feeling these two are tied together based on the fact that they change in tandem whenever I adjust the Sangoma Phone Desktop Client Service port.

I am loathe to recommend anything for pbxact but your haproxy front-end is listening on all addresses port 2443 and proxying connections to ‘local’ services on 127.0.0.1 and various ports , so you can use ss -nt to show listening ports and addresses with and without haproxy running.

I suggest though that you revert all your changes or restore to a working backup.

EDIT: We were using 2443 in my original post, but I moved it back to 6443 before running your recommended test below.

I have before and I think setting up the Sangoma Phone Desktop client service is what causes this conflict.

this is with haproxy NOT running:

[root@pbx ~]# sudo ss -ntlp | grep 6443
LISTEN 0 128 [::]:6443 [::]:* users:((“httpd”,pid=14757,fd=26),(“httpd”,pid=14756,fd=26),(“httpd”,pid=14755,fd=26),(“httpd”,pid=14754,fd=26),(“httpd”,pid=14749,fd=26),(“httpd”,pid=14748,fd=26),(“httpd”,pid=14643,fd=26),(“httpd”,pid=14634,fd=26),(“httpd”,pid=14551,fd=26),(“httpd”,pid=14550,fd=26),(“httpd”,pid=14549,fd=26),(“httpd”,pid=14547,fd=26),(“httpd”,pid=14546,fd=26))

this is what it looks like with haproxy running (and httpd stopped):

[root@pbx ~]# sudo ss -ntlp | grep 6443
LISTEN 0 128 *:6443 : users:((“haproxy”,pid=25760,fd=7))

I will definitely open a ticket with Sangoma on this, but I’m wondering if the best option for testing purposes is to just change the port for one of haproxy or httpd manually to see if that gets everything working.

remove the grep to see what front-enda/backends are servicing under the various scenarios, your backends must run on ports and addresses proxied by haproxy

You’re a warrior for digging this deep. I ...'d out my IPs from this.

This is with httpd running (and HAProxy stopped)
[root@pbx ~]# ss -nt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44942
ESTAB 0 0 127.0.0.1:44938 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:6006 127.0.0.1:51216
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44552
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44932
ESTAB 0 0 127.0.0.1:53978 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:39540 127.0.0.1:3306
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44940
ESTAB 0 0 127.0.0.1:3306 127.0.0.1:39540
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44530
ESTAB 0 0 127.0.0.1:51254 127.0.0.1:6006
ESTAB 0 0 127.0.0.1:44940 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54960
ESTAB 0 64 ...:22 ...:52309
ESTAB 0 0 127.0.0.1:44530 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:53334 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54958
ESTAB 0 0 127.0.0.1:6379 127.0.0.1:53816
ESTAB 0 0 127.0.0.1:54956 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44930
ESTAB 0 0 127.0.0.1:54960 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44946
ESTAB 0 0 127.0.0.1:6379 127.0.0.1:53978
CLOSE-WAIT 1 0 127.0.0.1:41176 127.0.0.1:6086
ESTAB 0 0 127.0.0.1:44702 127.0.0.1:5038
CLOSE-WAIT 1 0 127.0.0.1:41374 127.0.0.1:6086
ESTAB 0 0 ...:46370 157.245.248.238:443
ESTAB 0 0 127.0.0.1:44942 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44938
ESTAB 0 0 127.0.0.1:42760 127.0.0.1:3306
ESTAB 0 0 127.0.0.1:44928 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:3306 127.0.0.1:39798
CLOSE-WAIT 1 0 127.0.0.1:34544 127.0.0.1:6002
ESTAB 0 0 127.0.0.1:44936 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44692
CLOSE-WAIT 1 0 127.0.0.1:34496 127.0.0.1:6002
ESTAB 0 0 127.0.0.1:44692 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:6006 127.0.0.1:51254
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:48218
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54962
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44936
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:45254
ESTAB 0 0 127.0.0.1:51216 127.0.0.1:6006
ESTAB 0 0 127.0.0.1:54658 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:54962 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44934
ESTAB 0 0 127.0.0.1:6379 127.0.0.1:53334
ESTAB 0 0 127.0.0.1:44946 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:48218 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:44934 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44928
ESTAB 0 0 127.0.0.1:3306 127.0.0.1:42760
ESTAB 0 0 127.0.0.1:44930 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:46520
ESTAB 0 0 ...:22 ...:52232
ESTAB 0 0 127.0.0.1:45254 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:54958 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:53816 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:46520 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44702
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54956
ESTAB 0 0 127.0.0.1:34266 127.0.0.1:6082
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44996
ESTAB 0 0 127.0.0.1:39798 127.0.0.1:3306
ESTAB 0 0 127.0.0.1:44996 127.0.0.1:5038
CLOSE-WAIT 1 0 127.0.0.1:41180 127.0.0.1:6086
ESTAB 0 0 127.0.0.1:44932 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54658
ESTAB 0 0 127.0.0.1:48016 127.0.0.1:8088
ESTAB 0 0 [::ffff:...]:6443 [::ffff:...]:52885
ESTAB 0 0 [::ffff:...]:6443 [::ffff:...]:52886
ESTAB 0 0 [::ffff:127.0.0.1]:8088 [::ffff:127.0.0 .1]:48016
ESTAB 0 0 [::ffff:127.0.0.1]:6082 [::ffff:127.0.0 .1]:34266
FIN-WAIT-2 0 0 [::ffff:...]:443 [::ffff:...]:52987
ESTAB 0 0 [::ffff:127.0.0.1]:44552 [::ffff:127.0.0 .1]:5038

This is with HAProxy running (and httpd stopped)
[root@pbx ~]# ss -nt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44942
ESTAB 0 0 127.0.0.1:44938 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44552
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44932
ESTAB 0 0 127.0.0.1:53978 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:39540 127.0.0.1:3306
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44940
ESTAB 0 0 127.0.0.1:3306 127.0.0.1:39540
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44530
ESTAB 0 0 127.0.0.1:44940 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54960
ESTAB 0 0 127.0.0.1:44530 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:53334 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54958
ESTAB 0 0 127.0.0.1:6379 127.0.0.1:53816
ESTAB 0 0 127.0.0.1:54956 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44930
ESTAB 0 0 127.0.0.1:54960 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44946
ESTAB 0 0 127.0.0.1:6379 127.0.0.1:53978
ESTAB 0 0 127.0.0.1:44702 127.0.0.1:5038
ESTAB 0 0 ...:46370 157.245.248.238:443
ESTAB 0 64 ...:22 ...:53109
ESTAB 0 0 127.0.0.1:44942 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44938
ESTAB 0 0 127.0.0.1:42760 127.0.0.1:3306
ESTAB 0 0 127.0.0.1:44928 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:3306 127.0.0.1:39798
ESTAB 0 0 127.0.0.1:44936 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44692
ESTAB 0 0 127.0.0.1:44692 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:48218
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54962
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44936
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:45254
ESTAB 0 0 127.0.0.1:54658 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:54962 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44934
ESTAB 0 0 127.0.0.1:6379 127.0.0.1:53334
ESTAB 0 0 127.0.0.1:44946 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:48218 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:44934 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44928
ESTAB 0 0 127.0.0.1:3306 127.0.0.1:42760
ESTAB 0 0 127.0.0.1:44930 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:46520
ESTAB 0 0 ...:22 ...:52232
ESTAB 0 0 127.0.0.1:45254 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:54958 127.0.0.1:27017
ESTAB 0 0 127.0.0.1:53816 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:46520 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44702
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54956
ESTAB 0 0 127.0.0.1:5038 127.0.0.1:44996
ESTAB 0 0 127.0.0.1:39798 127.0.0.1:3306
ESTAB 0 0 127.0.0.1:44996 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:44932 127.0.0.1:5038
ESTAB 0 0 127.0.0.1:27017 127.0.0.1:54658
ESTAB 0 0 [::ffff:127.0.0.1]:44552 [::ffff:127.0.0.1]:5038

Both haproxy and apache are claiming your DO.ip.address:443 but haproxy has

http-request redirect code 301 location https://%[hdr(host)]:3443/?action=reset if main_path

which currently returns

Unknown proxy destination. Please make sure CloudConnect Agent is running on your remote server.

I think you should wait for Sangoma

1 Like

Thank you for taking the time on this, dicko. I’m going to share this thread with sangoma and post the solution once it’s found.

1 Like

I spoke with Sangoma support and they got it working. My understanding is haproxy is running by default on the newest versions of system admin. Here is what support sent me:

"
As noted, in current version of system admin the haproxy.conf is being generated. I went ahead and enabled it via: fwconsole sa scdswitchovertohaproxy

You can also check the status via : haproxy-ctl status

There was also an httpd error_log entry:

] [php7:error] [pid 1334] [client 127.0.0.1:39774] PHP Fatal error: Uncaught Exception: Directory /var/run/asterisk/ssl doesn’t exist, and I can’t make it. in /var/www/html/admin/libraries/BMO/PKCS.class.php:593\nStack trace:\n#0 /var/www/html/admin/libraries/BMO/PKCS.class.php(56): FreePBX\PKCS->checkPermissions()\n#1 [internal function]: FreePBX\PKCS->__destruct()\n#2 {main}\n thrown in /var/www/html/admin/libraries/BMO/PKCS.class.php on line 593, referer: https://127.0.0.1:3333/admin/config.php?display=sysadmin&view=portmgmt

The issue was the /var/run/asterisk/firewall directory was owned by root. Once I ran fwconsole chown the prems are now good. We also rebooted the PBX a few times and confirmed apache and haproxy are no longer conflicting.
"

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.