For the past few days we have been having a problem where Sangoma Connect takes a long time to respond and the clients get disconnected frequently. What I notice is that all http related traffic is slowed down considerably, even the web interface is very slow on the server.
We have around 300 users and most of them use the Sangoma connect desktops client. I can see there are hundreds of httpd processes running on the server (over 250). I guess that since Sangoma Connect and Zulu also use the apache server to communicate with Asterisk they may be causing the slow down. This is a very robust server with 16 cores and 32gb of memory, using an SSD as the main drive, cpu utilization rarely goes over 2 and memory is around 18GB.
How can I know if Sangoma Connect or Zulu (we are migrating from Zulu to SC at the moment) are saturating Apache? When SC refuses to connect I have to stop Apache and restart it. After that I restart all other services lice Sangoma connect, zulu, ucp, etc. That clears the problem for a while but it comes back randomly.
This server has been in operation for months without any trouble with the around the same amount of users. Why are we seeing this now? Is there a way to optimize Apache for the high load? Any pointers on where to look for the bottleneck? Thanks
I would try an fwconsole restart or possibly pkill node if something has gone rogue it may just be an edge case but I would just start the daemons over clean
If you run top do you see a bunch HTTP processes running? We noticed this running sangomaconnect on a busy server. fwconsole restart resolved it until it happened again a few days later. Looks like maybe old HTTP processes were getting stuck and not shutting down for some reason, so it keeps forking more until it maxes out the server limit. It may be related to SMS messages, which seem to generate a lot of HTTP activity, but not sure.
There are 260 apache processes running at peak times. I will try to do a full shutdown today when there are no calls (24 hour operation). I do think this may be that we are hitting a resource limit somewhere but we have been working with the same amount of users for some time now and the problem just started showing up this week.
Hi @cursor Can you please check if you have Haproxy running in your system?
We have added haproxy to handle the desktop client connections, if not running then I would suggest raise support ticket so our team can help you out to configure/run the haproxy.
I restarted HAPROXY using “systemctl start haproxy” and it is now running but Sangoma Phone stills takes a long time to connect (and the web page to reload). I have a ticket open with Sangoma since last friday but no answers yet.
I can see that there are 258 http processes running when we have the slow down. The number only goes down after office hours when there are less users. Are we hitting a limit on Apache? How can we increase the number of users it can handle at once?
Only http related services are slow, you can ssh into the server and everything is responsive. Since Sangoma Phone, Zulu and the web page depend on Apache this is why I am focused there.
I think I finally found a solution to this problem. It seems that we were hitting the MaxRequestWorkers limit in Apache (256 by default). This is why during heavy traffic times anything that depends on Apache was very slow or not connecting at all. I have increased the limit to 450 for the moment and I can see that we are using around 350 processes during peak times.
I do not know why this only started happening a few days ago but the only thing I can think of is that we are moving users from Zulu to Sangoma Phone. Both use Apache on the back end but I guess maybe Sangoma Connect makes extra connections? We probably hit the number of sangoma phone users that went over the limit.
Sangoma support has been useless during this. They simply logged into the server via ssh and ran htop, then they told me that they did not see any spikes in cpu. They simply ignored everything I wrote about cpu and memory behavior, haproxy not running and my focus on Apache.