This might be the solution to the huge CPU-spikes. Please read this

The original poster is encouraged to bring back their original posts at any time

1 Like

Hi all,

First things first: thanks for making my account un-freezed and this topic visible again @tm1000. I really appreciate it.

Then I want to make clear; I have a lot of respect for Sangoma and FreePBX, and this thread wasn’t a attack at all. I just want to share a solution that worked for me and might help other people. I made this thread so we can all come to a even better solution.

Also, I would like to say thank you, to anybody that was reacting to this post with other solutions nor criticism against my solution. It helps a lot, to find a more efficient solution.

Sangoma, thanks for making FreePBX great. Together we can make FreePBX even greater than it already is.

Greets,
Serge

1 Like

According to the internal QA ticket on this these are the modules that now use the “job” consolidated functionality. Meaning it is now ‘one’ cron to rule them all. At least for these tasks that happen every minute. We’ve been working on this for a long time because of what the community said (but could not replicate the issue ourselves). This also helps solve a few other issues such as if you disable a module it will disable it’s cron jobs automatically and also remove them on uninstallation.

  • framework version 14.0.12
  • calendar v14.0.3
  • dashboard version 14.0.7
  • pms version 14.0.2.39
  • vqplus version 14.0.4.3
  • pagingpro version 14.0.2.17
  • timeconditions version 14.0.2.18

There’s been no formal write about this because it just entered testing yesterday and again. Kind of hard for me to work something up when we can’t replicate it ourselves. Once these/those are updated you’ll see that the “every minute” crons now turn into a single cron (you’ll notice it also randomizes a sleep from 0-30):

* * * * * [ -e /usr/sbin/fwconsole ] && sleep $[RANDOM\%30] && /usr/sbin/fwconsole job --run --quiet 2>&1 > /dev/null

Hi Andrew,

It seems like we’re getting closer to a permanent solution. I know it is hard to solve a problem if there is no formal or explained ticket of the specific problem. I have a lot of respect for you as developer.

I will look at the solution you made very soon. I will let you know the results.

Thanks in advance, for helping.

Greets,
Serge

One ringt o rule them all

Running many processes in parallel is always much more resource intensive than running them serially.

Good call sangoma

There were formal FreePBX Open Source issue tickets. The issue comes from the fact that we (as Sangoma employees) can’t log into any user’s system unless they have a support contract with us, to investigate this. That is for liability reasons (Imaging if I went into your system and broke it and you lost business revenue as a result. I’d be liable for damages and so would Sangoma). Therefore you have to open a support ticket where-as you enter into an agreement with Sangoma and it’s employees (you agree to this in the TOS for support services)

It’s not until someone opens a support ticket up about it can we look into it on the systems it’s happening in and unfortunately that sometimes means spending money with Sangoma. I don’t know how to word that to sound any better, but this is sometimes what needs to happen, unfortunately.

Reiterating. Sangoma does run a cloud infrastructure, but we never experienced these issues. What we did experience and fix I documented in a blog post (Performance Improvements in FreePBX) and all of those fixes are in the open source product. So that should have already helped people out. As for running cron processes minutely, some of those processes people were saying were killing their systems were curious, like time conditions, which doesn’t do anything major. But I could never replicate it so it was hard to see myself.

Also your account was “freezed” because you started this whole thread and then went through and removed every single response you posted, before I even got involved. Leaving an unbalanced amount of replies. That’s pretty close to trolling a forum system. So I locked your account.

One could word that "a curious developer who noticed a burgeoning problem in his own forum might go outside the pale to reach out to a compliant user to agree without charging them, to mutually investigate the problem, no? not much cost to Sangoma and working with a sophisticated end user is free , just beneficial to all.

The question is at what point financially, 1 no, 5 maybe, 25 probably needs investigation.

But what is irresponsible in the long run is to say “lots of people don’t have this problem, please go away” is that not "sweeping it under the carpet?

Here is the double edge sword of all this. Sangoma employees, like @tm1000, spend numerous hours on the forum providing help and support for the community. They not only do this during normal office hours, they do it after hours and weekends. Having support tech and project developers active in the community is how Sangoma tries to mitigate support costs for the community.

The problem with this, and this can be found in various threads about CPU spikes, is that when users are asked for specific details, test results, configuration information and don’t provide it either fully or at all it makes it hard for anyone (Sangoma employee or community member) to actually help at all. These things are going to be requested when you are paying for support time so why treat it like it means nothing when asked on the forum for free?

This is why paid support exists and has to be used in certain cases. Once the clock is ticking and there is a monetary cost applied it seems the information no one deemed important during the attempts to get free support find it vital to provide.

While the idea of Sangoma waving the support fee for one or two users that are having an issue like this CPU one is noble, it’s a slippery slope. Could turn out that the issue is not a Sangoma issue but a user issue that has nothing to do with Sangoma (improper VM config, etc.). When Sangoma says “This is the issue, it’s not related to us. You’ll need to fix it” and the user goes “Oh well how do I do that? Can you help?” is it fair then to start charging them?

This of course then leads to the “Well we looked at 3 user’s systems and it was all user error. Who is the next one to get free support to figure this out?” How long do you be noble and find that the 2 dozen people that reported the issue only 4 had real issues Sangoma needed to fix?

My response has always been the same on most of these types of threads. “I cant reproduce it” and “please go to support”. It’s the “please go to support” comment that people usually have the most issue with, I dont think I’ve ever told someone to “go away” (“go to support” usually winds up in my inbox anyways but we get the support guys to go through it first) but I get your gist there and hear your point.

“cannot reproduce” seems to irk people the same way. There’s a classic example of an issue with Yealink years ago where I kept telling people I couldn’t reproduce the bug with Yealink and people got upset and I got upset because I was sitting there with a Yealink phone in the Neenah office with @xrobau. It’s the stuff you don’t see behind the scenes that causes a lot of these assumptions, thats unfortunate but I can’t document my life like a reality show :slight_smile: (not that you expected that). These situations always seem to wind up with thinking that Sangoma isn’t trying or Sangoma doesn’t care or that I’m pushing people away when most offten I am spending my own “free” time looking at these issues before I actually comment “can not reproduce”.

When this issue first came up about CPU I went to the cloud team and I had a conversation with the lead cloud employee here at Sangoma and he stated he did not see those issues in our cloud environment. Since I don’t run cloud services I took him at his word. This is what he does for a living after all.

Here’s a few classic examples of statements that people have said where I go on the defensive and it turns into a spitting match:

  • “Sangoma is going to get rid of user and device mode”.

    • False. We literally and coditivly (I made that word up) can not. It’s impossible. FreePBX would blow up and not work. Additionally I just worked with a member of the Digium team on Voicemail changes Asterisk had to insure these changes would still work with the open source hot desking model that is in FreePBX for User and Device mode. Work is still being put into that
  • “There is high CPU and Sangoma don’t want to give the solution”

    • Just because we can’t reproduce something doesn’t mean Sangoma doesn’t want to give the solution. What seemed to be overlooked originally was that I said I couldn’t replicate it and please go to support. I only have so many hours in the day to deal with forum threads and forum issues and most of those hours are when I am not actively at work (to which my wife says “why are you on that site again”)
3 Likes

Ok let’s go to res publica . . .

Does any user here have a problem in this general area? It is an up or down answer.

I suppose I havent been following this issue with CPU spikes, so I am somewhat out of the loop. In my tiny little cloud, I have 89 images running on two hyperV hosts. Each VM has 2 vCPUs and 2048MB of RAM.

I havent noticed any CPU spikes, here is a small sample from a random VM.

Well, you said that somebody of Sangoma said that he doesn’t have any of these problem in the cloud. Than why didn’t you compared the cloud environment and the open-source project? Just to see if there is something different in this area? Sometimes I think that Sangoma wants to force us to go to their cloud environment. Am I the only one?

And I have a problem @dicko. But you already knew that ;p

Yes, because they aren’t.

1 Like

Seems good. On which system are you running? Just centos7 with the default configurations and packages? And do you feel comfortable with sharing the crontabs in (just a ss):

/var/spool/cron/crontabs/asterisk

It’s a brand new stock install I fired up to test another issue Ive been having.

SNG7.6 GA (SNG7-FPBX-64bit-1904-2.iso)

[[email protected] ~]# cat /var/spool/cron/asterisk
* * * * * [ -x /var/lib/asterisk/bin/schedtc.php ] && /var/lib/asterisk/bin/schedtc.php
* * * * * [ -x /var/www/html/admin/modules/dashboard/scheduler.php ] && /var/www/html/admin/modules/dashboard/scheduler.php > /dev/null 2>&1
12 * * * * /usr/sbin/fwconsole util cleanplaybackcache -q
@daily  [ -e /var/www/html/admin/modules/sysadmin/bin/check_portal.php ] && /var/www/html/admin/modules/sysadmin/bin/check_portal.php >/dev/null 2>&1
@daily  [ -x /var/lib/asterisk/agi-bin/ddns_client.php ] && /var/lib/asterisk/agi-bin/ddns_client.php >/dev/null 2>&1
@hourly  [ -x /var/lib/asterisk/bin/storage.php ] && /var/lib/asterisk/bin/storage.php >/dev/null 2>&1
59 0 11 * * /usr/sbin/fwconsole backup --backup=68131415-7e52-44f1-bf61-74668347252c  > /dev/null 2>&1
*/1 * * * * /usr/sbin/fwconsole calendar --sync 2>&1 > /dev/null
15 0 * * * /usr/sbin/fwconsole certificates --updateall -q 2>&1 >/dev/null
*/15 * * * * [ -e /etc/asterisk/firewall.enabled ] && touch /var/spool/asterisk/incron/firewall.firewall
30 22 * * * /usr/sbin/fwconsole recordingreports -s -c -q > /dev/null 2>&1
0 0 1 * * php /var/lib/asterisk/agi-bin/backuprecordings.php  > /dev/null 2>&1
@daily [ -x /var/lib/asterisk/bin/freepbx_sipstation_check ] && /var/lib/asterisk/bin/freepbx_sipstation_check 2>&1 > /dev/null
*/15 * * * * /usr/sbin/fwconsole userman --syncall -q
@daily  [ -x /var/lib/asterisk/agi-bin/update_license.php ] && /var/lib/asterisk/agi-bin/update_license.php --delay
14 4 * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma listonline --sendemail -q > /dev/null 2>&1
14 6 * * * [ -e /usr/sbin/fwconsole ] && /usr/sbin/fwconsole ma installall --sendemail -q > /dev/null 2>&1

Why wouldn’t we want you to use our cloud environment. Sangoma is a for profit company. Sangoma is not a charity. Sangoma has shareholders (you can find us on STC.V https://finance.yahoo.com/quote/STC.V/). Whenever this comes up I am always amazed that people think it’s wrong that Sangoma won’t share their secret cloud sauce and that Sangoma wants people to use their cloud service.

Our secret cloud sauce doesn’t have anything to do with FreePBX. We use the same FreePBX code base as you do. There are no modifications to the base code. Anything secret is all about how to manage a cloud environment. From the hardware to the monitoring services and I’m not going to give that information away for free? Then you’d just go copy how Sangoma does cloud and I’d be out of a job? (also Sangoma could literally sue me because that is disclosing corporate IP). Since FreePBX is open source I can’t (and have no intention) on hiding the fixes we do to Framework. But the Framework Sangoma Cloud uses is the same Framework you use.

I’m not forcing you to use Sangoma’s Cloud environment. But why wouldn’t I want you to use it? I’ve given you here (and in the other threads, posts, fixes and patches) ways we are mitigating these issues for you. Again. The code base is the same.

This thread is JUST FULL of crazy. Threads like this make me glad that I can comment here as an Official Arsehole™, and not as a Sangoma Employee, because I don’t need to be nice.

A few quick refutations.

  1. If you’re caring about ‘CPU Spikes’ then you’re running in a virtualized environment. If you’re NOTICING them, you’re probably running on ancient hardware. This is going to be 100% of your problem. I notice you haven’t actually said what hardware you’re running on, so you’re probably ashamed to admit you’re using 10 year old CPUs, or whatever.

  2. Sangoma’s cloud stack is built using exactly the same FreePBX as you guys download. It has MORE commercial stuff in it than you would get on a standard install. So if your statement was correct, it would be much worse for them.

  3. Cherry picking statements to make people seem to agree with your agenda is a dick move. @tm1000 and I know that logging into peoples machines without an indemnity is just asking to get sued. So if you’re unhappy with the free support you get here, then grow a pair and cough up the $200 or whatever it’s going to cost you to get dedicated access to his time. Yes. You can pay a small amount of money to get him to care extremely deeply about your issues and (hopefully) fix them.

  4. Deleting replies is ALSO a dick move. If you’ve said something that makes you look like an idiot, then congrats. You probably are one. Edit your post, and add your correction to the end of it. Or, even better, DON’T CLICK SUBMIT BEFORE YOU’VE THOUGHT ABOUT WHAT YOU’RE SAYING. If people engaged their brains BEFORE using their keyboard, the world would be a much better place.

Even worse is everyone acting all surprised when you’re pointed at big public blog posts saying ‘This is how to fix your problem’, because they haven’t bothered to read them.

I’m not even apologetic about posting this rant. This entire thread makes me sad, and I hate to think how Andrew is feeling.

6 Likes

Hi,

Welcome to the thread Official Arsehole™. Thanks for letting your opinion over here. Let me reply to your points.

It is definnely not the hardware that are causing the problems. Don’t say things that aren’t even true, because that perfectly shows how stupid your reaction is. With all the respect of course. Just don’t do it. We run on perfect hardware, and great intel CPUs.

Andrew said that the cloud-enviroment is based on the FreePBX opensource core, but does have some extra nor changed features in the system. I just said, that Andrew probably could take a look at the difference between the cloud-enviroment and the open-source project. Just to make sure. Why not?

I never said that I’m not happy with the support over here. Actually, I don’t ask for support in this thread. I’m sharing a solution that worked for ME and MIGHT help others. If 25+ people are reporting a problem, it can be a global problem. Not just only for me. There are a lot of people in this and other threads that haves the same issue.

I didn’t wanted to delete my replies at all. The system freezed my account, and then Andrew freezed this whole thread. I couldn’t change anything. How am I suppose to edit, react, nor change something if my account is blocked? If you read my thread, then you can see I explained this already.

Again: I have a lot of respect for Andrew, Sangoma and FreePBX. I’m just sharing possible problems nor solutions. Why is this a problem? That’s where a forum is for. I’m just trying to help other people, or come to a permanent solution with Sangoma development team, and other users here.

And that would be? What is the hardware, VM environment and the CPUs?

He is referring to the fact that when you get a hosted PBX with Sangoma, you get commercial modules included. So it’s still FreePBX that you and I would install but instead of having to pay extra for the commercial modules they are baked into the VM costs.

I’ve already covered this. 25 of 500,000 (or more) isn’t a global problem. It’s a very small percentage of the user base. Not even a full percent.

That is a lie. Please don’t say things that are untrue. You deleted your posts, edited the opening post to praise dicko and also changed the subject of the thread. All of this was done before @tm1000 did anything to hide the thread. I was still in the thread when you removed your posts and the system refreshed the changes.

Ok @shuijsen can you just stop. Just because we let you back in here after you deleted all of your replies doesn’t mean you can just go all out on your replies calling other people stupid

No it does not. It is the same FreePBX you are using.

Its. the. same. code. base.

You went through and “deleted” all of your content here way before the system detected anything. After you removed everything you posted

I deleted this topic due to some unleft risks facts. For a better solution please visit a thread of @dicko. He got some great threads, with a lot of powerful and efficient solutions.

Then I made the thread “invisible” because what was the point. All of your responses including your original comment were removed. It was just replies arguing at a black hole. I then silenced your account because you went and removed all of your replies on this thread making it unbalanced and bordering on trolling.

Lorne actually posted about you deleting all of your replies before I was even actually at my computer.

Shortly after that I unlisted the thread. See for yourself. Judging from what the forum is saying Lorne posted 7 hours ago. Then I unlisted the thread 6 hours ago. About an hour after you went through and removed everything of value from this thread.