so, I have been doing some diagnostics and trying to impliment several suggestions here and I noticed the following. now before I show what I did, I dont want to get off topic too much as I beleive the following is NOT my main problem, just noticed this little issue while doing diagnostics…
Late at night, I was the only active call on the PBX. My internet connection is 1GBPS and I am running a VPS PBX via CyberLynk with 4 CPU, 8G ram, 59 extensions.
I placed a single call, the only active call from my VoIP phone to my cell phone. I placed it on hold at the VoIP phone and listened to the MOH for a while. Consistently, literally, every minute (1-s seconds after the minute) there was a noticeable glitch in the audio. Granted its brief, but it is noticeable. When I look at cron there are only 2 crons that run every minute
I assume with the random on the 2nd one that its probably not the culprit? I see the issue literally 1 second after each minute so its possible that this RANDOM was 1? Is this calculated are boot up and remain constant or is it RANDOM/30 each minute?
To be clear, I do not believe this is my main issue. The audio quality issue I am having is much longer in duration. This issue is very short like a single stutter (fraction of a second) in the audio and really only noticeable when listening to predictable audio like music. The main question for this post was for a much longer drop/distortion, long enough to make the remote caller ask the VoIP user to repeat what they said.
Anyone else using Cyberlynk for a virtual PBX? Can you run this test? I did it on multiple PBX’s and still hear it.
I had a similar phenomenon every 10s or so a glitch during the MOH or any music (IVR). As I already mentioned, it was a problem of the SIP-trunk provider. Can’t you just set up a test trunk to a different provider? In Austria/Switzerland there is e.g. sipcall.ch/ sipcall.at. You can create an account in minutes and just charge it with 20€ and you are ready to go. I know it is a non-technical approach, but there are many providers out there, which arent fit for the job…and many customers dont care about stutters and glitches during a VoIP call…they dont know better
I would install Homer on a separate VPS and start capturing traffic and stats - see if there is any other commonality (date/time, numbers, length, etc). Homer also provides a MOS score which may help indicate which calls had issues.
I think priority is to determine whether this is happening internally (ext to ext) or externally only.
If external only, then I would look at your provider and routing to that provider or as someone suggested, temporarily setup a new provider and use that to test.
If both internal and external, then most likely the VPS or something along their route.
Can you setup a PBX using a different VPS provider and see whether the problems persist?
I am already in the process of setting up an AWS instance and a separate SIP provider. One way or another I will find whats causing the issues but I am very disappointed that I cant even setup a PBX with 1 extension without having some audio quality issues.
Even a Raspberry Pi would be a long way from overloading on 1 call; this has to be a network, service provider, or hosting problem, not an Asterisk or FreePBX one.
we use Vultr for our hosted FreePBX servers and always install an Edgemaec 2900e at the customer premise to act as the SBC ALG with traffic shaping enabled and we have very little if ever a problem no matter on asymetrical or symetrical connections eve non cable modems
I agree. Unfortunately both CyberLynk and Telnyx claim there are no issues on their end yet I still have a problem. I filled out all the forms to create an account at sip station and have yet to receive an email or call about adding the account, dont know why it cant be done online???
I am really hoping the switch to AWS builds a very stable and robust environment.
This doesn’t make sense. What are you referring to?
I think you have your answer here. It’s the network path between CL and Telnyx.
The un-managed plan at CyberLynk appears to get you FreePBX spun up from an ISO, which you can do at Vultr for 1/3 the cost. Try a $6 Vultr VM in one of their datacenters nearest your location and connect your Telnyx account.
Interesting note on CyberLynk,
Endpoint Manager and System Admin Pro module licenses are provided by Sangoma to FreePBXHosting.com customers at no charge, subject to Sangoma’s software licensing and support terms, policies, and procedures. Contact Sangoma Support for any questions regarding licensing or support terms.
Wonder what that’s about? But it’s not worth 3x the cost for an unmanaged VM.
@dvsatech I have the same exact issue. I am a small MSP and have about 25 shared machines on cyberlynk. I have one dedicated VPS for a major client and use bandwidth for trunks. My dedicated VPS has never had any issues. Some of my pbx’s on shared services have had this issue starting last year. Seems like only new systems have this issue. I mostly use ssd.25 As you, I have added extra cores with no relief. I have even switch the cyberlynk plans with more RAM with little to no help. htop command seems to show everything good. This has really caused some dissatisfaction with some of my customers. Cyberlynk support is really great, but sometimes I don’t think they understand the issue that I am having. This is the general response they typically give
"There are a few commercial modules using a lot of resources - if you aren’t using them I’d suggest disabling them: iotserver, queuestats, xmpp, pms, and anything else you can see using a lot of resources if you run ‘htop’ in the command line.
If you disable xmpp, please run the following from the linux command line to disable the inbuilt database for it:
systemctl stop mongod
systemctl disable mongod "
I also have gotten this
"There are probably a bunch of things you can disable on this server to reduce its resource utilization or you could upgrade to one of the larger SSD packages. They are hosted on hardware with faster I/O and fewer VMs so you should see an increase in performance.
But by default FreePBX Distro has a lot of things enabled that many users will not need. This can include cron tasks that run PHP scripts as often as every minute, some of which do not follow the global randomization method that prevents overloading system resources. (example: the iotserver jobs in asterisk user’s crontab) There are also multiple services and support systems that are not needed by many users such as XMPP, Sangoma Smart Office, Sangoma Property Management, User Control Panel, Phone Apps, etc. all of which have some degree of performance cost."
Some of the servers affected have endpoints on symmetrical Fiber DIA with only like 5 phone’s w/ QOS controls in place. I can not really find a rhyme or reason to these problems as well. Sometimes it’s call quality issues, but mostly its system recording breaking up and being choppy in the voicemails and ivr’s
Hope something in this helps. I am not bashing Cyberlynk, but wish we could figure this out!
I want to note, they claim “I am the only customer having this issue”. I have noticed the issue has become more prevalent in the last year as well. I couldn’t be sure who was the cause as Telnyx has gone through a merger (it appears) and some major upgrade to their network and infrastructure. Nothing was announced but I would constantly get updates about infrastructure changes and over time it was clear it was a complete network overhaul with the anticipated growing pains. Overall Telnyx support has been grate and they have on many occasions after providing the call ID pulled all the data for the call, analyzed it and indicated what the audio issue was but in every case telnyx and cyberlynk claimed the other was responsible .Many issues I had were related to what I believe to be a faulty ATA at telnyx digital->analog edge but over time the problem went away )I assume they located the bad device after enough user complaints).
I have told CyberLynk multiple times that they should recommend changing the cron jobs that run every minute to +/- a random few seconds as I believe all of the VM’s running FreePBX on the same hardware all run database tasks all on the minute. Each VM is adding cumulative I/O and delay which could be eliminated if the 1 minute scripts were randomly run over the 60 second period… Maybe even set the time based on the VM number, 1st one runs at +0, 2nd runs at +30, next runs at +15, and so on, evenly distributing the processes over 60 seconds. If its assigned at build time it doesn’t have to be “random”. When I complained that its their design flaw they upgraded all my PBX’s to SSD for free. Problem was dramatically reduced but still exists.
I spoke with Cyberlynk about dedicated services but was not comfortable with he limited redundancy. Other than 2 power supplies, the machines are bare metal without an VM. I would be much happier if they built a server with a private VM, and no other users on it. I would even be ok if they let me build a few VM’s on it.
I have my own data center but I use VPS for the redundancy and high speed internet connectivity. Also, in my data center I can do 1Gbps Internet but I don’t have all the redundancies of a VPS. Being able to save an image and spin on on a new machine in minutes has its benefits plus I dont have a generator and limited battery backup.
I have multiple FreePBXs installed on VMs. Some of VMs are just 1vCore 1GB RAM and 10G HDD for mini offices and they all work ok. I use dedicated servers in OVH and proxmox as a hypervisor. If I were you I would check different voip service provider using the same VM and see if there is still the problem.
Problem present: hosting provider is at fault. Problem nonexistent: voip service sucks
I do not know the regulations in your country, but in many countries in Europe you can have voip service up and running within 10 minutes.
Here is a simple perl script to measure and report scheduling delays. Start it up in the background, let it run for at least an hour, then post the output that it wrote to schedtime.log .
#!/usr/bin/perl
use Time::HiRes qw( time sleep );
$smalld = 0.010; # more delay than this is a "small" jitter
$badd = 0.030; # more delay than this is a "bad" jitter
open(LOG, ">schedtime.log") or die;
select((select(LOG), $|=1)[0]);
print LOG "time\tgood\tsmall\tbad\tmax\n";
$good = $small = $bad = $max = 0;
$duesec = 0;
for ($i = 0; $i < 60; ++$i) {
$bads[$i] = 0;
}
$timedue = int((($timenow = time) + 61) / 60) * 60;
sleep($timedue - $timenow);
while (1) {
$delay = ($timenow = time) - $timedue;
$max = $delay if $delay > $max;
if ($delay < $smalld) {
++$good;
}
elsif ($delay < $badd) {
++$small;
}
else {
++$bad;
++$bads[$duesec];
}
$timedue += 0.01;
if (int($timedue * 100 + .5) % 100 == 0) { # started new second
$timedue = int($timedue + .5); # set to exact integer
if (++$duesec == 60) { # started new minute
$duesec = 0;
for ($i = 0; $i < 60; ++$i) {
last if $bads[$i]; # at least one bad in last minute
}
if ($i < 60) { # one or more bad
($i, $mm, $hh) = gmtime($timedue);
$badstr = sprintf("%02d:%02d ", $hh, $mm);
for ($i = 0; $i < 60; ++$i) {
$badstr .= $bads[$i] > 9 ? 'X' : $bads[$i];
$bads[$i] = 0; # reset for next minute
}
print LOG $badstr, "\n";
}
if ($timedue % 600 == 0) { # crossed 10 minute boundary
($i, $mm, $hh) = gmtime($timedue);
printf(LOG "%02d:%02d\t%d\t%d\t%d\t%1.4f\n", $hh, $mm, $good, $small, $bad, $max);
$good = $small = $bad = $max = 0;
}
}
}
sleep($timedue - $timenow) if $timedue > $timenow;
}
Not to hijack, but on suppliers that work for us with FreePBX. Just on our experience Digital Ocean, Vultr and OVH with Proxmox are all good providers for FreePBX. We start them all on 1gb memory and 1 vCPU for small systems, never had a problem with those providers. We do up the type of VM (memory/cpu) at certain levels of concurrent calls we expect customers to do, seems to work well.
Most of the providers above are always giving away $100 vouchers to use for a month. No reason not to give them a go?
Thanks for the info. Which one of these can scale to large systems and just how big can it go? I’m not looking for cheap, I am looking for reliability, scalability, and #1 objective is best audio quality. If anything cost is last. I’m not looking to throw money away but I need carrier class service.
they all can scale to ridiculous levels, DigitalOcean at least can dynamically scale up and down the CPUs and memory but disk size can only be increased.
Just to throw my 2 cents into the ring I’ve been using a $5 plan with vultr since 2017. The only real problem I’ve had was when I rebooted it at one point and we lost SMTP to our mail server (we realized it a couple days later when we weren’t getting any voicemail transcription notification emails). That took an hour and a support ticket to resolve after I assured them that the only emails from that server are going to our mail server.
Surprised it worked in the first place? At least if it was port 25. They have a default block on all outbound port 25 unless you open a support ticket.
