Has anyone else seen this? I started a machine loading yesterday and came in this morning and it had been running for 16 hours - I thought it was a fluke, so I restarted it, and came back this afternoon and it has been running for 6 hours - it get’s to the place where it is actually configuring FreePBX (package 584 of 6-something) and that is where it dies.
Previous Sangoma-7 distros took a LONG minute at that point, but they succeeded in installing - I am going back to a previous version of the .ISO, but has anyone else seen this? It is reproducible.
Yeah…something is weird - This is a Dell Server with 128G of RAM (Currently using 52) and a PERC-6i - Not a killer RAID card but not terrible - about 300MB/Sec Read and Write on a 2TB RAID-5 - here is where it hangs on 13 and 14:
So far it has been installing for 51 minutes - but it’s not CPU, or Disk Bound - looking at performance monitor, it is not using any CPU (although the measurement is crude - it’s doing something) and Disk usage is peaking out at a little over 1MB/Sec - I just don’t know what it is doing so SLOWLY! This machine is a Dual E5540 Xeon - It get’s a Passmark of 7928 - but like I say, it’s not showing hardly ANY CPU utilization:
Well…do you know of any efficient way to clone it once it’s installed - It’s kind of a pain to go back and change ETH0 if you change the MAC and since I have to have different MAC’'s (because they are running on the same host) it’s either load it from scratch (and wait) or clone it and fix it.
Are you doing the nodejes updates sequentially? Those updates take forever - that would explain it - if you run the upgrade scripts on a fresh load of 13, the nodejs updates take forever and are easily 90% of the time it takes - how can we tell what it is doing? Can we switch to another console and see?
I don’t know how to answer this question so I don’t think you understand what I am talking about.
When you install the system nodejs 8.9.4 is installed. There is no “doing updates sequentially”. Not sure what that means.
There are over 120 freepbx modules, some of the modules also instal node packages, pm2, zulu, ucp, there is a list of requirements they install, node modules also depend on other modules… etc.
I mean that if you run the upgrade scripts sequentially on a fresh install to bring it to current, nodejs is installed over and over again - each individual script re-installs it again and again - it is about 90% of the total time.
On the first upgrade script, the machine is yummed up to current - that doesn’t happen again.
But every iteration of the script re-does nodejs over and over again.
Just looked on the server I tried to load - it is still not finished!
Something is WAY wrong - I can’t load a box - Have you all moved servers overseas? The only thing I was thinking last night was that I do have Geo-IP blocking engaged - I only allow basically North America and friendly European countires - during the install, where are you pulling from?
I am going to turn it off and re-attempt to install Sangoma-7 and report back.
No. That is not what I said and that is not what we are doing. The install is installing all FreePBX modules. Some of them have nodejs dependencies and are also installed. “Upgrade scripts” are a tool that was only used in distro 6. This is not distro 6. This is distro 7. Furthermore the script itself would echo to the screen that it was installing nodejs but all it was doing was runing: yum install nodejs, if you already had it then it wasn’t doing anything. Anyways this point is moot, this is distro 7.
No. It does not.
I’ve already told you the issue. You keep getting stuck on the fact that we are installing nodejs over and over again. I have tried to tell you that this is not the case.
More explaination
There are over 100+ modules in FreePBX.
During the install of the RPM “FreePBX”. It goes out and upgrades these modules. Some of these modules (I have SAID this already) require Nodejs deps. Like pm2, zulu, ucp. Therefore these will query the nodejs registry to download their components.
This IS NOT “re-do[ing] nodejs over and over again”.
You are getting rate limited by the nodejs servers (probably, maybe, it’s all a guess). There is nothing we can do about this. It’s not our servers.
Greg, Please read this carefully before jumping to conclusions. It’s exhausting having to reexplain this to you constantly. I have tried to make it clear what is going on with your connection.
It’s not your servers - it’s an incompatibility with Hyper-V Server 2016 - probably something Microsoft introduced in a recent patch, but very reproducible across multiple versions and multiple hosts - I went all the way back to 6.12.65 and it hangs in the same place.
Still scratching my head, I loaded VirtualBox on my machine and loaded the machine there - 13 Minutes Start to Finish!!!
So yeah, a real bomb on Server 2016 Hyper-V - unfortunately I have upgraded all my servers to 2016 so I don’t know if this is the case on 2012 also.
Greg, Please read this carefully before jumping to conclusions. It’s exhausting having to reexplain this to you constantly. I have tried to make it clear what is going on with your connection.
Listening is a two-way street - and the only one who should be exhausted here is me - I am the one that figured this out with no help from you Andrew.
It’s not slowly iterating and just taking a while - it is crashing - and I don’t know how to tell in what manner it’s crashing, but it is not just going slow - it is crashing.
Perhaps someone out there with another Server 2016 could confirm my findings - I have tried it on 2 Servers and a desktop - I am pretty sure it is a CentOS compatibility problem with patched Server 2016/Windows 10 Hyper-V.
The issue is that the installation is timing out. You don’t see anything in top because it’s probably nodejs waiting on a response from the registry that never comes. You’ll need to figure out your network issues to overcome that. There’s really nothing else that can be done about it from our side apart from rewriting the rpm. Which is next on the list of things to do. I tried to help you out by stating it was node installations but beyond that you really have to fix your network problems.
Jumping to “it’s your sequential updates” isn’t helpful. It’s not the cause. Distro 7 rpm installation installs nodejs once. Then each module goes out and also gets what it needs from the npm registry. Most often the issues happen there as their servers timeout and start to rate limit.
So no. It’s not nodejs sequential updates (we don’t do that). It’s probably the modules waiting for a response from either mirror or the nom registry. You won’t see network connections in top if they are waiting for a response (everything would be at 0)
I’ve tried to be helpful here in letting you know what the issue is but you’ve seemed to figure it all out on your own without me getting involved. Good luck.
If anyone chooses to go the way of creating the VM in VirtualBox to move it to Hyper-V for production, before you shut it down in VirtualBox, you need to do the following from the CLI:
You have to do this BEFORE you shut down in VirtualBox, or it will not boot in Hyper-V - make sure when you create the VM in VirtualBox that you also pick VHD for the Disk - this is the format used by Hyper-V (.vhd and .vhdx).
Well, I am going to mark this as solved - although I am not sure this solved it exactly, or if it was a combination of problems that were corrected in leading to fixing it but here goes:
My PBX Virtualization environment is very homogeneous in that all three of the machines I host on are the same - Dell PowerEdge R720’s with Dual Processors, 128G of RAM and 2-Terrabyte RAID 5 running Windows Server 2016 Standard - I think this was part of the problem in that all three machines are functionally identical, so even though I was testing on multiple machines, I was really testing on the same machine if you see what I mean - this was bad testing methodology on my part.
All three hosts were “Stuck” on Patch KB4077525 - here: https://support.microsoft.com/en-us/help/4077525/windows-10-update-kb4077525 - they had all tried multiple times to apply it and had failed over and over - I am thinking perhaps it was partially applied - but who knows. Broke my own rule here - Patch to current before you complain about a malfunction.
Testing was blowing my mind because a scratch load failed (works now!) but if you loaded the VM somewhere else (like VirtualBox) and the moved it over to the hosts, it would boot, but it was experiencing 50% packet loss - while on the same machine, VM’s of the exact same OS and Patch level were working perfectly and servicing customers - it was only the “New” Machines - perhaps another side affect of the failed update, because now that it is updated, it loads in 13 minutes - just like it should! I am sure this Packet Loss is why the install was failing - it was just giving up.
I found a few articles about NIC Teaming (something I was using) being the problem - it was not, so don’t waste time on this - DO have current drivers ALWAYS for Hyper-V - that is an old story - but teaming was not a factor whatsoever.
So I don’t know - maybe all this will help someone not waste a day and a half like I did chasing this down - and I think I can say definitively that if your install is hanging on the FreePBX portion of the install, hit Alt-F2 and start a Ping-Stream and see if you are dropping packets - this is the real reason the install failed.
