UCP Node is not running

Hi there,

I’ve setup a vm running AsteriskNOW starting from back on version 3, and after a recent upgrade I started experiencing some issues with UCP.
Some background info:
Recently upgraded from AsteriskNOW 6.12.65-27 to version 10.13.66-7 (FreePBX version 13), step by step.
The server is currently running Asterisk 13 at the moment (was on Asterisk 13 before the upgrade without problems), asterisk -r > core show version shows Asterisk 13.7.0.
The OS is SHMZ OS (or CentOS) 6.6 64bit.
Hypervisor is ESXi 5.5, vmware-tools is installed via vmware repos.

The problem is that, I can log in to UCP but this will show up on the top: “Unable to connect UCP Node Server because: ‘Error: xhr poll error’”.
Also on the Dashboard, the UCP Daemon has a flame icon to indicate that the UCP Node is not running.
However, as I look in the console I can stop and start the UCP node without problems.

fwconsole stop ucpnode

Stopping UCP Server
Stopped UCP Server

fwconsole start ucpnode

Starting UCP Server
Started UCP Server

Looking at the log files from /var/log/asterisk/freepbx.log, most of the logs are just warnings of depreciated function and Info that ucpnode started/stopped successfully.
However, I do occasionally see a CRITICAL log that states Connection attmemp to AMI_failed, and Tried to link /var/lib/asterisk/agi-bin/areminder failed to copy from module directory.
Searching the forums it was suggested to do a fwconsole chown to fix the permission errors, but the same logs appeared afterwards.

Next I found a discussion that states that the nodejs package is corrupt.
So I executed the below commands.

yum remove nodejs
yum remove npm (This gave me “Package(s) npm available, but not installed.”)
yum install nodejs
fwconsole ma uninstall ucpnode
fwconsole ma remove ucpnode
fwconsole ma download ucpnode
fwconsole ma install ucpnode

There were no error messages performing the above, but it didn’t help either.
I also found a suggestion that the NPM needs to be upgraded, so I entered the following.

npm install npm -g

No errors, and I did the nodejs/ucpnode removal and reinstall again but the same thing.
npm --version gives me “3.7.1” as the version number.

I did find two types of errors in /var/log/asterisk/ucp_err.log, stating that it cannot find module ‘mariasql’ and ‘…/build/Release/sqlclient’, but couldn’t find an answer aside from the nodejs/ucpnode reinstall.
The content of ucp_out.log was blank.

I also tried to telnet into port 8001 from localhost and also another vm in the same network, but was rejected. I used localhost(from the server only), Local IP and fqdn and the results were the same.
I tried manually adding an iptable entry (iptables -A INPUT -p tcp --dport 8001 -j ACCEPT), but that made no difference.
The firewall of FreePBX was Off so I tried enabling this and did the setup, but port 8001 was still not accessible.
Would this be the issue?

Aside from the above, the Dashboard seems to be in well shape and I don’t have any problems aside from UCP.
Please let me know if any other information is needed.
Thank you.

Just wanted to add that the Port is set at default to 8001 in the Advanced Settings for UCP NodeJS Server, and is set to Enabled.

You need to reinstall ucp node.

Thank you very much for your reply.
I did run the fwconsole command to reinstall ucp node, and also did the reinstall from the web UI but the results were the same.
Just to check I did this again but the UCP Daemon is still in flames.

Please find: /var/www/html/admin/modules/ucpnode/node/logs/install.log

and paste it here.

Thank you very much for the quick reply; I pasted the contents of the log file below.
After looking at that install log, it looks like I have a problem with the make command.
So I then tried to install gcc with “yum install gcc”, but now I get the following error:

Error: Package: glibc-2.12-1.149.el6_6.9.i686 (updates)
Requires: glibc-common = 2.12-1.149.el6_6.9
Installed: glibc-common-2.12-1.149.shmz65.1.1.x86_64 (@updates)
glibc-common = 2.12-1.149.shmz65.1.1
Available: glibc-common-2.12-1.149.el6.x86_64 (base)
glibc-common = 2.12-1.149.el6
Available: glibc-common-2.12-1.149.el6_6.4.x86_64 (updates)
glibc-common = 2.12-1.149.el6_6.4
Available: glibc-common-2.12-1.149.el6_6.5.x86_64 (updates)
glibc-common = 2.12-1.149.el6_6.5
Available: glibc-common-2.12-1.149.el6_6.7.x86_64 (updates)
glibc-common = 2.12-1.149.el6_6.7
Available: glibc-common-2.12-1.149.el6_6.9.x86_64 (updates)
glibc-common = 2.12-1.149.el6_6.9
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

Is this because of the difference between the shmz and el6_6 builds?

—install.log contents—

[email protected] install /var/www/html/admin/modules/ucpnode/node/node_modules/bufferutil
node-gyp rebuild

make: Entering directory /var/www/html/admin/modules/ucpnode/node/node_modules/bufferutil/build' CXX(target) Release/obj.target/bufferutil/src/bufferutil.o make: g++: Command not found make: *** [Release/obj.target/bufferutil/src/bufferutil.o] Error 127 make: Leaving directory /var/www/html/admin/modules/ucpnode/node/node_modules/bufferutil/build’
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack at ChildProcess.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:820:12)
gyp ERR! System Linux 2.6.32-504.8.1.el6.x86_64
gyp ERR! command “node” “/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js” “rebuild”
gyp ERR! cwd /var/www/html/admin/modules/ucpnode/node/node_modules/bufferutil
gyp ERR! node -v v0.10.40
gyp ERR! node-gyp -v v3.2.1
gyp ERR! not ok
npm WARN install:[email protected] [email protected] install: node-gyp rebuild
npm WARN install:[email protected] Exit status 1

[email protected] install /var/www/html/admin/modules/ucpnode/node/node_modules/node-expat
node-gyp rebuild

make: Entering directory /var/www/html/admin/modules/ucpnode/node/node_modules/node-expat/build' CC(target) Release/obj.target/expat/deps/libexpat/lib/xmlparse.o make: cc: Command not found make: *** [Release/obj.target/expat/deps/libexpat/lib/xmlparse.o] Error 127 make: Leaving directory /var/www/html/admin/modules/ucpnode/node/node_modules/node-expat/build’
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack at ChildProcess.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:820:12)
gyp ERR! System Linux 2.6.32-504.8.1.el6.x86_64
gyp ERR! command “node” “/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js” “rebuild”
gyp ERR! cwd /var/www/html/admin/modules/ucpnode/node/node_modules/node-expat
gyp ERR! node -v v0.10.40
gyp ERR! node-gyp -v v3.2.1
gyp ERR! not ok
ucp-server@ /var/www/html/admin/modules/ucpnode/node
±- [email protected]
±- [email protected] extraneous
±- [email protected] extraneous
±- [email protected]
±- [email protected]
±- [email protected]
±- [email protected]
±- [email protected] extraneous
±- [email protected] extraneous
±- [email protected] extraneous
±- socket.io@1.3.6
`-- [email protected] extraneous

npm ERR! Linux 2.6.32-504.8.1.el6.x86_64
npm ERR! argv “node” “/usr/bin/npm” “update”
npm ERR! node v0.10.40
npm ERR! npm v3.7.1
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script ‘node-gyp rebuild’.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the node-expat package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs node-expat
npm ERR! Or if that isn’t available, you can get their info via:
npm ERR! npm owner ls node-expat
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /var/www/html/admin/modules/ucpnode/node/npm-debug.log

So after researching a bit more, it looks likes it’s asking for the i686 version of glibc-common, but there’s nothing in the repos.
I saw an article which states that the versions between x86_64 and i686 has to match. Is there an easy way to fix this, or am I looking at the wrong place?

OK, the UCP Node is now running with a green icon in the dashboard. :slightly_smiling:

The fix was simple; a new patch for Asterisknow came out (10.13.66-8).
I was then able to install gcc with “yum install gcc”, but the error was still there.
With the previous Andrew’s advice I looked at the install.log again and found out that it was actually looking for g++, so I then installed “yum install gcc-c++”.
After going through the removal and re-install of ucpnode again, everything was back to normal.

I wouldn’t have been able to fix the above if I wasn’t told were to look.
Thank you very much for your support!

This helped me thanks. Upgraded to the latest (10.13.66-9) removed and reinstalled, the issue was gone.

Glad to know that my post helped someone :slightly_smiling:

1 Like

This has been a continuous and on-going issue for many months across many versions for us.

The UCP Daemon is continuously found in a stopped state on all of my recent FreePBX servers.

I think this needs to be investigated further.

Sure. So provide some logs and details about your configuration

I glad to send info. What logs would be most helpful? I would like to be concise.

Most of my systems with the issues are those built within the past year from the FreePBX distro. 13 with Asterisk 13.

I keep them up to date with either the commercial system module or by running the update scripts.
Many have one or more commercial modules.

/var/log/asterisk/ucp_out.log and /var/log/asterisk/ucp_err.log

I see one of the causes is loosing the connection to Asterisk:
Lost Connection to Asterisk
Shutting down server on port 0.0.0.0:8001

The component is not resilient and does not restart when Asterisk is running. You can simulate this with a simple “service asterisk restart”

Also, It would be nice if the log had date/time stamps.

But it appears to be resilient. When I do a “service asterisk restart” this is what I get in my logs

Lost Connection to Asterisk
Shutting down server on port :::8001
Shutting down server on port :::8003
Regained Connection to Asterisk
Server up and running at 8001 port
Secure Server up and running at 8003 port

Hmm, I have two servers confirmed doing this.

My ucp_out logs all end with this and the dashboard also shows UCP not running:
Lost Connection to Asterisk
Shutting down server on port :::8001

It is a bit hard to correlate the data without time stamps.

The ucp_err log has this:

/var/www/html/admin/modules/ucpnode/node/lib/freepbx.js:81
throw “There was an error with MySQL Connection”;
^
There was an error with MySQL Connection

/var/www/html/admin/modules/ucpnode/node/lib/freepbx.js:81
throw “There was an error with MySQL Connection”;
^
There was an error with MySQL Connection

/var/www/html/admin/modules/ucpnode/node/lib/server.js:137
if (serverS.address()) {
^
TypeError: Cannot call method ‘address’ of null
at stop (/var/www/html/admin/modules/ucpnode/node/lib/server.js:137:14)
at EventEmitter. (/var/www/html/admin/modules/ucpnode/node/lib/server.js:93:4)
at EventEmitter.emit (events.js:117:20)
at Socket.emit (events.js:95:17)
at TCP.close (net.js:466:12)

/var/www/html/admin/modules/ucpnode/node/lib/server.js:159
if (serverS.address()) {
^
TypeError: Cannot call method ‘address’ of null
at stop (/var/www/html/admin/modules/ucpnode/node/lib/server.js:159:14)
at EventEmitter. (/var/www/html/admin/modules/ucpnode/node/lib/server.js:115:4)
at EventEmitter.emit (events.js:117:20)
at Socket.emit (events.js:95:17)
at TCP.close (net.js:466:12)

Why is your MySQL connection going down?

I’m also getting a similar error in the ucp_err.log:

/var/www/html/admin/modules/ucpnode/node/lib/freepbx.js:93
{ [Error: Disconnected from the server] code: 10001 }

/var/www/html/admin/modules/ucpnode/node/lib/freepbx.js:93
throw “There was an error with MySQL Connection”;

Any thoughts on what could be the issue? I just updated all the modules a couple days ago. Still on distro version 10.13.66-13

MySQL appears to be running properly.

FYI, I think I fixed this. I apologize if this is something I would have known if I RTFM.

I think my certificate was out of wack. I had paid for a comodo ssl certificate that I manually created and put in place for Apache. The web server worked fine.

I just re-generated the certificate using the certificate manager in FreePBX. Then I edited /etc/http/conf.d/ssh.conf and set these directives:
SSLCertificateFile /etc/asterisk/keys/integration/webserver.crt
SSLCertificateKeyFile /etc/asterisk/keys/integration/webserver.key

I checked the certificate as the default certificate to be used in certificate manager.

Restarted httpd
Ran a fwconsole restart
My UCP node no longer throws an error and no errors in ucp_err.log.

Does the UCP node use a certificate to talk to MySQL? (Sorry if dumb question).