Upgrade: UCP Server Failed:

Just upgraded from ~6.12.65-28 -> 10.13.66-6. UCP Server won’t start. In the GUI you see UCP node is not running. Starting or stopping it throws an error:

[[email protected] upgradescripts]# fwconsole stop ucpnode
Stopping UCP Server
UCP Server Failed:
[[email protected] upgradescripts]# fwconsole start ucpnode
Starting UCP Server
UCP Server Failed:

It’s not running BTW

[[email protected] asterisk]# ps -ef | grep ucp
root      1746  2358  0 00:50 pts/1    00:00:00 grep ucp

It actually will start manually:

[[email protected] node]# node node_modules/forever/bin/forever start -m 1 -a -l /var/log/asterisk/ucp_forever.log -o /var/log/asterisk/ucp_out.log -e /var/log/asterisk/ucp_err.log --uid 'ucp' index.js
warn:    --minUptime not set. Defaulting to: 1000ms
warn:    --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
info:    Forever processing file: index.js
[[email protected] node]# cat /var/log/asterisk/ucp_out.log
Starting FreePBX...FreePBX is Ready!
Asterisk version is: 13.5.0
Listening on port 0.0.0.0:8001
Loading all UCP Modules...
        Loading...conferencespro.js
        Loading...xmpp.js
Done!
[[email protected] node]# cat /var/log/asterisk/ucp_err.log
[[email protected] node]# cat /var/log/asterisk/ucp_forever.log
Starting FreePBX...FreePBX is Ready!
Asterisk version is: 13.5.0
Listening on port 0.0.0.0:8001
Loading all UCP Modules...
        Loading...conferencespro.js
        Loading...xmpp.js
Done!

The thing is, it seems to want a different version of npm than is installed

“engines” : {
“node”: “>= 0.10.29”,
“npm”: “>= 2.14.0”
}

[[email protected] log]# rpm -ql nodejs-0.10.40-1.shmz65.1.6.i686 | grep /usr/bin
/usr/bin/node
/usr/bin/npm
[[email protected] node]# /usr/bin/node --version
v0.10.40
[[email protected] node]# /usr/bin/npm --version
1.4.28

While npm and node arguments are in there they are not referenced. So we haven’t updated them. That is not your issue. It’s most likely permissions. Try starting it the way you did as the actual user. Not root.

There is no “actual user”, as far as I know. “asterisk” exists, but does not accept logins. However, I have tried installing ucpnode from module admin in the GUI as well as with fwconsole as root; same difference. I checked the audit.log in case it was an SELinux problem, but there’s no issue there.

In fact I get 4 events to the audit.log when trying to start with fwconsole which indicate it’s trying to sucessfully to run as asterisk

type=USER_START msg=audit(1448845085.743:7170): user pid=11845 uid=0 auid=0 ses=1057 msg='op=PAM:session_open acct="asterisk" exe="/sbin/runuser" hostname=? addr=? terminal=? res=success'
type=CRED_ACQ msg=audit(1448845085.743:7171): user pid=11845 uid=0 auid=0 ses=1057 msg='op=PAM:setcred acct="asterisk" exe="/sbin/runuser" hostname=? addr=? terminal=? res=success'
type=CRED_DISP msg=audit(1448845085.745:7172): user pid=11845 uid=0 auid=0 ses=1057 msg='op=PAM:setcred acct="asterisk" exe="/sbin/runuser" hostname=? addr=? terminal=? res=success'
type=USER_END msg=audit(1448845085.746:7173): user pid=11845 uid=0 auid=0 ses=1057 msg='op=PAM:session_close acct="asterisk" exe="/sbin/runuser" hostname=? addr=? terminal=? res=success'

A quick scan of /var/www/html shows nothing not owned by asterisk.

I get a successful fail2ban entry:

[2015-11-29 20:17:19] SECURITY[2325] res_security_log.c: SecurityEvent="SuccessfulAuth",EventTV="2015-11-29T20:17:19.412-0500",Severity="Informational",Service="AMI",EventVersion="1",AccountID="admin",SessionID="0xb6d00524",LocalAddress="IPV4/TCP/0.0.0.0/5038",RemoteAddress="IPV4/TCP/127.0.0.1/50286",UsingPassword="0",SessionTV="2015-11-29T20:17:19.412-0500"

You have to jump through some hoops, but it seems to run fine as asterisk too.

[[email protected] node]# runuser asterisk -s /bin/sh -c "node node_modules/forever/bin/forever start -m 1 -a -l /var/log/asterisk/ucp_forever.log -o /var/log/asterisk/ucp_out.log -e /var/log/asterisk/ucp_err.log --uid 'ucp' index.js"
warn:    --minUptime not set. Defaulting to: 1000ms
warn:    --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
info:    Forever processing file: index.js

[[email protected] asterisk]# cat ucp_forever.log
Starting FreePBX...FreePBX is Ready!
Asterisk version is: 13.5.0
Listening on port 0.0.0.0:8001
Loading all UCP Modules...
        Loading...conferencespro.js
        Loading...xmpp.js
Done!

[[email protected] asterisk]# ps -ef | grep /usr/bin/node
asterisk 13386     1  0 20:43 ?        00:00:00 /usr/bin/node /var/www/html/admin/modules/ucpnode/node/node_modules/forever/bin/monitor index.js
asterisk 13392 13386  0 20:43 ?        00:00:01 /usr/bin/node /var/www/html/admin/modules/ucpnode/node/index.js

I don’t know, something with fwconsole I guess.

Fwconsole works. It has to be something else.

Well, I replaced “node” with my own file and fwconsole never tried to invoked it, and quit with the same error. While I’m sure fwconsole works when it works, right now mine does not look like it working. So, how does it work? Is there a configuration file with metadata, or is it all baked in? Does it have dependencies, or is it stand alone? What does it do before it actually starts the service, because that would seem to be where it’s failing? Any other suggestions?

Never mind for now, I’ll have a look at “Ucpnode.class.php” apparently.

SELinux must be disabled - it must be off, totally. The reason it’s not starting is because the httpd context will be blocking it.

Turn off selinux, reboot, and you’ll find everything will be fixed.

Alright, I see what the problem is. It worked for me when I started it manually because I knew to give it a shell. The fwconsole class for it dosen’t give it a shell, and it also swallows the failure “This account is currently not available.”

The asterisk user is set up as a service account (okay, maybe I’m the one that locked it down, I can’t say for sure).

[[email protected] node]# cat /etc/passwd | grep asterisk
asterisk:x:498:498::/var/lib/asterisk:/sbin/nologin

…at least mine is anyway. So, the runuser needs a -s, as in the 2nd line below.

$process = new Process('runuser -l asterisk -c "'.$command.'"');

$process = new Process('runuser -l asterisk -s /bin/sh -c "'.$command.'"');

…and don’t even start with me on SELinux, I have about zero tolerance for people who go around disabling security becuase they can’t figure it out.

That’s great but the requirements for freepbx require disabling selinux so you are already on a custom non-supported configuration. Glad you got it figured out though

Heh. You have to basically give it full shell access, and access to all of asterisk, so it’s not actually DOING anything. So that’s why we say turn it off, and it’s the standard response. People don’t know what they’re doing, and go crazy trying to lock things down, and end up with a broken system.

Amusingly, this is EXACTLY what you have done trying to lock things down.

I’m not sure if that’ll get past peer review. By explicitly defining a shell, you’re significantly reducing security. If someone has configured the asterisk user to use rbash or similar, because they have a good reason to, and they know what they’re doing, that bypasses their security.

However, I don’t get to make that call. Feel free to create a pull request, and we’ll have a look at it.

However, what I think WOULD be a much better thing to do is to actually add a check to install that validates that the user hasn’t erroniously set the shell incorrectly.

I identified my issue, but I have to think for a bit about how I want to solve it permanently. I don’t leave shells on accounts that are not intended to have logins. I also don’t expect vendors to cater to non-standard configurations which have limited value/interest to the rest of the install base. I’m not going to officially submit that, it’s a hack I would not write myself, and vendors should not be fixing something they didn’t break. … but I don’t really want to patch every upgrade either. …I’m kicking around the idea of a runuser proxy, but I’m not sure what I’ll end up with yet.

I’m sorry about that SELinux quip, but you hit a sore spot because I’m tired of being lectured about it; if I had $1 for every time… I have it running on over 30 servers, yet that’s everyone’s go-to answer, because I’m assumed to be clueless until proven otherwise… you’re certainly not the first, and you won’t be the last.

There’s your problem. It is meant to have a login. We do a lot of privilege separation that way. Just lock the password, and give it a normal shell. It’ll be fine.

Edit: That should be the account you’re logging in to, to manage your FreePBX machine. It, 100%, should be a real account. Don’t use password auth, use ssh keys.

I use my own account, with ssh keys, and then su to root where necessary. Don’t want to log in as the same account which is running services, two things/people should not share an ID. It makes forensics difficult, among other things.

1 Like

I’ve decided to add a function to .bashrc that will effectively intercept the runuser command, check /etc/passwd for a valid shell for the user in the -l parameter, and if none, insert

-s /bin/
, before passing the command to the runuser binary. That way, only root will selectively override the users [non]-shell, and only if it didn’t have a valid one to begin with. That preserves the locked down user profile, moves all dependencies out of FreePBX, and very narrowly impacts system behavior, all through a user customizable profile file.

I’m probably the only person on the planet that cares about this, but I’ll post the code once it’s implemented and tested.