SSH backup and restore issues

I am trying to get SSH backup and restores working. I have read the URL below however it is for 2.8 and also running as root. I am running 2.10.0.41 of the module.

http://www.freepbx.org/news/2010-05-30/high-availability-backup-and-restore

I have created a backup user, generated the RSA keys, copied the key to the primary server and from the backup server command line I am able to SSH to it without being prompted for authentication.

When I configure the SSH server in the module for the path I am using /home/backupuser/.ssh/id_rsa and the username.

When I run the backup I get the error below

Saving Backup 3…done!
Intializing Backup 3
Connecting to remote server…
Warning: Identity file /home/backupuser/.ssh/id_rsa not accessible: Permission denied.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
Something went wrong when connecting to remote server. Aborting!

I also tried for the sake of following the article to use root and the /var/lib/asterisk/.ssh directory for the private key. I confirmed that I could SSH without being prompted but when I tried the backup I got the following error.

Saving Backup 3…Intializing Backup 3
Connecting to remote server…
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for ‘/var/lib/asterisk/.ssh/id_rsa.pub’ are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /var/lib/asterisk/.ssh/id_rsa.pub
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
Something went wrong when connecting to remote server. Aborting!
done!

Can someone point me in the right direction to get the backup/restore working as the backup user instead of root?

bump

I started over today, I found the guide below and changed it a little to not use root.

http://literature.schmoozecom.com/backup_restore-module/WarmSpare-Setup/Warm_Spare-Setup-Guide.pdf

Enable public key authentication on the destination server by uncommenting the following lines

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

Primary Server

chsh -s /bin/bash asterisk

passwd asterisk

chmod 755 /var/lib/asterisk

Backup server

chsh -s /bin/bash asterisk

passwd asterisk

Test basic SSH (with password) connectivity

ssh asterisk@primary -v

Generate your keys

sudo -u asterisk ssh-keygen

Copy public key to primary server

sudo -u asterisk ssh-copy-id -i /var/lib/asterisk/.ssh/id_rsa.pub asterisk@p-primary

Test the connection

ssh -i /var/lib/asterisk/.ssh/id_rsa asterisk@primary

I did end up running the commands you had mustardman but I can’t confirm that they are required.

This is what I see on the primary server logs when I click the server in the restore section of the module.

Jun 17 14:43:33 primary sshd[4861]: Failed password for asterisk from port 42711 ssh2
Jun 17 14:43:33 primary sshd[4861]: Failed password for asterisk from port 42711 ssh2
Jun 17 14:43:33 primary sshd[4870]: Connection closed by

I am using username asterisk and /root/.ssh/id_rsa when defining the server but when I run it I get the following error

Saving Backup 3…done!
Intializing Backup 3
Connecting to remote server…
Warning: Identity file /root/.ssh/id_rsa not accessible: Permission denied.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
Something went wrong when connecting to remote server. Aborting!

Sorry for so many posts, could you answer the following questions?

  1. When configuring the SSH server in FreePBX can you tell me what the key location and path should be?

  2. When creating the backup job, what should I choose for storage, is that where it is backing up to still then will use that to restore from. More of a two step process vs a single?

I ran tail “var/log/secure -f” while connecting and saw that it was saying

Authentication refused: bad ownership or modes for directory /var/lib/asterisk/.ssh

OK I got it working by running

chmod 700 /var/lib/asterisk/.ssh
chmod 600 /var/lib/asterisk/.ssh/authorized_keys

I also ran the commands below while troubleshooting but these I don’t think are necessary.

chmod go-w ~/
chmod 700 /var/lib/asterisk/.ssh
chmod 600 /var/lib/asterisk/.ssh/authorized_keys

Now to test the actual backup/restore, I will report back. Thank you for the guide.

I resolved the “Warning: Identity file /root/.ssh/id_rsa not accessible: Permission denied.” error by moving the RSA keys to /var/lib/asterisk/.ssh/

However I am still seeing

Jun 17 14:43:33 primary sshd[4861]: Failed password for asterisk from port 42711 ssh2
Jun 17 14:43:33 primary sshd[4861]: Failed password for asterisk from port 42711 ssh2
Jun 17 14:43:33 primary sshd[4870]: Connection closed by

and the backup is failing

Saving Backup 3…done!
Intializing Backup 3
Connecting to remote server…
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
Something went wrong when connecting to remote server. Aborting!

Here is the debug output.

ssh asterisk@primary -p 9522 -v
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to primary port 9522.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host ‘primary’ is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure. Minor code may provide more information
No credentials cache found

debug1: Unspecified GSS failure. Minor code may provide more information
No credentials cache found

debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug1: Offering public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password

I just followed this guide per your instructions, it copied the public key over but when I go to SSH it is still prompting for a password. I had tried setting this all up before posting here so its possible I messed something up along the way.

I am going to keep troubleshooting but am open to any suggestions.

Try from command prompt try “su” as the user doing the backup. So if the user is asterisk you would “su asterisk” on the source server and then ssh to the destination server "ssh asterisk@destinationserverIP. The first time you do this the destination server adds the source server as a trusted host but you have to manually answer “yes” for that to happen. After that it should just work.

I’ve not used this SSH function in the backup before, but…

“Warning: Identity file /home/backupuser/.ssh/id_rsa not accessible: Permission denied.”

What are permissions on this file? You say “from the backup server command line I am able to SSH to it without being prompted for authentication.” Did you do this as root or did you su to the backup user?

Whatever user the backup utility uses is the one you should try manually connect by ssh to test. Maybe it uses root. I don’t use it so I’m not sure. Using your example the user is “backupuser” since that is where you have the key stored. If it’s root then you need to move it to /root/.ssh

Permissions of that file should be rw by the backup user so “chmod 600 /home/backupuser/.ssh/id_rsa”
“chown backupuser. /home/backupuser/.ssh/id_rsa”

When I try to switch to the asterisk user I get the error message below. I am able to SSH without authenticating when I am logged in as backupuser or root.

su asterisk
This account is currently not available.

I think the permissions are set correctly

ls -la /home/backupuser/.ssh/

drwx------ 2 backupuser backupuser 4096 Apr 30 11:44 .
drwx------ 3 backupuser backupuser 4096 Apr 30 11:51 …
-rw------- 1 backupuser backupuser 1675 Apr 30 11:42 id_rsa
-rw-r–r-- 1 backupuser backupuser 411 Apr 30 11:42 id_rsa.pub
-rw-r–r-- 1 backupuser backupuser 417 Apr 30 11:44 known_hosts

bump

Perhaps user asterisk is set to nologin and you need to set it to /bin/bash. Also asterisk user home directory is usually /var/lib/asterisk so you need to put the private keys in there.

post the output of ‘finger asterisk’

using your example it’s backupuser not asterisk. Is there a reason you are not using user asterisk?

I just set this up for myself. Seems to work ok. FreePBX v2.11 backup/restore has some additional enhancements for doing this. There are check boxes to apply freepbx settings after restore while at the same time disabling trunks so you don’t have the same trunks registered on 2 servers…(thumbs up)!

I found the instructions on this site confusing so I didn’t use them. He is adding a public key on the primary server as root which is the easiest way but might not be a good idea security wise. I did everything as user asterisk on both primary and secondary server.

If you cannot connect via ssh after copying keys make sure that /var/lib/asterisk is chmod 755. By default it is 775 which ssh will not allow for key pairs in strict (default) mode which is why you have to remove group write to make it work. You will know if you have this problem because you will see an error in /var/log/secure that says “Authentication refused: bad ownership or modes for file /var/lib/asterisk/.ssh/authorized_keys”

Another thing to keep in mind is that asterisk user usually is set to /sbin/nologin. To be able to log in via ssh remotely you will have to change that to /bin/bash so “chsh -s /bin/bash asterisk”. A third thing is that asterisk user almost always has no password. You cannot ssh in as a user if they do not have password or keys. In order to copy the keys you need to have a password so it’s a chicken and egg situation.

Here are what I would consider the easiest possible instructions for setting up the key pair part. Tested on CentOS 6. This is my first crack at this. I will probably refine this more next try by adding a custom backup user like what you did.

On primary server:
#chsh -s /bin/bash asterisk
#passwd asterisk
somepassword
#chmod 755 /var/lib/asterisk

On secondary server:
#chsh -s /bin/bash asterisk
#su asterisk
$ssh-keygen

The last command will create the rsa 2048 bit key pair
$ssh-copy-id -i asterisk@primaryserverIP
This one command will create the remote directories, public (authorized_keys) key file, and set permissions for everything.

To test
$ssh asterisk@PrimaryServerIP
If it works without asking for password then you are good. First time it will ask to add host fingerprint so answer yes.

Now that keys are added remove asterisk password on primary server
#passwd -d asterisk

If your ssh port is something other than port 22
#ssh-copy-id -i ‘-p xxxx asterisk@PrimaryServerIP’
#ssh -p xxxx asterisk@PrimaryServerIP’

Thank you much mustardman.

The asterisk user it was set to /sbin/nologin, I changed it per your instructions.

Will setting a password on the asterisk account cause any issues, I see it is removed but should I do this after hours on the primary production system?

/var/lib/asterisk is currently set to 775 (I think), should it work as is?

drwxrwxr-x 13 asterisk asterisk 4096 Oct 15 2012 asterisk

If you follow my instructions exactly it will work. I guarantee it. There are other ways to copy over the key without creating an asterisk password so you don’t have to do that. Up to you if you know enough to do it other ways. My instructions were made for anyone to follow regardless of skill level.

My instructions also remove the password after so no security worries.

I just found a bug in the FreePBX v2.11 Backup/Restore module. It won’t restore the CDR DB automatically with the “Restore Here” box checked which would be the standard setting for a HA setup. You have to create a separate backup job just for CDR DB and manually restore. I filed a bug report.