Install Hung - Web Site Won't Work w/curl?

A power failure a couple of weeks ago broke the one node in our HA cluster - 3 of the 4 drbd nodes reported an invalid media type, so it could not mount them. We are stuck on version 13 due to having HA, PBXact branded. Thankfully, the HA aspect worked & we’ve been running on the secondary while FedEx took their sweet time bringing me the drive. (I could’ve done a reinstall on the same drive, but would rather save it in case I need some of the configs.) I bought a replacement drive of the same model/size & have reinstalled it, but ran into an issue. Every time the server boots, it hangs for a while and then says:

Some firstboot error occured, and the system is not properly setup.
Check to see if you have internet access and re-run /etc/pbx_first_boot.sh
Press ctrl-alt-f1 to continue

I do not remember doing anything like this when installing the system in the past, so I looked at this file:

#!/bin/bash

# OUTPUT

if grep 'debug' /installed-version
then
  echo "SKIPPING THE DEFAULT RUN OF FIRSTBOOT"
  sleep 5
  exit 0
else
  echo "STARTING PBX FIRST BOOT"
fi

log_info(){
  echo "${@}" >> /dev/tty8
}

# LOGGING TO TTY8 and logfile
ORIGCONSOLE=$(fgconsole)
echo "starting on console 8, myconsole = ${ORIGCONSOLE}"
sleep 1
chvt 8
set -x -v
exec > /var/log/pbx/install/firstboot.log
exec 2>&1

#INCLUDING stringent.sh
# stringent.sh copyright michael potter 2008

# stringent.sh is intended to reduce the problems associated with
# using bash by turning on bash options that make errors more
# apparent.  This will not eliminate problems and I am sure there
# is some case where some problem will be made worse.
# USE AT YOUR OWN RISK.

set -o errexit  # errexit first
set -o nounset
set -o pipefail    # if you fail on this line, get a newer version of bash.

function traperr
{
   declare -i i;
   declare -i nestlevel;
   declare Message=${1:-""}

   nestlevel=${#FUNCNAME[@]}

   if (( $nestlevel <= 2 ))
   then
      echo "ERROR: ${BASH_SOURCE[1]}:${BASH_LINENO[0]} $Message" >&2
   else
      echo "ERROR: ${FUNCNAME[1]}(${BASH_SOURCE[1]}:~${BASH_LINENO[0]}) $Message" >&2
      for (( i = 2 ; i < $nestlevel ; i++ ))
      do
         echo "      ${FUNCNAME[$i]}(${BASH_SOURCE[$i]}:~${BASH_LINENO[($i-1)]})" >&2
      done
   fi
   # if BASH_SUBSHELL is 0, then script will exit anyway.
   if (( $BASH_SUBSHELL >= 3 ))
   then
      kill $$
   fi
   echo -e "Some firstboot error occured, and the system is not properly setup.\nCheck to see if you have internet access and re-run /etc/pbx_first_boot.sh\nPress ctrl-alt-f1 to continue" >> /dev/tty8
   exit 1
}

function traperrsimple
{
   # Use this function if the above function fails
   echo "ERROR: ${BASH_SOURCE[0]} ${LINENO}" >&2
   # if BASH_SUBSHELL is 0, then script will exit anyway.
   if (( $BASH_SUBSHELL >= 1 ))
   then
      kill $$
   fi
}

set -o errtrace
trap traperr ERR

function errexiton
{
   set -o errexit
   trap traperr ERR
}

function errexitoff
{
   set +o errexit
   trap '' ERR
}
#END INCLUDE

log_info "Testing for internet access"
INTERFACE=$(ip route show | awk ' /^default/ { print $5 }')
MACADDR=$(ip -o link show ${INTERFACE} | awk '/link/ {print $13}')
MACHASH=$(echo -en ${MACADDR} | md5sum - | awk '/-/ {print $1}')
ARCH=$(uname -m)
FREEPBXVERS=$(head -n 1 /etc/schmooze/pbx-version)
INSTALLTYPE=$(cat /installed-version | sed -e 's/.*kickstart-\([^.]*\).*/\1/')
BRAND_RETURN=$(curl -k -s -m 30 -A "Get Brand" } https://push2.schmoozecom.com/deployment/brand)
MYBRAND=$(echo ${BRAND_RETURN} | python -c 'import simplejson as json,sys;obj=json.loads(sys.stdin.read());print obj["brand"]')
if [[ -n ${MYBRAND} ]]
then
  echo ${MYBRAND} > /etc/schmooze/pbx-brand
fi
BRAND=$(head -n 1 /etc/schmooze/pbx-brand)
UARETURN=$(curl -s -m 30 -A "FreePBX Firstboot ${BRAND}-${FREEPBXVERS} ${ARCH} ${INSTALLTYPE}"  --data mac=${MACADDR} --data mhash=${MACHASH} http://kickstart.freepbxdistro.org/install-log/)

echo ${UARETURN} | grep 'passed'
log_info "Internet Access Test Passed"

errexitoff

# pull in all freepbx modules and load them
log_info ""
log_info "Updating all FreePBX modules. This can take 5-10 Minutes."
#Update all Modules now to make sure they are current
/usr/sbin/fwconsole ma installlocal
# Refresh Signatures due to issue with HA module and 32bit
/usr/sbin/fwconsole ma refreshsignatures
# Reload dialplan now
/usr/sbin/fwconsole chown
/usr/sbin/fwconsole r

# Write out apache conf.d files
/etc/init.d/incrond restart
/var/www/html/admin/modules/sysadmin/hooks/update-ports

# Stop iSymphony
/etc/init.d/iSymphonyServerV3 stop

errexiton
rm -rf /tmp/*

log_info ""
log_info "Running updatedb to update DB for locate command. This can take a minute."
updatedb

# if we've made it to this point, things have worked (minus the error exit stuff)

# Remove the entry from firstboot
log_info ""
log_info "Firstboot has finished, removing firstboot script"
rm -rf /etc/pbx_first_boot.sh
sed -i '/\/etc\/pbx_first_boot.sh/d' /etc/rc.d/rc.local

log_info ""
log_info ""
log_info ""
log_info "Install is now 100% complete"
chvt ${ORIGCONSOLE}

Not really knowing what this does, I began copy/pasting line by line into a console to see if I could figure out why it doesn’t seem to think there’s internet access. (There is.) This line seems to be the problem:

 UARETURN=$(curl -s -m 30 -A "FreePBX Firstboot ${BRAND}-${FREEPBXVERS} ${ARCH} ${INSTALLTYPE}"  --data mac=${MACADDR} --data mhash=${MACHASH} http://kickstart.freepbxdistro.org/install-log/)

Running curl with the same parameters only without -s (silent) results in a timeout after 30 seconds. If I load http://kickstart.freepbxdistro.org/install-log/ directly in a browser, I get this:

{"status":"failed","data":{"ua":"user agent string","ipaddr":"aaa.bbb.ccc.ddd","test":0},"errormsg":"Mac Hash did not match"}

This makes sense, since I’m not sending along any of the data the site expects from my PC’s browser. No matter what I’ve done, this site does not connect from curl on the PBX. I’ve done a yum update, but it had no impact. My hunch is that something on this site has been changed since mid-2019 (which is when I last did an install on either of these systems) which is preventing the old version of curl included in version 13 from getting a response.

Anyone have any ideas?

I think I’ve fixed this. Not figured out or fixed why curl won’t connect, but worked around it. The script appears to be some kind of phone-home mechanism that sends out information on your system when you do an installation, removing itself once it succeeds. Here’s what I’ve done so far:

fwconsole ma installlocal

This resulted in many errors & a subsequent run spammed an error with a bright red background.

fwconsole ma activate <deployment ID #>

Succeeded, rebranding the install to PBXact.

fwconsole ma upgradeall
yum install ipset

Buried in the scroll was a message about the ipset utility not being installed, so I installed that manually with the second line.

fwconsole ma upgradeall
fwconsole ma upgradeall
fwconsole ma upgradeall

Each time I executed this command, more modules were updated. The fourth time, nothing was upgraded.

fwconsole chown
fwconsole ma refreshsignatures

There’s one line in there that says:

checking builtin
Signature Invalid
Could not find signed module on remote server.

No idea what this means, but was hoping that logging into the GUI & running the upgrade to 10.13.66-22 from -21 would automatically fix it. This did not happen and it still reports the same thing, though I do not know if it will pose an issue going forward. The upgrade to -22 through the GUI did fix the High Availability option not being in the menu though.

My next step is to re-join it to the HA cluster & hope that all the configuration synchronizes properly, but wanted to document what I’ve done thus far in detail while it’s still fresh in my head in case it’s useful for others in the future.

Aaaannnnnd HA won’t work. :frowning: This is displayed by the automatic checks:

/var/spool/incron/root is not setup correctly.

Really curious as to why all this crap wasn’t necessary when I installed systems last time a couple years ago… really do hate internet-based configurations on systems like this! Give me a download to write a USB drive with a functional version of the system with all the dependencies any day. No, it won’t be the most updated versions, but it would work

Going to pick this up again tomorrow. incrond is running, so I don’t know what it’s looking for. All I’ve been able to find about this error is people asking about it without solutions. I’d appreciate any advice & will again report back for future searchers if/when I get it figured out.

After beating my head into a wall for a couple of hours trying to figure out why it wasn’t working, I found a workaround - I simply copied /var/spool/incron/root from the other node. :slight_smile: (What I found online about this error seems to be related to firewall not running <it is & I reconfigured it 3 times> and/or a version of incrond greater than that which is installed on the PBX.) Thankfully, copying this file over & restarting incrond resulted in the HA module being OK with joining the existing node. It’s re-synchronizing now, so hopefully this is just about done. :smiley:

If anyone needs it, here is the /var/spool/incron/root file I copied over to eliminate this issue. No real idea what it controls, why it wasn’t built when the corrupt node was installed, or if the settings in it are appropriate for other machines. (I figure that if I were in a spot where I didn’t have this file and found a thread online, I’d much rather have a way to fix it even if the settings aren’t 100% what I want.) It appears to be related to the stuff in the system admin module, but :man_shrugging:. One thing that does not show in the forum software is there is a single empty line at the very start… again, no idea if this matters or not. :frowning:

/var/spool/asterisk/sysadmin/network IN_CLOSE_WRITE /usr/sbin/sysadmin_network -a
/var/spool/asterisk/sysadmin/dns IN_CLOSE_WRITE /usr/sbin/sysadmin_dns
/var/spool/asterisk/sysadmin/epm IN_CLOSE_WRITE /usr/sbin/sysadmin_epm
/var/spool/asterisk/sysadmin/tz IN_CLOSE_WRITE /usr/sbin/sysadmin_time_zone
/var/spool/asterisk/sysadmin/vpnstop IN_CLOSE_WRITE /etc/init.d/openvpn stop
/var/spool/asterisk/sysadmin/vpnstart IN_CLOSE_WRITE /usr/sbin/sysadmin_openvpn
/var/spool/asterisk/sysadmin/vpnget IN_CLOSE_WRITE /usr/sbin/sysadmin_openvpn -d
/var/spool/asterisk/sysadmin/phone_configs IN_CLOSE_WRITE /usr/sbin/sysadmin_phone_config
/var/spool/asterisk/sysadmin/mdadm IN_CLOSE_WRITE /usr/sbin/sysadmin_mdadm
/var/spool/asterisk/sysadmin/intrusion_detection_stop IN_CLOSE_WRITE /etc/init.d/fail2ban stop
/var/spool/asterisk/sysadmin/intrusion_detection_start IN_CLOSE_WRITE /usr/sbin/sysadmin_intrusion_detection_restart
/var/spool/asterisk/sysadmin/intrusion_detection_banned IN_CLOSE_WRITE /usr/sbin/sysadmin_intrusion_detection_banned
/var/spool/asterisk/sysadmin/intrusion_detection IN_CLOSE_WRITE /usr/sbin/sysadmin_intrusion_detection
/var/spool/asterisk/sysadmin/email_setup IN_CLOSE_WRITE /usr/sbin/sysadmin_email_setup
/var/spool/asterisk/sysadmin/update_system IN_CLOSE_WRITE /usr/sbin/sysadmin_update_system -a
/var/spool/asterisk/sysadmin/ftp_setup IN_CLOSE_WRITE /usr/sbin/sysadmin_ftp
/var/spool/asterisk/sysadmin/ups_setup IN_CLOSE_WRITE /usr/sbin/sysadmin_ups
/var/spool/asterisk/sysadmin/update_system_cron IN_CLOSE_WRITE /usr/sbin/sysadmin_update_set_cron
/var/spool/asterisk/sysadmin/restart_httpd IN_CLOSE_WRITE /usr/sbin/sysadmin_restart_httpd
/var/spool/asterisk/sysadmin/restart_xmpp IN_CLOSE_WRITE /etc/init.d/prosody restart
/var/spool/asterisk/sysadmin/portmgmt_setup IN_CLOSE_WRITE /usr/sbin/sysadmin_portmgmt
/var/spool/asterisk/sysadmin/amportal_restart IN_CLOSE_WRITE /usr/sbin/sysadmin_amportal_restart
/var/spool/asterisk/sysadmin/wanrouter_restart IN_CLOSE_WRITE /usr/sbin/sysadmin_wanrouter_restart
/var/spool/asterisk/sysadmin/dahdi_restart IN_CLOSE_WRITE /usr/sbin/sysadmin_dahdi_restart
/var/spool/asterisk/sysadmin/incron_restart IN_CLOSE_WRITE /etc/init.d/incrond restart
/var/spool/asterisk/sysadmin/update_sysadmin_rpm IN_CLOSE_WRITE yum -y update sysadmin
/usr/local/asterisk/ha_trigger IN_CLOSE_WRITE /usr/sbin/sysadmin_ha
/var/spool/asterisk/sysadmin/restapps_restart IN_CLOSE_WRITE /usr/sbin/sysadmin_restapps_restart
/var/spool/asterisk/sysadmin/restart_queuecallback IN_CLOSE_WRITE /usr/sbin/sysadmin_queuecallback_restart
/var/spool/asterisk/sysadmin/xmpppresence_restart IN_CLOSE_WRITE /usr/sbin/sysadmin_xmpppresence_restart
/var/spool/asterisk/sysadmin/hostname_setup IN_CLOSE_WRITE /usr/sbin/sysadmin_hostname_setup

If anyone has any ideas of why things were so complex to simply install a system compared to what I did before, I’d be interested in hearing them. Don’t mind saying that I absolutely detest fixing something in this way… doesn’t make sense, don’t know why it was necessary, and don’t know why the fix worked. :slightly_frowning_face:

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.