Upgrade to FreePBX 14 from 13 failing

Hi,

I’m trying to upgrade from FreePBX 13 to FreePBX 14, but it fails quite early in the process with the folllowing error:

[[email protected] ~]# distro-upgrade
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                             ┃
┃         Sangoma 6 to 7 Upgrade Tool         ┃
┃                                             ┃
┃    Distro Upgrade - Version 1707-11.sng7    ┃
┃    Build Date: 2017-07-24                   ┃
┃                                             ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

 Checking prerequsites...
        Checking bitsize of machine     [ ✔ ] - x86_64
        Checking available disk space   [ ✔ ] - 199G Available
        Checking for outdated system    [ ✔ ] - No pending updates.
        Checking for HA Setup           [ ✔ ] - Not a HA system
 All prerequsites passed!

Are you ready to upgrade your machine to SNG7? This process requires
two reboots, and will download approximately 200mb of files before
starting. There will be no interruption to service until this machine
is rebooted.

Download files required for upgrade [Yn]? y
######### Starting setup upgrade on Thu Aug 10 22:24:46 BST 2017 #########
######### Creating upgrade repofile #########
######### Installing needed packages #########
######### Running preupgrade #########
######### Running upgrade-tool #########
######### An error occured, please check /var/log/sngupdate #########
Error! setup_upgrade did not exit cleanly. Not enabling upgrade service
[[email protected] ~]#

I get the following in the sngupdate log file:

######### Running upgrade-tool #########
upgrade                                                  | 2.9 kB     00:00
.treeinfo                                                |  658 B     00:00
setting up repos...
getting boot images...
Traceback (most recent call last):
  File "/usr/bin/redhat-upgrade-tool", line 310, in <module>
    main(args)
  File "/usr/bin/redhat-upgrade-tool", line 209, in main
    kernel, initrd = f.download_boot_images() # TODO: force arch?
  File "/usr/lib/python2.6/site-packages/redhat_upgrade_tool/download.py", line 413, in download_boot_images
    copy2(initrd, initrdpath)
  File "/usr/lib64/python2.6/shutil.py", line 95, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.6/shutil.py", line 52, in copyfile
    copyfileobj(fsrc, fdst)
  File "/usr/lib64/python2.6/shutil.py", line 31, in copyfileobj
    fdst.write(buf)
IOError: [Errno 28] No space left on device
######### An error occured, please check /var/log/sngupdate #########

If I run df -i I get the following:

[[email protected] ~]# df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda2      15204352 202298 15002054    2% /
tmpfs            235818      1   235817    1% /dev/shm
/dev/sda1         25688     48    25640    1% /boot

I’m currently running FreePBX 10.13.66-21. Any idea what might be wrong, and what i can do to fix it???

Does df -i show free space, or just free iNodes? What does df -h or just df say?

Ok. It looks like it might be a space issue! But how do I go about fixing this???

Thx.

[[email protected] ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       229G   18G  199G   9% /
tmpfs           922M     0  922M   0% /dev/shm
/dev/sda1        97M   97M     0 100% /boot
[[email protected] ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      239434304 18799184 208472560   9% /
tmpfs             943272        0    943272   0% /dev/shm
/dev/sda1          99150    99147         0 100% /boot
[[email protected] ~]# df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda2      15204352 202300 15002052    2% /
tmpfs            235818      1   235817    1% /dev/shm
/dev/sda1         25688     48    25640    1% /boot
[[email protected] ~]#

This is what’s in the /boot directory. Is there anything I can safely remove from there???

[[email protected] boot]# ls -l
total 92972
-rw-r--r--  1 root root   105195 Nov 22  2013 config-2.6.32-431.el6.x86_64
-rw-r--r--  1 root root   108107 Oct 26  2016 config-2.6.32-642.6.2.el6.x86_64
drwxr-xr-x. 3 root root     1024 Feb  2  2013 efi
drwxr-xr-x. 2 root root     1024 Feb  8  2017 grub
-rw-------  1 root root 16777211 Apr 13  2015 initramfs-2.6.32-431.el6.x86_64.img
-rw-------  1 root root 23162073 Feb  8  2017 initramfs-2.6.32-642.6.2.el6.x86_64.img
-rw-r--r--  1 root root 35303424 Aug 10 22:29 initramfs-redhat-upgrade-tool.img
drwx------. 2 root root    12288 Feb  2  2013 lost+found
-rw-r--r--  1 root root   193758 Nov 22  2013 symvers-2.6.32-431.el6.x86_64.gz
-rw-r--r--  1 root root   215530 Oct 26  2016 symvers-2.6.32-642.6.2.el6.x86_64.gz
-rw-r--r--  1 root root  2518236 Nov 22  2013 System.map-2.6.32-431.el6.x86_64
-rw-r--r--  1 root root  2618532 Oct 26  2016 System.map-2.6.32-642.6.2.el6.x86_64
-rwxr-xr-x  1 root root  4128368 Nov 22  2013 vmlinuz-2.6.32-431.el6.x86_64
-rwxr-xr-x  1 root root  4270096 Oct 26  2016 vmlinuz-2.6.32-642.6.2.el6.x86_64
-rw-r--r--  1 root root  5396240 May 18 09:01 vmlinuz-redhat-upgrade-tool
[[email protected] boot]#

ok. I removed the two initramfs-2.6 img files which has freed up some space. hopefully that’s not done any lasting damage!

ls -l / df -l now reports:

[[email protected] boot]# ls -l
total 53813
-rw-r--r--  1 root root   105195 Nov 22  2013 config-2.6.32-431.el6.x86_64
-rw-r--r--  1 root root   108107 Oct 26  2016 config-2.6.32-642.6.2.el6.x86_64
drwxr-xr-x. 3 root root     1024 Feb  2  2013 efi
drwxr-xr-x. 2 root root     1024 Feb  8  2017 grub
-rw-r--r--  1 root root 35303424 Aug 10 22:29 initramfs-redhat-upgrade-tool.img
drwx------. 2 root root    12288 Feb  2  2013 lost+found
-rw-r--r--  1 root root   193758 Nov 22  2013 symvers-2.6.32-431.el6.x86_64.gz
-rw-r--r--  1 root root   215530 Oct 26  2016 symvers-2.6.32-642.6.2.el6.x86_64.gz
-rw-r--r--  1 root root  2518236 Nov 22  2013 System.map-2.6.32-431.el6.x86_64
-rw-r--r--  1 root root  2618532 Oct 26  2016 System.map-2.6.32-642.6.2.el6.x86_64
-rwxr-xr-x  1 root root  4128368 Nov 22  2013 vmlinuz-2.6.32-431.el6.x86_64
-rwxr-xr-x  1 root root  4270096 Oct 26  2016 vmlinuz-2.6.32-642.6.2.el6.x86_64
-rw-r--r--  1 root root  5396240 May 18 09:01 vmlinuz-redhat-upgrade-tool
[[email protected] boot]# df -l
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda2      239434304 18799888 208471856   9% /
tmpfs             943272        0    943272   0% /dev/shm
/dev/sda1          99150    59988     34042  64% /boot
[[email protected] boot]#

Would this be enough free space to run the upgrade?

I’ve since read that I perhaps shouldn’t run the upgrade script if I’m running DAHDI, which I am. Any advice on this?

Ok.I took the plunge, moved some of the files from /dev/sda1 and went for the Upgrade. Unfortunately, I’ve managed to break FreePBX along the way.

When I log in at the console I don’t get the FreePBX artwork and just get thrown to the command line. Asterisk doesn’t start automatically either. I can get Asterisk running to an extent by just typing asterisk at the command line (not with DAHDI yet, but I’ve not looked into that yet), but I can’t get FreePBX up and running. If I type fwconsole as root, I get the following error:

[[email protected] conf]# fwconsole
PHP Fatal error:  Incompatible file format:  The encoded file has format major ID 4, whereas the Loader expects 7 in /var/www/html/admin/modules/sysadmin/agi-bin/LoadLicenseIfExists.php on line 0
Whoops\Exception\ErrorException: Incompatible file format:  The encoded file has format major ID 4, whereas the Loader expects 7 in file /var/www/html/admin/modules/sysadmin/agi-bin/LoadLicenseIfExists.php on line 0
Stack trace:
  1. () /var/www/html/admin/modules/sysadmin/agi-bin/LoadLicenseIfExists.php:0
[[email protected] conf]#

I also had to mess around a bit to get Apache working. Listen 80 had appeared in two config files, and firewall rules had to be set up.

Any idea what’s going wrong, and is it recoverable???

Hi!

Please follow the troubleshooting directives as the bottom of that wiki page, you need to create a ticket and upload two files in it.

edit: looks like tickets with the same issue are being closed as duplicates of https://issues.freepbx.org/browse/FREEPBX-15500

Someone else had the same problem in the following thread (which I had created):

It’s related to the way the commercial modules source code is made unavailable and, normally, the script should have taken care of this, see:

I am not a 100% sure which script you run, the original one or the updated one which most likely has a fix for this and I can’t check right now as the FreePBX wiki seems to be down from here…

Good luck and have a nice day!

Nick

Please provide the output from the following command so that we can determine which version of the RPM was run:

rpm -q --qf "[%{NAME} %{VERSION}-%{RELEASE}\n]" distro-upgrade 

Unfortunately, the WIKI seems to be down right now, so I can’t see any troubleshooting guide.

I get the following:

[[email protected] ~]# rpm -q --qf "[%{NAME} %{VERSION}-%{RELEASE}\n]" distro-upgrade
package distro-upgrade is not installed
[[email protected] ~]#

When upgrading, I definitely used the latest script, build was dated 11 August IIRC. I can’t get access to the WIKI site to confirm the revision, and I can’t remember where I downloaded it to on my machine.

Edit: Looking back through my command line history, I can see that this is what I ran:

[[email protected] ~]# yum -y install http://package1.sangoma.net/distro-upgrade-1707-12.sng7.noarch.rpm
[[email protected] ~]# distro-upgrade

I also had to…
[[email protected] ~]# yum remove libmp3lame-3.98.4-4.el6.x86_64
…part way through the upgrade to get out of an endless loop. I understand from the other posts that all el6 packages need to be removed as part of the upgrade process. I think this is a package I may have manually installed a long time ago to get some other functionality working.

Right. I’ve managed to make some good progress thanks to the ‘issues’ link provided by Marbled:

https://issues.freepbx.org/browse/FREEPBX-15500

From here I downloaded and ran the post_update script, which ran until it got to modifying the httpd.conf file, where it then hung up. After waiting an age, I gave up and reset the machine.

I can now access FreePBX via the GUI :slight_smile:

Only problem I now have is that I can’t get DAHDI to work. It looks like I might be running an old version:

[[email protected] dev]# yum list *dahdi*
Loaded plugins: fastestmirror, kmod, versionlock
Loading mirror speeds from cached hostfile
Installed Packages
asterisk13-dahdi.x86_64                           13.17.0-3.sng7                                               @sng-pkgs
dahdi-firmware.noarch                             3.0.0-1.sng7                                                 @sng-pkgs
dahdi-firmware-xorcom.noarch                      1.0-1                                                        @anaconda-CentOS-201207061011.x86_64/6.3
dahdi-linux.x86_64                                2.11.1-1.26.sng                                              installed
dahdi-linux-devel.x86_64                          2.11.1-1.26.sng                                              @sng-pkgs
dahdi-linux-kmod-debuginfo.x86_64                 2.11.1-22_centos6.2.6.32_642.6.2.el6.x86_64                  @pbx/6
dahdi-tools.x86_64                                2.11.1-1.23.sng                                              @sng-pkgs
dahdi-tools-doc.x86_64                            2.11.1-1.23.sng                                              @sng-pkgs
kmod-dahdi-linux.x86_64                           2.11.1-22_centos6.2.6.32_642.6.2.el6.x86_64                  @pbx/6
schmooze-dahdi.noarch                             1.0.0-2                                                      @pbx/6
Available Packages
asterisk11-dahdi.x86_64                           11.25.1-1.sng7                                               sng-pkgs
asterisk14-dahdi.x86_64                           14.6.0-3.sng7                                                sng-pkgs
dahdi-firmware-a4a.noarch                         a0017-1.3.sng                                                sng-pkgs
dahdi-firmware-a4b.noarch                         d001e-1.3.sng                                                sng-pkgs
dahdi-firmware-a8a.noarch                         1d0017-1.3.sng                                               sng-pkgs
dahdi-firmware-a8b.noarch                         1f001e-1.3.sng                                               sng-pkgs
dahdi-firmware-hx8.noarch                         2.06-1.3.sng                                                 sng-pkgs
dahdi-firmware-oct6114-032.noarch                 1.07.01-1.3.sng                                              sng-pkgs
dahdi-firmware-oct6114-064.noarch                 1.05.01-1.3.sng                                              sng-pkgs
dahdi-firmware-oct6114-128.noarch                 1.05.01-1.3.sng                                              sng-pkgs
dahdi-firmware-oct6114-256.noarch                 1.05.01-1.3.sng                                              sng-pkgs
dahdi-firmware-oct6126-128.noarch                 01.07.04-1.3.sng                                             sng-pkgs
dahdi-firmware-tc400m.noarch                      MR6.12-1.3.sng                                               sng-pkgs
dahdi-firmware-te133.noarch                       780019-1.3.sng                                               sng-pkgs
dahdi-firmware-te134.noarch                       780017-1.3.sng                                               sng-pkgs
dahdi-firmware-te435.noarch                       e0019-1.3.sng                                                sng-pkgs
dahdi-firmware-te436.noarch                       10017-1.3.sng                                                sng-pkgs
dahdi-firmware-te820.noarch                       1.76-1.3.sng                                                 sng-pkgs
dahdi-firmware-vpmoct032.noarch                   1.12.0-1.3.sng                                               sng-pkgs
[[email protected] dev]# 

How can I fix this???

I replied in the other post you replied to…

Looks like you need to apply

https://issues.freepbx.org/browse/FREEPBX-15491?focusedCommentId=108399&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-108399

(this was the ticket I opened, I had, amongst other things, DAHDI issues…)

Don’t install the sangoma-devel package, you don’t want to use that repo under normal circumstances and the package is now available from the usual repos…

Good luck and have a nice day!

Nick

Thanks. That was useful. DAHDI is now updated, but unfortunately, it still hasn’t fixed my problem:

[[email protected] dev]# yum list *dahdi*
Loaded plugins: fastestmirror, kmod, versionlock
Loading mirror speeds from cached hostfile
Installed Packages
asterisk13-dahdi.x86_64                                         13.17.0-3.sng7                                 @sng-pkgs
dahdi-firmware.noarch                                           3.0.0-1.sng7                                   @sng-pkgs
dahdi-linux.x86_64                                              2.11.1-1.26.sng                                installed
dahdi-linux-devel.x86_64                                        2.11.1-1.26.sng                                @sng-pkgs
dahdi-tools.x86_64                                              2.11.1-1.23.sng                                @sng-pkgs
dahdi-tools-doc.x86_64                                          2.11.1-1.23.sng                                @sng-pkgs
kmod-dahdi-linux.x86_64                                         2.11.1-9.sng7                                  @sng-pkgs
Available Packages
asterisk11-dahdi.x86_64                                         11.25.1-1.sng7                                 sng-pkgs
asterisk14-dahdi.x86_64                                         14.6.0-3.sng7                                  sng-pkgs
dahdi-firmware-a4a.noarch                                       a0017-1.3.sng                                  sng-pkgs
dahdi-firmware-a4b.noarch                                       d001e-1.3.sng                                  sng-pkgs
dahdi-firmware-a8a.noarch                                       1d0017-1.3.sng                                 sng-pkgs
dahdi-firmware-a8b.noarch                                       1f001e-1.3.sng                                 sng-pkgs
dahdi-firmware-hx8.noarch                                       2.06-1.3.sng                                   sng-pkgs
dahdi-firmware-oct6114-032.noarch                               1.07.01-1.3.sng                                sng-pkgs
dahdi-firmware-oct6114-064.noarch                               1.05.01-1.3.sng                                sng-pkgs
dahdi-firmware-oct6114-128.noarch                               1.05.01-1.3.sng                                sng-pkgs
dahdi-firmware-oct6114-256.noarch                               1.05.01-1.3.sng                                sng-pkgs
dahdi-firmware-oct6126-128.noarch                               01.07.04-1.3.sng                               sng-pkgs
dahdi-firmware-tc400m.noarch                                    MR6.12-1.3.sng                                 sng-pkgs
dahdi-firmware-te133.noarch                                     780019-1.3.sng                                 sng-pkgs
dahdi-firmware-te134.noarch                                     780017-1.3.sng                                 sng-pkgs
dahdi-firmware-te435.noarch                                     e0019-1.3.sng                                  sng-pkgs
dahdi-firmware-te436.noarch                                     10017-1.3.sng                                  sng-pkgs
dahdi-firmware-te820.noarch                                     1.76-1.3.sng                                   sng-pkgs
dahdi-firmware-vpmoct032.noarch                                 1.12.0-1.3.sng                                 sng-pkgs
[[email protected] dev]# modprobe wctdm24xxp
modprobe: FATAL: Module wctdm24xxp not found.
[[email protected] dev]#

and lspci -vv reports (amongst other things):

03:00.0 Communication controller: Tiger Jet Network Inc. Tiger3XX Modem/ISDN interface
        Subsystem: ATCOM Technology co., LTD. AX400P 4-port analog card
        Flags: bus master, medium devsel, latency 64, IRQ 11
        I/O ports at cc00 [size=256]
        Memory at fe7ff000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 2
        Kernel modules: netjet, hisax

Any other pointers?

That is not a supported card in DAHDI. You must of hand complied DAHDI at one point before for that card. Digium only provides DAHDi for Digium cards.

This card definitely worked out of the box with FreePBX 13 / CentOS 6.x and stock DAHDI. These are the instructions I got with the card, when I first got it, and they worked fine:

Here is the simple instruction as the card is Plug-&-Play installation once Dahdi has installed. In the Linux CLI; please run the below commands:
1) modprobe wctdm24xxp
2) service asterisk stop
3) service asterisk start
4) amportal restart
5) dahdi_genconf
6) dahdi_cfg -vvvv
7) lsdahdi (you should see all your ports)
8) reboot (reboot your system, & all the ports should be in service)

This is the card: http://www.nicherons.com/TDM410P.html

Not sure. I am not familiar with the card and only worked with Digium and Sangoma cards before. Maybe something changed in newer DAHDI to not support it.

Hi!

Can you do a

locate wctdm24xxp

and post the result here?

If it doesn’t work, issue a /etc/cron.daily/mlocate before…

Can you also issue a

uname -r

and once again post the results here?

I am no expert in the naming of modules relative to the kernel version but I am expecting a mismatch…

Somehow a weak-updates folder seems to fix things for me here (looks like it’s its purpose to do that), it is as if the mismatch got self corrected on my box?

Modprobing that module works on my system but I have an A200 so I cannot say if things actually work for a Digium card…

Good luck and have a nice day!

Nick

Thanks for the help, guys. Here’s the results of the tests:

[[email protected] dev]# locate wctdm24xxp
[[email protected] dev]# /etc/cron.daily/mlocate
[[email protected] dev]# locate wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp/wctdm24xxp.ko
[[email protected] dev]# uname -r
3.10.0-514.26.2.el7.x86_64
[[email protected] dev]#

Hi!

This is what I get

[[email protected] admin]# locate wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp/wctdm24xxp.ko
/usr/lib/modules/3.10.0-514.26.2.el7.x86_64/weak-updates/dahdi/wctdm24xxp
/usr/lib/modules/3.10.0-514.26.2.el7.x86_64/weak-updates/dahdi/wctdm24xxp/wctdm24xxp.ko

And those entries in weak-updates are symbolic links back to /usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/…

I think, like I said no guru in this, that there is some sort of kernel version mismatch…

I will post about it in the ticket I opened since it did deal with DAHDI issues after the update (amongst other things…).

Good luck and have a nice day!

Nick

If you feel adventurous and have nothing to lose, apparently running

weak-modules --add-kernel

would take care of adding those symlinks…

Only do this as a last resort, I tested this and it seems to works fine but less than an hour ago I didn’t know about this and I might be wrong about it…

Good luck and have a nice day!

Nick

Well, that worked :thumbsup:

[[email protected] dev]# weak-modules --add-kernel
[[email protected] dev]# locate wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp/wctdm24xxp.ko
[[email protected] dev]# /etc/cron.daily/mlocate
[[email protected] dev]# locate wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp
/usr/lib/modules/3.10.0-514.2.2.el7.x86_64/extra/dahdi/wctdm24xxp/wctdm24xxp.ko
/usr/lib/modules/3.10.0-514.26.2.el7.x86_64/weak-updates/dahdi/wctdm24xxp
/usr/lib/modules/3.10.0-514.26.2.el7.x86_64/weak-updates/dahdi/wctdm24xxp/wctdm24xxp.ko
[[email protected] dev]# modprobe wctdm24xxp
[[email protected] dev]#

[[email protected] dev]# lsdahdi
### Span  1: WCTDM/4 "Wildcard TDM400P REV I Board 5" (MASTER)
  1 FXO        FXSKS        RED
  2 FXS        FXOKS       (In use)
  3 FXS        FXOKS
  4 FXO        FXSKS        RED
[[email protected] dev]#

So, do you think the weak-modules command is a workaround, or is this what was meant to happen if the upgrade had worked correctly???

Many thanks for your help. Would never have got there without it!!!

Hi!

:smile:

I am glad…

I think it might depend on your exact definition of workaround…

To me it sounds like a workaround until those modules are compiled for that specific kernel, down to the fix/revision level…

Currently they are compiled for a kernel with the same kernel version, major revision and minor revision, the only difference is in the fix/revision level.

(In theory, something which should not affect module compatibility I guess…)

It’s quite possible running weak-modules (but automatically) might be the way to deal with this situation…

My upgrade didn’t work correctly… I was one of the first to try it and report back…

The difference might be that I had the previous kernel at one point in time (I do not know if I did) and it got updated which might have run weak-modules automatically…

In your case, maybe you never had that kernel so weak-modules never got run…

Take this with a BIG grain of salt, these are all suppositions, it’s been less than a day that I learned about this…

Good luck and have a nice day!

Nick