Upgrade process failed - stuck between 5.7 and 5.8

Hi All,
I have had trouble with upgrading to the latest Stable Asterisk, and I am now running an unhappy linux box and would appreciate help!

When I commenced the process, I was running Centos 64 Release 5.7 (Final) and Asterisk 1.8.9.3, and FreePBX 2.10.1.1

My CentOS machine was built via the distro downloaded from FreePBX.

My kernel version was: 2.6.18-274.3.1.el5

My issue was that I was affected by an asterisk bug in 1.8.9.3, resolved in 1.8.14 where a *2 transferred call would be dropped if the callee refused the call and hung up (when the call should have returned to the initial party who attempted the transfer).

So, in order to update, I followed the instructions and used the scripts here:

http://www.freepbx.org/forum/freepbx-distro/distro-discussion-help/asterisk-1-8-stable-release-versions-for-distro

I ran them, one at a time in order, from 1.889.210.57-2.

(I even ran them, first, in a test machine I had. It all went fine on the test machine. It did NOT go fine on the production machine!)

The problem arose during the upgrade in 1.812.210.58-1.sh This script, among other things, updates the underlying system from CentOS release 5.7 to Centos release 5.8 and upgrades the kernel to
2.6.18-308.4.1.el5-x86_64

This is where it all started to go wrong. The script downloaded 260M of updates, and finished without indicating an error, however it did not call for a reboot (as happened on my test machine when I ran the same script)
I rebooted anyway, and checked my Linux version, and was still at 5.7.

I tried an update manually:

Yum clean all
Yum update glibc* yum* rpm* python*
(this worked)
Yum update
I downloaded 260M of updates, and then had a dependency failure.

I then upgraded just the kernel
Yum update kernel-devel

This appeared to succeed, however when I rebooted using 2.6.18-308.4.1.el5 kernel, my system has no modules loading – no internet connection, no WanRouter, certainly no asterisk!

I have checked, and I have :
/usr/src/kernels/2.6.18-308.4.1.el5-x86_64/

I have
/lib/modules/2.6.18-274.3.1.el5/

But I DO NOT HAVE
/lib/modules/2.6.18-308.4.1.el5/

The new kernel is present and tries to load on boot:
Jul 29 13:49:44 PABX kernel: Linux version 2.6.18-308.4.1.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Apr 17 17:08:00 EDT 2012

However, this is an error appearing on startup:
Jul 29 04:48:01 PABX modprobe: FATAL: Could not load /lib/modules/2.6.18-308.4.1.el5/modules.dep: No such file or directory

I think that the underlying problem may be to do with dahdi and modules for my Sangoma A101 WanRouter card. (This card was not present in my test system, which successfully upgraded).
I am using the Wanpipe release 3.5.27 which is the latest stable release.

This appears in the logs from the attempted upgrade:
Jul 28 11:43:19 PABX kernel:
Jul 28 11:43:19 PABX kernel: wanpipe: WANPIPE Modules Unloaded.
Jul 28 11:43:19 PABX kernel: usbcore: deregistering driver sdlausb
Jul 28 11:43:19 PABX kernel: usbcore: deregistering driver xpp_usb
Jul 28 11:43:19 PABX kernel: dahdi: Telephony Interface Unloaded
Jul 28 11:43:45 PABX yum: Installed: kmod-dahdi-linux-2.6.0-2_centos5.2.6.18_274.3.1.el5.x86_64
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule '/etc/udev/rules.d/50-udev.rules:1’
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule ‘/etc/udev/rules.d/50-udev.rules:1’

The only way I can get my system to function correctly is to use the GRUB menu to boot into my earlier kernel - 2.6.18-274.3.1.el5.
Using this kernel, I get full system functions, however on reboot it attempts to load the new kernel, which fails.
I am not sure how to proceed from here?

My present system status:

CentOS release 5.7 (Final)
LSB Version: :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
uname -r
2.6.18-274.3.1.el5

Any help would be appreciated.

It sounds like you installed some RPM that is keeping the yum update from working with dependency issues. Do a yum update and post the results of that once it has run. Also the distro includes the wanpipe install so if you installed your own that might be conflicting. It is hard to say.

here is my yum log.
trouble started on July 28
No errors appear here, but I definitely saw dependencies errors (something about perl man I think. Would this be logged somewhere I can grab and post?)

Apr 07 14:35:57 Installed: sysadmin-2.5.1-1.noarch
Apr 07 14:36:02 Installed: schmooze-dahdi-1.0.0-2.noarch
Apr 07 14:36:42 Installed: wanpipe-3.5.24.4-kernel.2.6.18.274.3.1.el5.dahdi.2.5.0.1.x86_64
Apr 07 14:37:28 Installed: iSymphonyServer-fpbx-2.4.0_3025-1.noarch
Apr 21 18:36:16 Installed: 12:dhcp-3.0.5-29.el5_7.1.x86_64
Apr 23 21:55:22 Installed: tftp-0.49-2.el5.centos.x86_64
May 15 21:16:43 Installed: gdbm-1.8.0-26.2.1.el5_6.1.i386
May 15 21:16:44 Installed: bison-2.3-2.1.x86_64
May 15 21:16:44 Installed: flex-2.5.4a-41.fc6.x86_64
May 15 21:16:44 Installed: libtermcap-devel-2.0.8-46.1.x86_64
May 15 21:16:44 Installed: libtool-ltdl-devel-1.5.22-7.el5_4.i386
May 15 21:16:45 Installed: libtool-1.5.22-7.el5_4.x86_64
May 15 21:16:45 Installed: libtool-ltdl-devel-1.5.22-7.el5_4.x86_64
May 15 21:16:45 Installed: libtermcap-devel-2.0.8-46.1.i386
May 15 21:16:53 Installed: 4:perl-5.8.8-32.el5_7.6.i386
May 15 21:16:53 Installed: autoconf213-2.13-12.1.noarch
May 15 21:16:54 Installed: automake17-1.7.9-7.el5.2.noarch
May 15 21:16:54 Installed: bison-2.3-2.1.i386
May 15 21:16:54 Installed: flex-2.5.4a-41.fc6.i386
May 15 21:16:57 Installed: ncurses-devel-5.5-24.20060715.x86_64
May 15 21:16:58 Installed: automake15-1.5-16.el5.2.noarch
May 15 21:17:00 Installed: ncurses-devel-5.5-24.20060715.i386
May 15 21:17:00 Installed: automake16-1.6.3-8.el5.1.noarch
May 15 21:17:01 Installed: automake14-1.4p6-13.el5.1.noarch
Jul 28 11:40:15 Updated: freepbx-2.10.0rc1-1.noarch
Jul 28 11:40:24 Updated: iSymphonyServer-fpbx-2.4.0_3099-1.noarch
Jul 28 11:41:53 Erased: asterisk18-app_flite
Jul 28 11:42:52 Updated: asterisk18-core-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-addons-core-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-addons-mysql-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-addons-ooh323-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-voicemail-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-addons-bluetooth-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-curl-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-dahdi-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:52 Updated: asterisk18-doc-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:55 Updated: asterisk-sounds-core-en-gsm-1.4.21-1_centos5.noarch
Jul 28 11:42:55 Updated: asterisk18-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:55 Updated: asterisk18-addons-1.8.10.0-24_centos5.x86_64
Jul 28 11:42:57 Updated: asterisk-sounds-core-en-ulaw-1.4.21-1_centos5.noarch
Jul 28 11:43:00 Updated: asterisk-sounds-core-en-alaw-1.4.21-1_centos5.noarch
Jul 28 11:43:09 Installed: asterisk18-flite-debuginfo-1.8.10.0-24.x86_64
Jul 28 11:43:09 Installed: asterisk18-flite-1.8.10.0-24.x86_64
Jul 28 11:43:15 Installed: flite-devel-1.3-11.el5.x86_64
Jul 28 11:43:45 Installed: kmod-dahdi-linux-2.6.0-2_centos5.2.6.18_274.3.1.el5.x86_64
Jul 28 11:43:55 Updated: dahdi-linux-2.6.0-4_centos5.x86_64
Jul 28 11:43:55 Updated: dahdi-linux-debuginfo-2.6.0-4_centos5.x86_64
Jul 28 11:43:55 Updated: dahdi-linux-devel-2.6.0-4_centos5.x86_64
Jul 28 11:43:55 Installed: dahdi-linux-kmod-debuginfo-2.6.0-2_centos5.2.6.18_274.3.1.el5.x86_64
Jul 28 11:44:06 Updated: dahdi-tools-2.6.0-5_centos5.x86_64
Jul 28 11:44:06 Updated: dahdi-tools-doc-2.6.0-5_centos5.x86_64
Jul 28 11:44:07 Updated: dahdi-tools-debuginfo-2.6.0-5_centos5.x86_64
Jul 28 11:44:15 Updated: libtonezone-2.6.0-5_centos5.x86_64
Jul 28 11:44:16 Updated: libtonezone-devel-2.6.0-5_centos5.x86_64
Jul 28 11:44:23 Installed: libopenr2-debuginfo-1.2.0-1_centos5.x86_64
Jul 28 11:44:24 Installed: libopenr2-devel-1.2.0-1_centos5.x86_64
Jul 28 11:44:24 Installed: libopenr2-doc-1.2.0-1_centos5.x86_64
Jul 28 11:44:50 Installed: wanpipe-3.5.25-kernel.2.6.18.274.3.1.el5.dahdi.2.6.0.x86_64
Jul 28 11:45:15 Updated: freepbxdistro-header-2.10.0-5.noarch
Jul 28 11:45:16 Updated: fail2ban-0.8.6-9.noarch
Jul 28 11:49:32 Updated: asterisk18-core-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-addons-core-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-addons-mysql-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-addons-bluetooth-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-addons-ooh323-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-voicemail-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-curl-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:32 Updated: asterisk18-flite-debuginfo-1.8.11.0-26.x86_64
Jul 28 11:49:33 Updated: asterisk18-dahdi-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:33 Updated: asterisk18-doc-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:33 Updated: asterisk18-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:33 Updated: asterisk18-addons-1.8.11.0-26_centos5.x86_64
Jul 28 11:49:33 Updated: asterisk18-flite-1.8.11.0-26.x86_64
Jul 28 11:50:15 Updated: libtonezone-2.6.1-6_centos5.x86_64
Jul 28 11:50:16 Updated: dahdi-tools-debuginfo-2.6.1-6_centos5.x86_64
Jul 28 11:50:19 Installed: kmod-dahdi-linux-2.6.1-3_centos5.2.6.18_274.3.1.el5.x86_64
Jul 28 11:50:19 Updated: dahdi-linux-2.6.1-5_centos5.x86_64
Jul 28 11:50:26 Updated: wanpipe-3.5.26-kernel.2.6.18.274.3.1.el5.dahdi.2.6.1.x86_64
Jul 28 11:50:27 Updated: dahdi-tools-2.6.1-6_centos5.x86_64
Jul 28 11:50:27 Updated: dahdi-linux-kmod-debuginfo-2.6.1-3_centos5.2.6.18_274.3.1.el5.x86_64
Jul 28 11:50:27 Updated: dahdi-linux-debuginfo-2.6.1-5_centos5.x86_64
Jul 28 11:50:27 Updated: dahdi-tools-doc-2.6.1-6_centos5.x86_64
Jul 28 11:50:27 Updated: dahdi-linux-devel-2.6.1-5_centos5.x86_64
Jul 28 11:50:28 Updated: libtonezone-devel-2.6.1-6_centos5.x86_64
Jul 28 11:59:41 Updated: asterisk18-core-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:41 Updated: asterisk18-addons-core-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:41 Updated: asterisk18-addons-mysql-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-addons-bluetooth-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-voicemail-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-addons-ooh323-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-curl-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-flite-debuginfo-1.8.12.0-28.x86_64
Jul 28 11:59:42 Updated: asterisk18-doc-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-dahdi-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-1.8.12.0-28_centos5.x86_64
Jul 28 11:59:42 Updated: asterisk18-flite-1.8.12.0-28.x86_64
Jul 28 11:59:42 Updated: asterisk18-addons-1.8.12.0-28_centos5.x86_64
Jul 28 12:00:20 Updated: iSymphonyServer-fpbx-2.4.1_3314-1.noarch
Jul 28 12:03:11 Installed: kernel-2.6.18-308.4.1.el5.x86_64
Jul 28 12:03:16 Updated: wanpipe-3.5.26-kernel.2.6.18.308.4.1.el5.dahdi.2.6.1.x86_64
Jul 28 12:03:19 Installed: kmod-dahdi-linux-2.6.1-8_centos5.2.6.18_308.4.1.el5.x86_64
Jul 28 12:50:04 Updated: asterisk18-core-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:04 Updated: asterisk18-addons-core-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:04 Updated: asterisk18-addons-mysql-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:04 Updated: asterisk18-addons-bluetooth-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:04 Updated: asterisk18-voicemail-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-addons-ooh323-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-curl-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-flite-debuginfo-1.8.13.0-30.x86_64
Jul 28 12:50:05 Updated: asterisk18-doc-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-dahdi-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-addons-1.8.13.0-30_centos5.x86_64
Jul 28 12:50:05 Updated: asterisk18-flite-1.8.13.0-30.x86_64
Jul 28 13:10:48 Updated: glibc-common-2.5-81.el5_8.2.x86_64
Jul 28 13:10:58 Updated: glibc-2.5-81.el5_8.2.x86_64
Jul 28 13:10:58 Updated: popt-1.10.2.3-28.el5_8.x86_64
Jul 28 13:10:58 Updated: expat-1.95.8-11.el5_8.x86_64
Jul 28 13:11:00 Updated: glibc-headers-2.5-81.el5_8.2.x86_64
Jul 28 13:11:00 Updated: nscd-2.5-81.el5_8.2.x86_64
Jul 28 13:11:01 Updated: glibc-devel-2.5-81.el5_8.2.x86_64
Jul 28 13:11:04 Updated: glibc-2.5-81.el5_8.2.i686
Jul 28 13:11:04 Updated: expat-1.95.8-11.el5_8.i386
Jul 28 13:11:04 Updated: python-2.4.3-46.el5_8.2.x86_64
Jul 28 13:11:05 Updated: rpm-libs-4.4.2.3-28.el5_8.x86_64
Jul 28 13:11:06 Updated: rpm-4.4.2.3-28.el5_8.x86_64
Jul 28 13:11:11 Updated: python-libs-2.4.3-46.el5_8.2.x86_64
Jul 28 13:11:11 Updated: rpm-python-4.4.2.3-28.el5_8.x86_64
Jul 28 13:11:15 Updated: yum-3.2.22-39.el5.centos.noarch
Jul 28 13:11:15 Updated: yum-fastestmirror-1.1.16-21.el5.centos.noarch
Jul 28 13:11:15 Updated: yum-kmod-1.1.16-21.el5.centos.noarch
Jul 28 14:55:35 Updated: asterisk18-core-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:35 Updated: asterisk18-addons-core-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-addons-mysql-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-voicemail-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-addons-bluetooth-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-addons-ooh323-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-curl-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-flite-debuginfo-1.8.14.0-31.x86_64
Jul 28 14:55:36 Updated: asterisk18-doc-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:36 Updated: asterisk18-dahdi-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:37 Updated: asterisk18-1.8.14.0-31_centos5.x86_64
Jul 28 14:55:37 Updated: asterisk18-flite-1.8.14.0-31.x86_64
Jul 28 14:55:37 Updated: asterisk18-addons-1.8.14.0-31_centos5.x86_64
Jul 28 15:10:21 Installed: wanpipe-3.5.27-kernel.2.6.18.308.4.1.el5.dahdi.2.6.1.x86_64
Jul 28 22:00:37 Installed: kernel-devel-2.6.18-308.4.1.el5.x86_64

I would try a yum reinstall kernel* and see if that fixes your kernel issue.

Also what happens if you do a yum update now. Before doing that do a yum clean all to clear the cached repo data.

If I do reinstall kernel while running the older kernel, won’t it re-install the older one (the good one)?

I can’t boot into the new kernel then do it, as I have no modules running, so no internet access when running the new kernel.

can I do something like: yum reinstall kernel 2.6.18-308.4.1.el5

Thanks for your help so far.

(also, I will try the ‘yum update all’ tonight, after business hours. I have actually tried this more than once, but I will try and capture the errors so we can work out what to do about it. I guess i will have to remove the package(s) causing the update to fail, however I am afraid of ending up with a non-working freepbx system afterwards…)

I ran yum clean all then yum update, and got errors pointing to a conflict with perl 64 bit, vs perl 32 bit

file /usr/share/man/man1/perldsc.1.gz from install of perl-5.8.8-38.el5.x86_64 conflicts with file from package perl-5.8.8-32.el5_7.6.i386

so I removed the offending package (32 bit)

yum remove perl-5.8.8-32.el5_7.6.i386

yum update then successfully updated, and I got to 5.8

cat /etc/*release

CentOS release 5.8 (Final)

tried reinstall kernel, yum said ‘nothing to do’

[[email protected] ~]# yum reinstall kernel-2.6.18-308.4.1.el5.x86_64
Loaded plugins: fastestmirror, kmod
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Setting up Reinstall Process
Loading mirror speeds from cached hostfile
Package kernel-2.6.18-308.4.1.el5.x86_64 is allowed multiple installs, skipping
Nothing to do

did a yum clean all, and tried again

this claimed to work:
Installed:
kernel-devel.x86_64 0:2.6.18-308.11.1.el5

Complete!

This is a marginally later version kernel than I got when I tried the other day.

however, it does not exist as far as Grub is concerned, which only recognises the older (working) kernel, and the one I have been having trouble with.

Rebooting into the new kernel - same trouble as described above.

I am convinced that the problem relates to the Sangoma Wanrouter hardware - which I manually installed originally, using the Sangoma instructions on their wiki : http://wiki.sangoma.com/wanpipe-linux-asterisk-dahdi

I then recompiled the drivers for (both) new kernels, which created the directories for the kernels under /lib/modules which I did not previously have (ie /lib/modules/2.6.18-308.4.1.el5/ mentioned in my first post, and which the machine complained about in the error message above:
PABX modprobe: FATAL: Could not load /lib/modules/2.6.18-308.4.1.el5/modules.dep: No such file or directory )

I thought I would be fine, but on reboot, same error. Turns out that, while the directory was created, there is MUCH less content in the directory than in the directory associated with the working kernel. In particular, the specific file - modules.dep is not present in the new directory, even though it was created when I recompiled the drivers)

So, presently:
I now have OS Release 5.8.
I still cannot use the later kernels.
I can still use the initial kernel 2.6.18-274.3.1.el5
I still get these errors, even with the working kernel:

Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid KERNEL operation
Jul 28 11:43:53 PABX udevd[606]: add_to_rules: invalid rule ‘/etc/udev/rules.d/50-udev.rules:1’

So, I have a system that works, kinda.

I have modified my grub.conf so that the system defaults to the working kernel on reboot, until I can work out what else to do?

I now have it all working. Hooray!

I am now running:
cat /etc/*release
CentOS release 5.8 (Final)
uname -r
2.6.18-308.11.1.el5

I also have some theories about what went wrong. Actually, I think it was several things. Unfortunately I have tried so many things that it is difficult to nail down exactly what was done that helped, and what made things worse.

I will note this up really for the benefit of others who may experience something similar.

These are my conclusions:

1/ If you have a Sangoma hardware card (in my case A101-2), you need to recompile the wanrouter driver after a kernel upgrade.
2/ You probably also need to recompile your Dahdi first (although running the ./Setup from within /usr/src/wanpipe-3.5.27/ dir will probably recompile it anyway)
3/ When you do a kernel upgrade, use: yum upgrade kernel* so that you also get the headers and sources. Without these, your wanrouter will not compile. (* I am pretty sure this was 90% of my problem - I did not have all the sources so could not compile properly)

3a/ you will get errors when you run the kernel upgrade, because, until you recompile your wanrouter the modules are incompatible. eg:
WARNING: /lib/modules/2.6.18-308.11.1.el5/kernel/drivers/net/wan/wanpipe.ko needs unknown symbol dahdi_hdlc_putbuf
WARNING: /lib/modules/2.6.18-308.11.1.el5/kernel/drivers/net/wan/wanpipe.ko needs unknown symbol _dahdi_ec_span
WARNING: /lib/modules/2.6.18-308.11.1.el5/kernel/drivers/net/wan/wanpipe.ko needs unknown symbol dahdi_alarm_notify

This problem is resolved after you reboot into the new kernel and re-run the wanrouter setup script. (run the ./Setup from within /usr/src/wanpipe-3.5.27/)

4/ It is MUCH easier if you boot into the new kernel before trying to recompile the wanrouter/dahdi for the new kernel - I recommend NOT attempting to recompile for kernel B while still running kernel A (even though you can).

Some lesser conclusions, mainly things I think I broke while flailing around trying to fix this:

5/ when you use the freepbx upgrade scripts, they seem to insert a FreePBX repo definitions file for yum: /etc/yum.repos.d/FreePBX.repo
However, I had also in this directory: /etc/yum.repos.d/CentOS-Base.repo
both of these files had definitions for [base] [updates] [extras][centosplus] in them, so yum complained:
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration

I resolved this by renaming /etc/yum.repos.d/CentOS-Base.repo to /etc/yum.repos.d/CentOS-Base.repo.disabled

I am not certain what, if any, issue this error was causing, however it seems to me that I may have been using repos giving me packages that were incompatible with FreePBX.

6/ when troubleshooting kernel updates, it seems a good idea to update everything else first, so when you go for the big one, you aren’t having dependency issues or something which throws everything out.

7/ Watch out for i386 packages on 64 bit systems - they can give you conflicts. I ended up removing only the specific conflicting package I had (see earlier post). However, I have since read here: http://www.asteriskdocs.org/en/3rd_Edition/asterisk-book-html-chunk/Installing_id291070.html#Installing_id291082
that I should remove them all by running: yum remove *.i386 on my 64 bit system. (I haven’t - too scary on a running system - would have on initial setup if I had read this back then)

8/ (perhaps this should be number 1) Ask yourself, do you REALLY need to upgrade your kernel?! Then, channelling your inner Dirty Harry: Ask yourself one question: Do I feel lucky?
Well, do ya…punk?