Segmentation Fault (Core Dumped) Asterisk 1.8

Hi,

I am using FreePBX version 2.9.0.7 and have just upgraded Asterisk from version 1.4.24, (which was running with no issues), to the latest Asterisk version 1.8.7.1, by following instructions given here: (up to the end of the section dealing with Music on Hold).

http://www.powerpbx.org/content/rhel-asterisk-freepbx-install-guide-rhel-v6-asterisk-v18-freepbx-v29%20

The reason for doing so is to enable T.38 termination for the Digium Free Fax to Asterisk application, which was only introduced in version 1.6.

Although the upgrade was apparently successful, (according to YUM update), Asterisk now won’t start. I then downloded the RPM files and re-installed Version 1.8.7.1 using the instructions here:

http://www.freepbx.org/support/documentation/howtos/how-to-upgrade-asterisk

This has made no difference and I would be VERY grateful if someone might be able to help point me in the right direction to solve this problem.

The system is a CentosOS based FreePBX distro and FreePBX has been regularly updated.

The error output is as follows:

[root@pbx ~]# amportal start

SETTING FILE PERMISSIONS

Permissions OK

STARTING ASTERISK
/usr/sbin/safe_asterisk: line 145: 9737 Segmentation fault (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
/usr/sbin/safe_asterisk: line 145: 9774 Segmentation fault (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}
Asterisk ended with exit status 139
Asterisk exited on signal 11.
Automatically restarting Asterisk.
mpg123: no process killed


Asterisk could not start!
Use ‘tail /var/log/asterisk/full’ to find out why.


The logfile referenced shows no apparent errors:


[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘from-paul-custom’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-queues’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-on’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-off’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-group’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-busy-off-any’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-fax’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-callwaiting-cwon’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-vmmain’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-calltrace’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-speeddial’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-speakextennum’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-queue-toggle’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-pbdirectory’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-gabcast’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘parkedcalls’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-dnd-toggle’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-dnd-off’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-dictate-send’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘grps’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-dialvm’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘fmgrps’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-unavailable-on’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-test’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-toggle’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-meetme’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-off-any’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-local’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-busy-on’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘ext-findmefollow’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-busy-off’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-zapbarge’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-callwaiting-cwoff’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-userlogonoff’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-blacklist’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-speakingclock’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-recordings’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-pickup’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-fmf-toggle’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘vmblast-grp’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-echo-test’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-dnd-on’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-directory’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-dictate-record’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘from-internal-custom’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-chanspy’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-cf-unavailable-off’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘outrt-1’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘outrt-7’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘outrt-2’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘outrt-10’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘outrt-4’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-miscapps’ in context ‘from-paul’
[Oct 19 01:53:11] VERBOSE[10352] pbx.c: – Including context ‘app-fax’ in context ‘from-paul’

When running: asterisk -vvvvvvvvgc the console output shows much the same, terminating with the following message (which is not actually logged)

Segmentation fault (core dumped)

The only thing I notice is that the precise failure occurs at a slightly different point in the loading sequence each time it is run - so it doesn’t appear connected with the contexts.

Kind regards,

Andy Woolford

Looking further back through the startup logs I see the following errors, although they occur way before the segmentation fault occurs, which is why I missed them the first time.

[Oct 19 08:57:31] ERROR[20632] res_calendar.c: Unable to load config calendar.conf [Oct 19 08:57:31] VERBOSE[20632] loader.c: res_ael_share.so => (share-able code for AEL) [Oct 19 08:57:31] WARNING[20632] loader.c: Error loading module 'format_au.so': /usr/lib/asterisk/modules/format_au.so: cannot open shared object file: No such file or directory [Oct 19 08:57:31] WARNING[20632] loader.c: Module 'format_au.so' could not be loaded. [Oct 19 08:57:31] WARNING[20632] loader.c: Error loading module 'res_pktccops': /usr/lib/asterisk/modules/res_pktccops.so: cannot open shared object file: No such file or directory [Oct 19 08:57:31] WARNING[20632] loader.c: Error loading module 'chan_mgcp.so': /usr/lib/asterisk/modules/chan_mgcp.so: undefined symbol: ast_pktccops_gate_alloc [Oct 19 08:57:31] WARNING[20632] loader.c: Module 'chan_mgcp.so' could not be loaded. [Oct 19 08:57:31] VERBOSE[20632] config.c: == Parsing '/etc/asterisk/res_mysql.conf': [Oct 19 08:57:31] VERBOSE[20632] config.c: == Found [Oct 19 08:57:31] WARNING[20632] config.c: parse error: No category context for line 1 of /etc/asterisk/res_mysql.conf [Oct 19 08:57:31] ERROR[20632] res_config_mysql.c: Not loading res_config_mysql.conf

Looking through the /usr/lib/asterisk/modules directory I note that the following module files mentioned above are missing:

/usr/lib/asterisk/modules/format_au.so
/usr/lib/asterisk/modules/res_pktccops.so

Would this cause the segmentation error, and if so - do I need to download and install the missing files, or configure asterisk to ignore them somehow?

I tried adding the following to the /etc/asterisk/modules.conf file:

noload => format_au.so
noload => res_pktccops.so

It seems to have cured the logged errors - but makes no difference to the Segmentation fault. Asterisk still fails to run.

I must admit I’m stumped at this point. I’m not a programmer - and I need to be walked through any further diagnostics if anyone can spare the time.

Many thanks,

Andy Woolford

No diagnostics, did you do the 'make clean" as the first step?

Did you choose the right option in ‘make menuselect’?

Yum packages that are not part of a distro are dangerous. They are built against a different system, that is what causes the crash when it tries to link to the wrong drivers. I am sure many of your Asterisk modules are wrong also.

Chan_dahdi is probably equally messed up.

Best course of recovery is to download source files from digium and rebuilt Asterisk/Dahdi from source.

This does not require programming skills, it does require system administration knowledge.

If you are unsure why not buy FreePBX support time? We can take care of this for you in an hour or so and check the system in general.

Thanks for this.

Yes I did all that - including the Asterisk re-build from digium source, although I’m afraid I don’t know what ‘Dahdi’ is… Still - I can always download it and run a clean install.

I’ve actually got it running this morning after changing the /etc/asterisk/res_mysql.conf file. It turned out this only contained the sql username and password but seemed to be missing all of the FreePBX settings which I have just copied in from here: http://www.freepbx.org/configuration_files

But you’re right; things are a bit of a mess. Free Fax for Asterisk seems to have been replaced by something called Spandsp although the res_fax_digium.conf files are still present. I was thinking of downloading and reinstalling FFA, but not sure how that would affect the FFA free channel licence? Currently when I ring any internal extension which has fax enabled - it goes straight to fax tones instead of waiting for a faxdetect.

Also, music on hold is broken because it seems Zaptel and Zapdummy are missing.

There may be other problems too which are lurking.

I think I’ll try to press on myself with downloading and re-installing FFA and Zaptel, but if I’m still struggling I may well appreciate a bit of paid help. This is just a private project at the moment, so I’m very conscious of costs, (hence why I went open source in the first place) . Anyway, please advise of the “going rate” and how I can raise a ticket if I need to.

Kind regards,

Andy Woolford

Andy Woolford

Zaptel was renamed DAHDI three years ago for licensing issues. If you are running Zap with 1.8 that is half of your problem. You need DAHDI also.

I would not even attempt to load FFA with your system in the state it is in.

SpanDSP is an opensource package that provides similar functionality to FFA. FreePBX supports both. SpanDSP is included with the distro and works great.

If faxing is your main goal you are going to get all this done and still not have access to our Fax Pro module because you are not running the distro.

Okies…

Sold. :slight_smile:

So where do I need to go to raise a support ticket with you?

It’s not a sales pitch, please don’t feel pressured.

Just offering you one solution.

As I mentioned if you clear everything out and build DAHDI and then Asterisk (no yum at all) you should be in good shape.

If you want to use support setup SSH access to the system from an outside IP and click on the support link at the top of the page. If you want me to look at it, I will be available @ 19:00 EST.

Regards…Scott

Thank you. No pressure - honest. :slight_smile:

I’ll see how far I get and let you know. My machine is in Florida (although I am in the UK) and I have remote SSH access set up already, so it would be easy to do.

I don’t want to clear EVERYTHING out, because I have a fair number of customised .conf files in the /etc/asterisk directory and FreePBX was purring along like a kitten under asterisk 1.4. (before I went and messed it up…)

Kind regards,

Andy Woolford

Ok, is there someone in Florida to reboot and read the console if necessary?

If it is in a previous post, my apologies but what is the CentOS version and exact Kernel version (uname -r)

A kernel upgrade might be the pace to start.

Here is the pre-compile portion of my installs script, it makes sure all dependencies are resolved:


yum -y install kernel-devel.i686
yum -y install kernel-headers.i386
#
# Install Build Environment
#
yum -y install rpm-build newt-devel ncurses-devel ibusb-devel gcc* net-snmp* bison*
#
# This next group will build 99% of the packages on the net.
#
yum -y install  subversion  mlocate  openssl openssl-devel perl perl-Net-SSLeay perl-Crypt-SSLeay libtermcap-devel ncurses-devel doxygen curl-devel newt-devel mod_ssl crontabs vixie-cron speex speex-devel unixODBC unixODBC-devel libtool-ltdl libtool-ltdl-devel mysql-connector-odbc mysql mysql-devel php php-cli php-common php-gd php-pdo php-mbstring php-mcrypt flex screen  mod_ssl mysql mysql-devel mysql-server build-essential libxml2 libxml2-devgcc gcc-c++ wget bison php-mysql php-pear php-pear-DB php-mbstring make ncurses-devel libtiff-devel php-gd audiofile-devel gtk2-devel
#
# Install Scott's Editor
yum -y install joe
#
#
# Install Management Framework
#

yum -y install snmp*

#
# Update MySQL
#
yum -y install mysql*
#

For the next 6 hours yes… Then he’s coming over here.

Thanks for the scripts. Can I just run each one as-is? I recognise some of it as already installed.

uname -r

2.6.18-238.19.1.el5

and

cat /etc/redhat-release

CentOS release 5.6 (Final)

Regards,

Andy

If it makes any odds, this is also the CPU info:

vendor_id : GenuineIntel
cpu family : 6
model : 28
model name : Intel® Atom™ CPU 330 @ 1.60GHz

You can run as is, it will pickup any missing packages.

6 hours, tight timeline. I will be checking the boards in between meetings today. Let me know how you make out.

I’m sorry I’ve been running around town today as well so I wasn’t able to get back to you. You’re very kind - thankyou for your help :slight_smile:

I’m about to start the scripts now so I’ll post back here how it went.

Regards,

Andy

Hi Scott,

Here is a summary of the results: (I’ve only copied below what seem to be errors)

Thank you very much for providing this script. :slight_smile:


yum -y install rpm-build newt-devel ncurses-devel ibusb-devel gcc* net-snmp* bison*

No package ibusb-devel available.

yum -y install subversion mlocate openssl openssl-devel perl perl-Net-SSLeay perl-Crypt-SSLeay libtermcap-devel ncurses-devel doxygen curl-devel newt-devel mod_ssl crontabs vixie-cron speex speex-devel unixODBC unixODBC-devel libtool-ltdl libtool-ltdl-devel mysql-connector-odbc mysql mysql-devel php php-cli php-common php-gd php-pdo php-mbstring php-mcrypt flex screen mod_ssl mysql mysql-devel mysql-server build-essential libxml2 libxml2-devgcc gcc-c++ wget bison php-mysql php-pear php-pear-DB php-mbstring make ncurses-devel libtiff-devel php-gd audiofile-devel gtk2-devel

No package build-essential available.
No package libxml2-devgcc available.


Most packages were already installed and up to date, although a number of packages were selected to be updated. This failed the first time when a dependancy failed as follows:


—> Package speex-devel.i386 0:1.0.5-4.el5_1.1 set to be updated
–> Processing Dependency: speex = 1.0.5-4.el5_1.1 for package: speex-devel
–> Finished Dependency Resolution
speex-devel-1.0.5-4.el5_1.1.i386 from base has depsolving problems
–> Missing Dependency: speex = 1.0.5-4.el5_1.1 is needed by package speex-devel-1.0.5-4.el5_1.1.i386 (base)
Error: Missing Dependency: speex = 1.0.5-4.el5_1.1 is needed by package speex-devel-1.0.5-4.el5_1.1.i386 (base)
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
package-cleanup --dupes
rpm -Va --nofiles --nodigest

(I re-ran the above using --skip-broken)

Packages skipped because of dependency problems:
speex-devel-1.0.5-4.el5_1.1.i386 from base


A further 25 packages were then successfully updated.

yum -y install joe

Installed:
joe.i386 0:3.4-3

yum -y install snmp*

Installed:
snmpbrowser.i386 0:0.4-2.el5.rf snmpbrowser-debuginfo.i386 0:0.4-2.el4.rf

yum -y install mysql*

Installed:
mysql-bench.i386 0:5.0.77-4.el5_6.6 mysql-test.i386 0:5.0.77-4.el5_6.6

Obviously there were a couple of errors throughout, although I don’t know how significant they are.

I then sent the system down for a reboot and it seems to have come back up okay.

This process has fixed a yellow warning which was on the Op Panel, but unfortunately nothing else has changed. MOH is still broken and we are getting no progress tones (e.g. no ring tones). I guess I need to get Dahdi installed or updated next.

Dahdi is now up to date (2.5.0.1) and running.

But still no MOH… Still no ringtones come back from any extension, (although it does actually ring) and FFA is a bit of a mystery.

MOH: I am getting this error in the logfile:

res_musiconhold.c: Request to schedule in the past?!?!
(multiple times)

I thought it was Dahdi, but I’m assuming that’s okay now so something else may be broken… Also the audio is decidedly choppy. Maybe something is eating processor cycles?

My MOH is based on the mpg123 application which seems to be running, (as I have some custom streams in the /etc/asterisk/musiconhold_custom.conf file). “default” is set as a custom stream. But the streams don’t play and the test extension which I have set up just hangs up at step 3 below - which it never used to under Asterisk 1.4.

This is defined in the /etc/asterisk/extensions_custom.conf file:

exten => 500,1,Answer
exten => 500,2,Playback(pls-hold-while-try)
exten => 500,3,SetMusiconHold,default
exten => 500,4,WaitmusiconHold(60)
exten => 500,4,Hangup

FFA:I (re)installed FFA using the “install” feature on the Digium Addins page in FreePBX. But I have no idea what the Fax configuration is doing now. FFA comes up as installed - but so does Spansp.

Under the asterisk CLI:

pbx*CLI> fax show capabilities

Registered FAX Technology Modules:

Type : Spandsp
Description : Spandsp FAX Driver
Capabilities : SEND RECEIVE T.38 G.711

Type : DIGIUM
Description : Digium FAX Driver
Capabilities : SEND RECEIVE T.38 G.711 MULTI-DOC

2 registered modules

and

pbx*CLI> fax show licenses

Fax Licensing Information

Free fax licenses: 1
Total licensed ports: 1

Licenses Found:
File: FFA-MM5JN78MBUDR.lic – Key: FFA-MM5JN78MBUDR – Product: RESFFA – Host-ID: ba:5d:48:21:75:3c:9c:c5:53:e8:4d:13:4f:58:15:fd:f0:4a:39:8a – Ports: 1 (OK)

Any extension designated as a fax extension just answers with fax tones to any call without (apparently) waiting for NVfaxdetect. Furthermore, it doesn’t actually seem to receive a fax, although it makes all the right noises. (granted I am using Zoiper to try to send a test fax without success - so this might be a red herring). But the “instant-answer” problem means it can’t be used as a normal extension without disabling the fax function FreePBX. I just don’t know enough about linux to debug this on my own - although I’m quite enjoying the experience!

I don’t mind pressing on with your kind help - but fully understand if you’d rather not.

Kind regards,

Andy

You are throwing an awful low into the mix.

I have never configured streaming music, my guess is you didn’t turn on MP3 support in menuselect so the modules are not built.

You must have built SpanDSP, you will have to look at the logs to see which one FreePBX picks.

I have no experience with fax detect. I always use DID’s. T.38 gateway support in Asterisk 1.10 is going to be a deal changer. Look for a bunch of activity. I will be out at Astricon next week and learning about all the new Fax functionality is high on my list.

I’m real mixed up generally ;-p.

Because of my lack of Linux general knowledge, I’m not real good at installing and re-installing things. Yum seemed simple enough, but I just followed the instructions I found through g**gle on the principle it must have worked for someone - and that’s what got me into trouble in the first place!

Can you give me a tip on how to remove Spansp and leave FFA in place?

How do I check to see if the MP3 modules are built? If they are not, can I simply re-install asterisk 1.8 with the appropriate options? (I thought I’d set yes to everything).

Good luck at the show.

MOH has just sprung back to life. Weird or what!

Okay, we can take it out of the weird category and put it in the Dumb category. The reason I couldn’t get SetMusiconHold working on extension 500 is that the command is deprecated in Asterisk 1.8.

Instead of:

exten => 500,3,SetMusiconHold,default

we need…

exten => 500,3,Set(CHANNEL(musicclass)=default

Frankly I am lost and where you are at. If you download Asterisk source from Digium you don’t get SpanDSP.

If you are sill using yum then the whole ‘make’ etc. discussion is moot.

What kind of Ground Training do you do? I will setup your server and trade support if you can arrange CE-500 initial training for me.