We’ve had a production level freepbx distro (all modules up to date and also all yum updates applied) running for about 1 month, no issues. Last week things went crazy.
Now several times per day extensions unregister and can no longer register. No incoming calls work either. The only solution Ive found is either a full system restart or a fwconsole restart.
Things that changed last week.
I activated the “Queues Pro” commercial module, but I have disabled and removed it from the system since. Removing it made no difference.
I played with asternic’s convert to MP3 settings.
I might have updated modules last week. I don’t remember.
So now, a few things I see (though I can’t be sure they weren’t there before)… I get the warnings about task processor being over 3000 all the time. I’ve checked and sometimes tasks have the queue column up to the hundreds of thousands. The system is a Xen Server virtual machine, with 4 cores and 16GB, but only about 100 extensions and Ive never seen more than 40 simultaneous calls.
Things seem to work normally and then the unregistration thing happens. I thought it was the queued up tasks, but Ive now seen the unregistration happen with queues at zero.
What I discovered now is that it seems mixmonitor starts recording and never stops. It also creates dozens of tiny 44byte files for each call.
My latest test is stopping all recordings and see if there are any problems, but I just made that change so nothing to report yet.
While in the CLI Ive also seen this FRACK and some backtracing, though this wasnt folliowed by the unregistration so no clue if this is related.
Any tips on what I should try to look for in order to debug this issue?
thank you!
-amco
PS.- problem just happened even though the call recordings are off, so maybe those 44 byte files are unrelated
some of the tasks that give warnings are:
subm:ast_channel_topic_all-cached-0000057b
subm:ast_channel_topic_all-00000583
subp:ast_channel_topic_all-00000895
subm:cel_aggregation_topic-00000006
this following one happens most often
subm:manager_topic-00000007 5818352 1650025 1663207 2700 3000
here is the frack i caught