Disk or Database Bottleneck

[comtech]

Brief: https://privatebin.net/?1e99499390c18451#A6mhwNCDbw5FLtq8JGJiVgePjrY7rqDD4J4pN5iwLt9W

Full: PrivateBin

Locks: PrivateBin

Thread1: PrivateBin

[comtech]

@dicko

perhaps /var/log/messages

That was it. grep asterisk messages

[billsimon]

Still looks like it’s waiting on your database.

Just for fun:

asterisk -rx "cel show status"

[comtech]

@billsimon

asterisk -rx “cel show status”

Oh how I wish

 # asterisk -rx "cel show status"
CEL Logging: Disabled

[comtech]

Should I try to batch CDR? I cannot imagine this is the issue with only 17 calls/ 32 channels at the high point.

CEL is a database hog with busy queues

New fields in Advanced settings with cdr module ver. 16.0.33 [image] For those not familiar with the feature, when enabled, asterisk doesn’t write cdr records in real time, instead is caches them and writes in batches. edit - found a bug. If the file /etc/asterisk/cdr.conf already exists, the content is lost when the symlink is created. Updating FREEPBX-24076

[dicko]

I would first look for misconfiguration or a bug in your Sangomaconnect, of which I have no knowledge or interest in.

[billsimon]

@comtech

Oh how I wish

Yeah, and sorry, that’s all I got. It just really has that “CEL is tying up the database” smell so I had to double check.

[comtech]

I have the module installed, but no licenses for this server. I can uninstall the module.

[dicko]

afs-app-p12 as a systemd service is suspiciously appley, no?

[dicko]

Then do that

[comtech]

Hahaha, that the server name, but no apple relation.

[dicko]

Please explain

[comtech]

• SangomaConnect 16.0.44.27 uninstalled

[dicko]

And afs-app-p12 stopped?

[comtech]

I don’t know what to do with this (out of my element).

afs-app-p12 shows up in every line in that column in the /var/log/messages file.

afs-app-p12 is the name of the server.
Asterisk Feature Server - Application - Production unit 12

[dicko]

Your logs show you have a systemd afs-app-p12.service that is running, find that file in it is an ExecStart line , also

systemctl status afs-app-p12

Will be helpful, (that is a very strange name for a server, how and where did you concoct it from?)

Edit: Perhaps I misinterpreted your messages file

[comtech]

@dicko

systemctl status afs-app-p12

# systemctl status afs-app-p12
Unit afs-app-p12.service could not be found.

@dicko

that is a very strange name for a server, how and where did you concoct it from?

I have an eccentric boss that likes what he likes.

[dicko]

Yeah, i led you wrong, sorry

[comtech]

All good, you’re right far more often and I appreciate the help.

[jcolp (Joshua Colp)]

The backtrace shows a lot of waiting on the underlying AstDB database access. Normally this is fast, but at the point in time the backtrace was taken an AMI connection was doing a database show and was blocking all the other access. AstDB is also used by the custom device state functionality, which was being queried from the dialplan at the time too and blocked waiting for access. There’s also some direct AstDB access from the dialplan that was blocked.

Either something on the FreePBX side is causing a slow down of handling of the responses and AMI connection connectivity, or the AstDB is having issues. It feels like underlying storage issues to me, but that’s just a gut feeling.