Bonus question: We saw this issue on Asterisk 20, and now also on 18.25… So I would think it’s not an Asterisk issue.
Sorry to say but I don’t think you have read the manual, navigate to a with the arrow keys then press enter to read it’s content
You should try it
Are there any other connections on netstat -tulpn, but with a large Send-Q?
Asterisk can lock up if AMI applications or PJSIP TCP endpoints keep connections but don’t process any data.
See the rest of the article. You are looking for confirmation that threads are waiting for locks, as a minimum. Ideally, you try and work out, from the names of the functions, where in the code it is waiting.
Once you have the output, and a reasonable case for there being a deadlock, you raise a bug report on github, including those logs and backtraces.
I’m sorry Dicko… not really sure what you want me to check in the content of the INVITE?
Today we had a “fresh” example… and ran the following command:
/var/lib/asterisk/scripts/ast_coredumper --running --no-default-search --pid=XXX
It only created 1 file: “/tmp/core-asterisk-running-2025-01-31T12-08-04Z”
When I try to read the dump using:
/var/lib/asterisk/scripts/ast_coredumper /tmp/core-asterisk-running-2025-01-31T12-08-04Z
… It showing this notification:
Examining /tmp/core-asterisk-running-2025-01-31T12-08-04Z
Does appear to be an asterisk coredump
Coredump indicates executable ‘/usr/sbin/asterisk’
Searching for asterisk module directory
Can’t extract asterisk module directory.
Either the executable ‘/usr/sbin/asterisk’ has no symbols
or it’s changed since the coredump was generated.
Either way we can’t use it. If you still have the
binary that created this coredump, or can recreate
the binary from the exact same code base and exact same
options that were used to to create the binary that generated
this coredump, specify its location with the
–asterisk-bin option.
No valid coredumps found
Any idea?
Negative… Only UDP socket for Asterisk:
udp 427776 0 0.0.0.0:5588 0.0.0.0:* 2254422/asterisk
navigate to an INVITE packet, press the enter key, the packet exchange will show as a series ofheaders which are red or green and go to or from the destination and the pbx, the first one will be the INVITE which on investigation (by pressing the enter key) will show the SDP negociation with ip addresses for you to match with your known network
Linux programs normally provide a ‘manual’ so try from a shell
man sngrep
If you are core dumping then that needs to be fixed before anything.
I know how sngrep works yes… and which headers are shown.
As I already tried to explain: The INVITE looks OK, but it’s not being “processed” by Asterisk.
where does the invite say its being sent to? do you have the same protocol and port listening where it was dispatched?, is that when you get a core dump?
Perhaps actually post the content of the INVITE so we can all agree that it is properly formed.
The INVITE is being sent to my FPBX machine. Protocol & port are indeed OK.
FYI: Nothing is being changed related to config… REGISTER packets are also not processed.
When I do a “core restart”, the same EXACT INVITE/REGISTER is again handled by Asterisk. (until the next random issue)
So please, don’t focus on the SIP packets.
What do you mean with “the core dump needs to be fixed first”? Asterisk is not crashing.
David suggested to run ast_coredumper for deadlocks.
OK, I am now unfocused.
1 Like
You need the debug symbols, see:
https://sangomakb.atlassian.net/wiki/spaces/SS/pages/31162494/Providing+Great+Debug
and, in particular, the heading “Backtraces (Segfaults/Core Dumps/Asterisk Crashing)”
Sorry, I don’t see what I need to specifically install on my Debian 12 machine.
The article looks outdated for Centos 7.
I would hope that they used the same package naming system.
You mean the “sangoma-devel” package? This one is not available…
Where is any output from the Asterisk console? Let’s see the output from a test call into the PBX.
From the system cli do asterisk -rvvvvvvvvv then when in the Asterisk console do pjsip set logger on then make a test call. Provide all output from the console.
Try without l flag to list connections and not just listen sockets: netstat -tupn
When I do a test call, I see the INVITE coming in (using sngrep/tcpdump)… but not in Asterisk cli / logger.
Problem seems the packet is “stuck” in the UDP socket… sitting in the kernel buffer (“Recv-Q”) waiting for Asterisk to read them.
I’m approaching this from an Asterisk point of view. I’d always build Asterisk with debug symbols available, but I don’t know how Sangoma have structured their FreePBX repositories, and I couldn’t quickly, find the repository, itself, so I can’t say what the package name is, or even whether the lack of one failed to get picked up by their QC.
I would assume that they set up the repository, as a source, and that the normal Debian package management tools will be able to list it, including brief descriptions of the package purposes.
Generally deadlocks that happen in commonly used code get picked up in QC, release candidate stage, or very soon after formal release, so, even without proper debug output, you should look to see if you are doing something that most users wouldn’t do, as that is where they may take longer to find and eliminate.