It should not be hard to see (from Asterisk’s point of view) what is going wrong, though it may not be easy to fix. As root, start a tcpdump capture of everything on the server, using a command like
tcpdump -s 0 -C 100 -W 100 -w rbuf -Z root &
This writes a circular buffer of 100 files (rbuf00, rbuf01, … rbuf99), each 100 MB long. Be sure that you have 10 GB of extra disk space available.
If there are ~100 active call legs @ ~ 80 kbps * 2 each, you’ll have ~1.4 hours of historical data. When trouble is reported, identify the time the call ended, find the file (based on modification time) containing the problem, copy it to your PC and analyze it with Wireshark.