Talking At Same Time Audio Cuts Out

I am using FreePBX with Twilio for SIP trunking and I am having an issue where if one side is talking at the same time as the other side of a call the audio cuts out some small parts can still be heard but almost all of the audio is cut out while talking at the same time. When using the echo test feature code if you talk continuously you can not hear the echo response except for small parts almost all of it gets cut out but if you say just one word it will work. When calling phone to phone over the trunk neither side can hear anything if both are talking at the same time. If you are talking while an announcement is playing the announcement will cut out. But there are no issues on either side if only one person is talking. There are also no issues if it is local call (extension to extension) that does not go over a SIP trunk. I have also tried a different SIP trunk (Nexmo) and different FreePBX servers (some with the FreePBX distro and some with a manual CentOS 7 install). I am using Vultr to host the FreePBX server.

What type of phone(s)? Have you tested to rule out the phone?

I have tested with two different cell phones both the cell phones can make other calls fine but when they call the SIP trunk it happens. But it does not happen when calling from the Nexmo trunk to the Twilio trunk only when a cell phone is calling the trunk. There is also around 300-500ms delay on the echo test when using a cell phone or trunk to trunk.