Some codecs working only one-way

The codecs in question are iLBC and G726

Here is the breakdown. My server is configured to allow ULAW,ALAW,iLBC, and G726

If I set my phone to allow only ULAW and place a call to another extension, the communication is both-ways and works fine.

If I disallow ULAW and allow either iLBC or G726 and place a call to the same other extension, I get functional audio only from me to that extension and hear nothing back. If I call the “directory” virtual extension on the server, I do not hear anything, but the phone is answered.
If I call from the other extension, to my phone, I get two way communication. My phone shows that it is on iLBC although I am not entirely sure if I can trust that. I think it might just show iLBC but use ULAW or whatever instead.

Let me know if anyone can shine a light on it.

btw My server is FreePBX 12.0.65 with Asterisk 13. I actually have access to two deployments in different locations and both act the same.