In fact, your assumption in the inverse of what is correct. If putting it on 5060 makes it work and moving it away from 5060 breaks it then something is mangling the traffic going through it, because it’s seeing 5060 and going ‘Oh, I know how to fix this’. And fixing it.
If you enable TLS you’ll have the same issue, because whatever’s doing the mangling won’t be able to mangle it any more.
Basically, fix your NAT settings in Asterisk.