I have seen a similar behavior over an IAX trunk between two systems here in the US. We ended up switching the trunk from IAX to SIP and saw immediate improvement in call quality. Could you try this?
So why IAX was introduced and why is all this hype im IAX2?
SIP is not an easy option for us. Our ITSPs block SIP protocols even if we change ports. But I never tried trunking both PBXs on SIP and port 21 for example.
I can bypass ITSP blocking by using VPN, but this ads some latency which I decided to use PBX on cloud to avoid VPN and latency.
SIP tends to work better in situations where TCP (and TLS as well) is used for signalling.
IAX uses UDP to communicate in binary both signalling and RTP data. If a packet gets dropped it could contain any number of parts from signalling and RTP data from one or multiple calls.