Extreme latency when using follow-me

I have a customer experiencing a 3-5 second delay when they call a certain users extension that goes out using follow me.
We are using SIP trunks and The destination number is a pots line.
The strange thing is, that if you pick up the phone and dial the POTS line directly, there is only a slight delay.
This is on a freepbx distro VMware VM. Centos 5, asterisk 1.8.12, freepbx 2.10 latest updates installed.
Nothing obvious shows up in a packet capture.