Multiple server call routing

I have three phone systems, all linked over a vpn many miles apart. All three phone systems have their own phone numbers. The IVR is on one phone system lets call main. Then we have office1 and office2.

When someone calls office1 it calls the IVR on main which is no problem it works great. However if they dial an option which then sends the call to office1 or office 2 the rtp traffic has to travel from office1 to main then back to office1 or office2 depending on the number dialed.
There is so much latency with sending the voice through these different legs that the audio starts to sound bad with laggy voice etc.

Is there a way that when someone calls in office1 or office2 then gets transferred back to office1 or office to the data doesn’t have to travel over the wan back and forth.

I believe what I’m looking for is reinvite, has anyone had experience doing this and will it work properly?

Currently:
office1---->Main---->Office1
or
office1---->Main---->office2

I want after they are transferred
office1---->office1
or
office1---->office2

Thanks