Try searching for “early media” and see if it helps. There are lots of threads about using it, and it may help you in some of your cases.
You might be running into a problem with the exterior router(s) losing their NAT sessions. The “doesn’t work after a long pause” thing is common in that scenario. Make sure, also, that your exterior routers are set up to forward NAT traffic correctly. You may or may not “need” this to be done, but it is a belt-and-suspenders precaution you can take to possibly prevent some of the edge cases out there.