Remote SIP phones that work depending on my router

I’ve run into an odd question that I was wondering if anyone had any ideas about. On my desk, in my office, I have a SPA504 phone that is connected to a remote Asterisk system in another state. The connection to the company with the Asterisk server works or fails depending on what router I use at my end – the remote end. I find lots of info on how to configure the router protecting the Asterisk box …

but … what needs to be configured in the router at the REMOTE end?