It’s actually normal for a few RTP packets to be sent to the remote private address. Of course, they don’t arrive. Your issue is that audio from the remote phone was not received by Asterisk. Once the first packet is received, Asterisk learns (from the source IP address and port), where it must send its RTP.
How it should work:
Phone sends INVITE to Asterisk, with SDP specifying its private address. Asterisk sees that the (public) source address of the INVITE does not match your NAT settings Local Networks, so it knows that the client is external.
When Asterisk sends a 183 Progress or 200 OK, it puts the value of External Address in the SDP, which of course should be the PBX public IP, and chooses a port from RTP Port Ranges (default is 10000 to 20000).
Phone starts sending audio to that address and port. The hardware router/firewall connecting the PBX has been configured to forward UDP ports 10000 through 20000 to the PBX LAN IP.
Asterisk sees incoming RTP and starts sending its RTP to the address and port from which the phone’s RTP came.
What’s likely to be wrong:
Your firewall is not correctly forwarding external RTP to the PBX. Or, your NAT settings are incorrect, such that the phone is not detected as being behind NAT, or the public IP address it is advertising is not correct.
Look at the 200 OK packet sent from the PBX and confirm that the IP address in the SDP is the PBX public IP. If so, your firewall is most likely failing to properly forward the RTP.