I 100% aggree with this post.
I had the same problems as OP running a phone through VPN. speedtest.net showed lots of bandwidth. ping showed low-(ish) latency. However, the issue was jitter.
I turned on jitter buffer (in softphone, in this case) and the problem was 100% solved, from terrible to terrific.