Fail2Ban not respecting Whitelist

New install of Distro Version 6.12.65-28 Asterisk 11.18. I set several external FQDN and IP addresses in the whitelist. With the service running it blocks all traffic SIP. I have to stop it for SIP calls to be completed. With it running we get blocked.