Found the issue for those interested.
Invalid Retries has to be set to disabled.
Timeout retries to 0
That way, if someone doesn’t input anything on the IVR, it will failover to the set destination.
Thanks for the assistance. I hope this helps the community.