Queues and IVR Breakout with Direct Dial Exten

My company has just recently made the decision to switch to a Tiered Queue system for answering calls vs our current IVR. I have gotten the queues setup and working like i want with the exception of the IVR breakout. We would like to be able to say in the announcement message for the queue, if you know your parties ext please dial it at anytime. However it takes the user inputting the ext number twice to get it to go. Direct Dial is enabled on the IVR and there are no options for it setup except for direct dial. The first time they get the invalid entry try again. Then it works just fine.

I also tried flipping the way inbound works, by putting the IVR first and setting a short timeout with no retries to go directly to the queue. This works like we want it however once you hit the queue you run into the same problem when trying to dial an ext. Is there a way to prevent this that i am just missing? Thanks

btw this is on Freepbx v13