Queue Failover Destination Issue

I don’t understand why calls into my queue go to the queue’s failover destination after a few seconds.
Max Wait Time and Agent Timeout are set to unlimited.
The member of the queue does have a voice mail enabled but I don’t want a queue call to go there, only when it is directly called.
Any ideas?