Queue User Restrictions?

In our FreePBX system, occasionally an external phone number (I.E. - XXX-XXX-XXXX) will become logged in as an agent in one of our queues. Is there a way to restrict the queue log in to only certain agents without making them static? We have our queue set up to direct to voicemail if no agents are logged in, and when agents are logged in, it rings a roundrobin to only the currently logged in agents (no static agents). In order to keep our redirect to voicemail active, we cannot add static agents to the queue, but need to restrict the queue to only certain extensions, or only internal extensions.

-Thanks!