Send Directory (dial your party's extension and dial by name) to voicemail for some users but ring for others

I have a typical IVR that says “if you know your party’s extension you may dial it now” and also press # for a dial by name directory. I am using the Directory feature to configure the list of permitted extensions, I have about 20 phones that I’d like to ring and then go to voicemail (i.e. behave normally) and about 40 extensions that I’d like to send straight to voicemail (plus some other phones that are not in the directory at all). The 40 extensions need to ring normally for internal calls. So only calls through the IVR from the outside go directly to voicemail. Is there a way to do this?

I am using FreePBX