So no consistency to CID/DID and no willingness to use an IVR. I would say there is no solution to this problem, unless you can do a whitelist like described here: Of Robocalls and Whitelists
That would force unrecognized callers to ‘press 1 to continue’ but after that they would be on the whitelist so it bypasses the challenge.