Enabling jitterbuffer between a SIP trunk and IVR

I have also read this topic but still couldn’t figure it out:

I would like to have the jitter buffer enabled only between the inboud trunk calls and IVR.

But when an external call from the trunk is further routed by IVR to an extenstion, I would like not to have the jitter buffer enabled anymore to avoid extra delay, as it will be handled by the Cisco phone itself.