So, the issue I was running into seems to be resolved by dialing *1 while on the intercom call.
My original post (One way audio between extensions)
I double-checked the setting in conferences and it’s set to not mute. I guess the question is why is it happening.