While cortical regions involved in processing binocular disparities have been studied extensively, little is known on how the human visual system adapts to changing disparity magnitudes. In this paper, we investigate causal mechanisms of coarse and fine binocular disparity processing using fMRI with a clinically validated, custom anaglyph-based stimulus. We make use of Granger causality and graph measures to reveal the existence of distinct rich and diverse clubs across different disparity magnitudes. We demonstrate that Middle Temporal area (MT) plays a specialized role with overlapping rich and diverse characteristics. Next, we show that subtle interhemispheric differences exist across various brain regions, despite an overall right hemisphere dominance. Finally, we pass the graph measures through the decision tree and found that the diverse clubs outperform rich clubs in decoding disparity magnitudes. Our study sets the stage for conducting further investigations on binocular disparity processing, particularly in the context of neuro-ophthalmic disorders with binocular impairments.