Filter-exchange imaging (FEXI) has already been utilized in several biomedical studies for evaluating the permeability of cell membranes. The method relies on suppressing the extracellular signal using strong diffusion weighting (the mobility filter causing a reduction in the overall diffusivity) and monitoring the subsequent diffusivity recovery. Using Monte Carlo simulations, we demonstrate that FEXI is sensitive not uniquely to the transcytolemmal exchange but also to the geometry of involved compartments: complex geometry offers locations where spins remain unaffected by the mobility filter; moving to other locations afterwards, such spins contribute to the diffusivity recovery without actually permeating any membrane. This exchange mechanism is a warning for those who aim to use FEXI in complex media such as brain gray matter and opens wide scope for investigation towards crystallizing the genuine membrane permeation and characterizing the compartment geometry.