The human brain has been documented to be spatially organized in a finite set of specific coherent patterns, namely resting state networks (RSNs). The interactions among RSNs, being potentially dynamic and directional, may not be adequately captured by simple correlation or anticorrelation. In order to evaluate the possible effective connectivity within those RSNs, we applied a conditional Granger causality analysis (CGCA) to the RSNs retrieved by independent component analysis (ICA) from resting state functional magnetic resonance imaging (fMRI) data. Our analysis provided evidence for specific causal influences among the detected RSNs: default-mode, dorsal attention, core, central-executive, self-referential, somatosensory, visual, and auditory networks. In particular, we identified that self-referential and default-mode networks (DMNs) play distinct and crucial roles in the human brain functional architecture. Specifically, the former RSN exerted the strongest causal influence over the other RSNs, revealing a top-down modulation of self-referential mental activity (SRN) over sensory and cognitive processing. In quite contrast, the latter RSN was profoundly affected by the other RSNs, which may underlie an integration of information from primary function and higher level cognition networks, consistent with previous task-related studies. Overall, our results revealed the causal influences among these RSNs at different processing levels, and supplied information for a deeper understanding of the brain network dynamics.