Large-scale functional networks have been extensively studied using resting state functional magnetic resonance imaging (fMRI). However, the pattern, organization, and function of fine-scale network activity remain largely unknown. Here, we characterized the spontaneously emerging visual cortical activity by applying independent component (IC) analysis to resting state fMRI signals exclusively within the visual cortex. In this subsystem scale, we observed about 50 spatially ICs that were reproducible within and across subjects, and analyzed their spatial patterns and temporal relationships to reveal the intrinsic parcellation and organization of the visual cortex. The resulting visual cortical parcels were aligned with the steepest gradient of cortical myelination, and were organized into functional modules segregated along the dorsal/ventral pathways and foveal/peripheral early visual areas. Cortical distance could partly explain intra-hemispherical functional connectivity, but not interhemispherical connectivity; after discounting the effect of anatomical affinity, the fine-scale functional connectivity still preserved a similar visual-stream-specific modular organization. Moreover, cortical retinotopy, folding, and cytoarchitecture impose limited constraints to the organization of resting state activity. Given these findings, we conclude that spontaneous activity patterns in the visual cortex are primarily organized by visual streams, likely reflecting feedback network interactions. Hum Brain Mapp 38:4613-4630, 2017. © 2017 Wiley Periodicals, Inc.