Intracortical inhibition plays a critical role in shaping activity patterns in the mature cortex. However, little is known about the structure of inhibition in early development prior to the onset of sensory experience, a time when spontaneous activity exhibits long-range correlations predictive of mature functional networks. Here, using calcium imaging of GABAergic neurons in the ferret visual cortex, we show that spontaneous activity in inhibitory neurons is already highly organized into distributed modular networks before visual experience. Inhibitory neurons exhibit spatially modular activity with long-range correlations and precise local organization that is in quantitative agreement with excitatory networks. Furthermore, excitatory and inhibitory networks are strongly co-aligned at both millimeter and cellular scales. These results demonstrate a remarkable degree of organization in inhibitory networks early in the developing cortex, providing support for computational models of self-organizing networks and suggesting a mechanism for the emergence of distributed functional networks during development.