Hearing loss during development leads to central deficits that persist even after the restoration of peripheral function. One key class of deficits is due to changes in central inhibitory synapses, which play a fundamental role in all aspects of auditory processing. This review focuses on the anatomical and physiological alterations of inhibitory connections at several regions within the central auditory pathway following hearing loss. Such aberrant inhibitory synaptic function may be linked to deficits in encoding binaural and spectral cues. Understanding the cellular changes that occur at inhibitory synapses following hearing loss may provide specific loci that can be targeted to improve function.