The auditory system is capable of perceptually restoring inaudible portions of speech. This restoration may be compromised as a result of hearing impairment, particularly if it is combined with advanced age, because of degradations in the bottom-up and top-down processes. To test this hypothesis, phonemic restoration was quantitatively measured with hearing-impaired listeners of varying ages and degrees of hearing impairment, as well as with a normal hearing control group. The results showed that the restoration benefit was negatively correlated with both hearing impairment and age, supporting the original hypothesis. Group data showed that listeners with mild hearing loss were able to perceptually restore the missing speech segments as well as listeners with normal hearing. By contrast, the moderately-impaired listeners showed no evidence of perceptual restoration. Further analysis using the articulation index showed that listeners with mild hearing loss were able to increase phonemic restoration with audibility. Moderately-impaired listeners, on the other hand, were unable to do so, even when the articulation index was high. The overall findings suggest that, in addition to insufficient audibility, degradations in the bottom-up and/or top-down mechanisms as a result of hearing loss may limit or entirely prevent phonemic restoration.