A fundamental question in visual perception research is whether the sensitivity to stimulus differences is limited by the sensory representation of the external stimulus, that is, the proximal stimulus, or by its perceptual representation, i.e., stimulus appearance. In the domain of lightness perception, the question translates into whether discrimination thresholds depend on the local luminance in the retinal image or on the apparent lightness of the corresponding image region. The majority of findings seem to indicate that sensitivity is limited by the sensory stimulus representation, which would imply different mechanisms for stimulus discrimination and appearance. We think this conclusion needs to be qualified. We report data suggesting that the relationship between discrimination and appearance judgments depends on how exactly they are being measured. We propose a theoretical account that provides a common mechanism for appearance and sensitivity. An interesting corollary of this model is that it also accounts for the perceptual phenomenon of assimilation.