Many studies on working memory have assumed that one can determine an individual's fixed memory capacity. In the current study, we took an individual differences approach to investigate whether visual working memory (VWM) capacity was stable irrespective of the number of to-be-remembered objects and participant age. Younger and older adults performed a change detection task using several objects defined by color. Results showed wide variability in VWM capacity across memory set sizes, age, and individuals. A marked decrease in the number of objects held in VWM was observed in both younger and older adults with low memory capacity, but not among high-capacity individuals, when set size went well beyond the limits of VWM capacity. In addition, a decrease in the number of objects held in VWM was alleviated among low-capacity younger adults by increasing VWM encoding time; however, increasing encoding time did not benefit low-capacity older adults. These findings suggest that low-capacity individuals are likely to show decreases in VWM capacity induced by overload, and aging exacerbates this deficit such that it cannot be recovered by simply increasing encoding time. Overall, our findings challenge the prevailing assumption that VWM capacity is fixed and stable, encouraging a revision to the strict view that VWM capacity is constrained by a fixed number of distinct "slots" in which high-resolution object representations are stored.