It has been shown that excessive white light exposure during retinal microsurgery can induce retinal damage. To address this problem, one can illuminate the retina with a device that alternates between white, and less damaging limited-spectrum light. The surgeon is then presented with a fully colored video by recoloring the limited-spectrum light frames, using information from the white-light frames. To obtain accurately colored images, while reducing phototoxicity, we have developed a novel algorithm that monitors the quality of the recolored images and determines when white light may be substituted by limited-spectrum light. We show qualitatively and quantitatively that our system can provide reliable images using a significantly smaller light dose as compared to other state-of-the-art coloring schemes.