We investigate the combination of two emerging X-ray imaging technologies, namely spectral imaging and differential phase contrast imaging. By acquiring spatially and temporally registered images with several different X-ray spectra, spectral imaging can exploit differences in the energy-dependent attenuation to generate material selective images. Differential phase contrast imaging uses an entirely different contrast generation mechanism: The phase shift that an X-ray wave exhibits when traversing an object. As both methods can determine the (projected) electron density, we propose a novel material decomposition algorithm that uses the spectral and the phase contrast information simultaneously. Numerical experiments show that the combination of these two imaging techniques benefits from the strengths of the individual methods while the weaknesses are mitigated: Quantitatively accurate basis material images are obtained and the noise level is strongly reduced, compared to conventional spectral X-ray imaging.