The effect of self-affine roughness on solid contact is examined with molecular dynamics and continuum calculations. The contact area and normal stiffness rise linearly with the applied load, and the load rises exponentially with decreasing separation between surfaces. Results for a wide range of roughness, system size, and Poisson ratio can be collapsed using Persson's contact theory for continuous elastic media. The compliance due to atomic-scale motion at the interface between solids has little effect on the area and normal stiffness, but can reduce the total transverse stiffness by orders of magnitude. The scaling of this effect with system size is discussed.