The blood flow response to a vasoactive stimulus demonstrates regional heterogeneity across both the healthy brain and in cerebrovascular pathology. The timing of a regional hemodynamic response is emerging as an important biomarker of cerebrovascular dysfunction, as well as a confound within fMRI analyses. Previous research demonstrated that hemodynamic timing is more robustly characterized when a larger systemic vascular response is evoked by a breathing challenge, compared to when only spontaneous fluctuations in vascular physiology are present (i.e., in resting-state data). However, it is not clear whether hemodynamic delays in these two conditions are physiologically interchangeable, and how methodological signal-to-noise factors may limit their agreement. To address this, we generated whole-brain maps of hemodynamic delays in nine healthy adults. We assessed the agreement of voxel-wise gray matter (GM) hemodynamic delays between two conditions: resting-state and breath-holding. We found that delay values demonstrated poor agreement when considering all GM voxels, but increasingly greater agreement when limiting analyses to voxels showing strong correlation with the GM mean time-series. Voxels showing the strongest agreement with the GM mean time-series were primarily located near large venous vessels, however these voxels explain some, but not all, of the observed agreement in timing. Increasing the degree of spatial smoothing of the fMRI data enhanced the correlation between individual voxel time-series and the GM mean time-series. These results suggest that signal-to-noise factors may be limiting the accuracy of voxel-wise timing estimates and hence their agreement between the two data segments. In conclusion, caution must be taken when using voxel-wise delay estimates from resting-state and breathing-task data interchangeably, and additional work is needed to evaluate their relative sensitivity and specificity to aspects of vascular physiology and pathology.