The last few years has seen a proliferation of wearable electrocardiogram (ECG) devices in the market with applications in fitness tracking, patient monitoring, athletic performance assessment, stress and fatigue detection, and biometrics, to name a few. The majority of these applications rely on the computation of the heart rate (HR) and the so-called heart rate variability (HRV) index via time-, frequency-, or non-linear-domain approaches. Wearable/portable devices, however, are highly susceptible to artifacts, particularly those resultant from movement. These artifacts can hamper HR/HRV measurement, thus pose a serious threat to cardiac monitoring applications. While current solutions rely on ECG enhancement as a pre-processing step prior to HR/HRV calculation, existing artifact removal algorithms still perform poorly under extremely noisy scenarios. To overcome this limitation, we take an alternate approach and propose the use of a spectro-temporal ECG signal representation that we show separates cardiac components from artifacts. More specifically, by quantifying the rate-of-change of ECG spectral components over time, we show that heart rate estimates can be reliably obtained even in extremely noisy signals, thus bypassing the need for ECG enhancement. With such HR measurements in hands, we then propose a new noise-robust HRV index termed MD-HRV (modulation-domain HRV) computed as the standard deviation of the obtained HR values. Experiments with synthetic ECG signals corrupted at various different signal-to-noise levels, as well as recorded noisy signals show the proposed measure outperforming several HRV benchmark parameters computed post wavelet-based enhancement. These findings suggest that the proposed HR measures and derived MD-HRV metric are well-suited for ambulant cardiac monitoring applications, particularly those involving intense movement (e.g., elite athletic training).