This retrospective cohort study of veterans diagnosed with diabetes mellitus evaluated the association between time-varying measures of glycemic control and the time to prostate cancer-specific mortality. Competing risk Cox regression models were developed to estimate the association of glycemic control and prostate cancer mortality for the entire sample and stratified by racial and ethnic groups. A total of 763,424 veterans with type 2 diabetes mellitus (T2DM) were included. In the fully adjusted models, moderate glycemic control [hemoglobin A1c (HbA1c) 7%-8%] was associated with a 23% (HR, 0.77; 95% confidence interval, 0.68-0.85) lower risk and poor glycemic control (HbA1c >8%) was associated with a 15% (HR, 0.85; 95% confidence interval, 0.71-0.99) lower risk of prostate cancer mortality compared with good glycemic control (HbA1c <7%), respectively. In the analyses stratified by race and ethnicity, moderate glycemic control was associated with a lower risk of prostate cancer mortality in non-Hispanic White and non-Hispanic Black veterans. Unlike many other cancers, there is an inverse association between prostate cancer risk and T2DM diagnosis. In this large, retrospective study of male veterans with T2DM, we observed an inverse association between glycemic control and prostate cancer mortality. Further research is required to verify this relationship in prospective studies and identify the potential mechanisms contributing to these findings.