Among women diagnosed with invasive breast cancer, 30% have a prior diagnosis of benign breast disease (BBD). Thus, it is important to identify factors among BBD patients that elevate invasive cancer risk. In the general population, risk factors differ in their associations by clinical pathologic features; however, whether women with BBD show etiologic heterogeneity in the types of breast cancers they develop remains unknown. Using a nested case-control study of BBD and breast cancer risk conducted in a community healthcare plan (Kaiser Permanente Northwest), we assessed relationships of histologic features in BBD biopsies and patient characteristics with subsequent breast cancer risk and tested for heterogeneity of associations by estrogen receptor (ER) status, tumor grade, and size. The study included 514 invasive breast cancer cases (median follow-up of 9 years post-BBD diagnosis) and 514 matched controls, diagnosed with proliferative or non-proliferative BBD between 1971 and 2006, with follow-up through mid-2015. Odds ratios (ORs) and 95% confidence intervals (CIs) were obtained using multivariable polytomous logistic regression models. Breast cancers were predominantly ER-positive (86%), well or moderately differentiated (73%), small (74% < 20 mm), and stage I/II (91%). Compared to patients with non-proliferative BBD, proliferative BBD with atypia conferred increased risk for ER-positive cancer (OR = 5.48, 95% CI = 2.14-14.01) with only one ER-negative case, P-heterogeneity = 0.45. The presence of columnar cell lesions (CCLs) at BBD diagnosis was associated with a 1.5-fold increase in the risk of both ER-positive and ER-negative tumors, with a 2-fold increase (95% CI = 1.21-3.58) observed among postmenopausal women (56%), independent of proliferative BBD status with and without atypia. We did not identify statistically significant differences in risk factor associations by tumor grade or size. Most tumors that developed after a BBD diagnosis in this cohort were highly treatable low-stage ER-positive tumors. CCL in BBD biopsies may be associated with moderately increased risk, independent of BBD histology, and irrespective of ER status.