Environmental toxicants viz lead or cadmium and phthalate esters (di(2-ethylhexyl) phthalate [DEHP], dibutyl phthalate [DBP], and diethyl phthalate [DEP]) widely found in different environmental strata are linked to deteriorating male reproductive health. The objective was to assess the relationships between the seminal lead, cadmium, and phthalate (DEHP, DBP, DEP) concentrations at environmental level and serum hormone levels and semen quality in non-occupationally exposed men and specify the effect of individual and combined exposure of toxicants on semen quality. A study of 60 male partners of couples attending the Andrology Laboratory of the Reproductive Biology Department, All India Institute of Medical Sciences (AIIMS), New Delhi, India for semen analysis to assess their inability to achieve a pregnancy was selected for the study. The results of univariate and stepwise multiple regression analysis in the unadjusted model showed a significant correlation between lead or cadmium and phthalates DEHP/DBP/DEP and sperm motility, sperm concentration, and DNA damage. After adjusting for potential confounders, an association with lead or DEHP was only observed. The present data shows that lead (Pb) or cadmium (Cd) or phthalates might independently contribute to decline in semen quality and induce DNA damage. Phthalates might influence reproductive hormone testosterone. These findings are significant in light of the fact that men are exposed to a volley of chemicals; however, due to the small sample size, our finding needs to be confirmed in a larger population.