Recently, maternal nutrient intake has been associated with human milk oligosaccharides (HMOs) composition. The goal of this study was to assess HMO composition in breast milk samples from vegan, vegetarian, and nonvegetarian lactating women. Second, we assessed impact of maternal body mass index (BMI), age, parity, and lactation stage on HMO composition. A cross-sectional analysis of HMO composition from vegan (n = 26), vegetarian (n = 22), and nonvegetarian (n = 26) lactating women was carried out. The majority of participants took dietary supplements. In an unadjusted bivariate model, there was no difference in individual HMO composition, total HMO-bound fucose and HMO-bound sialic acid, or diversity and evenness scores by diet group. When adjusting for factors that significantly differed between groups (maternal BMI and lactation stage), no differences in HMO composition were observed. Secretor status was significant for 13 of the outcome variables with the strongest positive relationship with total HMO (β = 0.922) and HMO-bound fucose (β = 0.910), and the strongest negative relationship with sialyl-lacto-N-tetraose b (LSTb) (β = -0.544). Lactation stage was significant for eight analytes, with the strongest positive impact on 3'sialyllactose (3'SL) (β = 0.433), and the strongest negative impact on 6'sialyllactose (6'SL) (β = -0.519). Maternal BMI had a significant positive relationship with total HMO composition (β = 0.113) and 3'SL (β = 0.325). Lactating women who consume plant-based diets do not produce different breast milk as it relates to HMO composition.