Diabetes mellitus is a chronic endocrine disease result from absolute or relative insulin secretion deficiency, insulin resistance, or both, and has become a major and growing public healthy menace worldwide. Currently, clinical antidiabetic drugs still have some limitations in efficacy and safety such as gastrointestinal side effects, hypoglycemia, or weight gain. Rosmarinus officinalis is an aromatic evergreen shrub used as a food additive and medicine, which has been extensively used to treat hyperglycemia, atherosclerosis, hypertension, and diabetic wounds. A great deal of pharmacological research showed that rosemary extract and its phenolic constituents, especially carnosic acid, rosmarinic acid, and carnosol, could significantly improve diabetes mellitus by regulating glucose metabolism, lipid metabolism, anti-inflammation, and anti-oxidation, exhibiting extremely high research value. Therefore, this review summarizes the pharmacological effects and underlying mechanisms of rosemary extract and its primary phenolic constituents on diabetes and relative complications both in vitro and in vivo studies from 2000 to 2020, to provide some scientific evidence and research ideas for its clinical application.