In patients with type 2 diabetes mellitus (T2DM), the intestinal flora is out of balance and accompanied by leaky gut. The flora is characterized by an increase in mucus-degrading bacteria and a decrease in fiber-degrading bacteria. Short-chain fatty acids (SCFAs), as the major fiber-degrading bacteria fermentation, not only ameliorate the leaky gut, but also activate GPR43 to increase the mass of functional pancreatic β-cells and exert anti-inflammation effect. At present, the gut microbiota is considered as the potential target for anti-diabetes drugs, and how to reverse the imbalance of gut microbiota has become a therapeutic strategy for T2DM. This review briefly summarizes the drugs or compounds that have direct or potential therapeutic effects on T2DM by modulating the gut microbiota, including biguanides, isoquinoline alkaloids, stilbene and C7N-aminocyclic alcohols.