The prognosis for patients with cancers known for a highly activated stromal reaction, including diffuse-type (scirrhous) gastric cancer, consensus molecular subtype 4 (CMS4) colorectal cancer, and pancreatic ductal adenocarcinoma, is extremely poor. To explore the resistance of conventional therapy for those refractory cancers, detailed classification and investigation of the different subsets of cancer-associated fibroblasts (CAFs) involved are needed. Recent studies with a single-cell transcriptomics strategy (single-cell RNA-seq) have demonstrated that CAF subpopulations contain different origins and marker proteins with the capacity to either promote or suppress cancer progression. Through multiple signaling pathways, CAFs can promote tumor growth, metastasis, and angiogenesis with extracellular matrix (ECM) remodeling; they can also interact with tumor-infiltrating immune cells and modulate the antitumor immunological state in the tumor microenvironment (TME). Here, we review the recent literature on the various subpopulations of CAFs to improve our understanding of the cell-cell interactions in the TME and highlight future avenues for CAF-targeted therapy.