Panax ginseng C. A. Meyer (P. ginseng) has a long history of medicinal use and can treat a variety of diseases. P. ginseng contains a variety of active ingredients, such as saponins, polypeptides, volatile oils, and polysaccharides. Among them, saponins have always been considered as the main components responsible for its pharmacological activities. However, more and more studies have shown that polysaccharides play an indispensable role in the medicinal value of ginseng. Modern biological and medical studies have found that ginseng polysaccharides have complex structural characteristics and diverse biological activities, such as immune regulation, anti-tumor, antioxidant, hypoglycemic, and anti-radiation functions, among others. Additionally, the structural characteristics of ginseng polysaccharides are closely related to their activity. In this review, the research background, extraction, purification, structural characteristics, and biological activities of ginseng polysaccharides from different parts of P. ginseng (roots, flowers stems and leaves, and berries) under different growth conditions (artificially cultivated ginseng, mountain ginseng, and wild ginseng) are summarized. The structural characteristics of purified polysaccharides were reviewed. Meanwhile, their biological activities were introduced, and some possible mechanisms were listed. Furthermore, the structure-activity relationship of polysaccharides was discussed. Some research perspectives for the study of ginseng polysaccharides were also provided.