Functional assessment is an essential part of rehabilitation protocols after stroke. Conventionally, the assessment process relies heavily on clinical experience and lacks quantitative analysis. In order to objectively quantify the upper-limb motor impairments in patients with post-stroke hemiparesis, this study proposes a novel assessment approach based on motor synergy quantification and multi-modality fusion. Fifteen post-stroke hemiparetic patients and fifteen age-matched healthy persons participated in this study. During different goal-directed tasks, kinematic data and surface electromyography(sEMG) signals were synchronously collected from these participants, and then motor features extracted from each modal data could be fed into the respective local classifiers. In addition, kinematic synergies and muscle synergies were quantified by principal component analysis (PCA) and k weighted angular similarity ( k WAS) algorithm to provide in-depth analysis of the coactivated features responsible for observable movement impairments. By integrating the outputs of local classifiers and the quantification results of motor synergies, ensemble classifiers can be created to generate quantitative assessment for different modalities separately. In order to further exploit the complementarity between the evaluation results at kinematic and muscular levels, a multi-modal fusion scheme was developed to comprehensively analyze the upper-limb motor function and generate a probability-based function score. Under the proposed assessment framework, three types of machine learning methods were employed to search the optimal performance of each classifier. Experimental results demonstrated that the classification accuracy was respectively improved by 4.86% and 2.78% when the analysis of kinematic and muscle synergies was embedded in the assessment system, and could be further enhanced to 96.06% by fusing the characteristics derived from different modalities. Furthermore, the assessment result of multi-modality fusion framework exhibited a significant correlation with the score of standard clinical tests ( R = - 0.87, P = 1.98e - 5 ). These promising results show the feasibility of applying the proposed method to clinical assessments for post-stroke hemiparetic patients.