The EEG-based emotion recognition is one of the primary research orientations in the field of emotional intelligence and human-computer interaction (HCI). We proposed a novel model, denoted as ICRM-LSTM, for EEG-based emotion recognition by combining the independent component analysis (ICA), the Riemannian manifold (RM), and the long short-term memory network (LSTM). The SEED and MAHNOB-HCI dataset were employed to verify the performance of the proposed model. Firstly, ICA was used to perform blind source separation (BSS) for the preprocessed EEG signals provided by the two datasets. Then, a series of the covariance matrices according to time order were extracted from the blind source signals, and the symmetric positive definiteness of covariance matrix allowed us to project them from RM space to Euclid space by logarithmic mapping. Finally, the transformed covariance matrices were taken as inputs of the LSTM network to perform the emotion recognition. To validate the effect of the ICRM method on the performance of the proposed model, we designed three groups of comparative experiments. The average accuracy of the ICRM-LSTM model were 96.75 % and 98.09 % in SEED and MAHNOB-HCI, respectively. Then we compared our model with the models didn't perform the ICA or RM method, where we found that the proposed model had better performance. Furthermore, to verify the robustness, we added three groups of Gaussian noise with different signal-to-noise ratios (SNR) to the preprocessed EEG signals, and the proposed model achieved a good performance in both the two datasets. The performance of our model was superior to most of existing methods which also employed the SEED or MAHNOB-HCI dataset. This paper demonstrated that the ICA and RM were helpful for EEG-based emotion recognition, and provided the evidence that the RM method could effectively alleviate the problem of the uncertain ordering of ICA.