Electromyogram (EMG) signal decoding is the essential part of myoelectric control. However, traditional machine learning methods lack the capability of learning and expressing the information contained in EMG signals, and the robustness of the myoelectric control system is not sufficient for real life applications. In this article, a novel model based on recurrent convolutional neural networks (RCNNs) is proposed for hand movement classification and tested on the noninvasive EMG dataset. The proposed model uses deep architecture, which has advantages of dealing with complex time-series data, such as EMG signals. Transfer learning is used in the training of multimodal model. The classification performance is compared with support vector machine (SVM) and convolutional neural networks (CNNs) on the same dataset. To improve the adaptability to the effect of arm movements, we fused the EMG signals and acceleration data that are the multimodal input of the model. The parameter transferring of deep neural networks is used to accelerate the training process and avoid over-fitting. The experimental results show that time domain input and 1-dimensional convolution have higher accuracy in the RCNN model. Compared with SVM and CNNs, the proposed model has higher classification accuracy. Sensor fusion can improve the model performance in the condition of arm movements. The RCNN model is a promising decoder of EMG and the sensor fusion can increase the accuracy and robustness of the myoelectric control system.