In-bed posture monitoring has become a prevalent area of research to help minimize the risk of pressure sore development and to increase sleep quality. This paper proposed 2D and 3D Convolutional Neural Networks, which are trained on images and videos of an open-access dataset consisting of 13 subjects' body heat maps captured from a pressure mat in 17 positions, respectively. The main goal of this paper is to detect the three main body positions: supine, left, and right. We compare the use of image and video data through 2D and 3D models in our classification. Since the dataset was imbalanced, three strategies were evaluated, i.e., down sampling, over sampling, and class weights. The best 3D model achieved accuracies of 98.90 ± 1.05% and 97.80 ± 2.14% for 5-fold and leave-one-subject-out (LOSO) cross validations, respectively. To compare the 3D model with 2D, four pre-trained 2D models were evaluated, where the best-performing model was the ResNet-18 with accuracies of 99.97 ± 0.03% for 5-fold and 99.62 ± 0.37% for LOSO. The proposed 2D and 3D models provided promising results for in-bed posture recognition and can be used in the future to further distinguish postures into more detailed subclasses. The outcome of this study can be used to remind caregivers at hospitals and long-term care facilitiesto reposition their patients if they do not reposition themselves naturally to prevent pressure ulcers. In addition, the evaluation of body postures and movements during sleep can help caregivers understand sleep quality.