Correct identification of the fundamental heart sounds is an important step in identifying the heart cycle stages. Heart valve pathologies can cause abnormal heart sounds or extra sounds, and an important distinguishing feature between different pathologies is the timing of these extra sounds in the heart cycle. In the design of an understandable heart sound analysis system, heart sound segmentation is an indispensable step. In this study classification of the fundamental heart sounds using continuous wavelet transform (CWT) scalograms and convolutional neural networks (CNN) is investigated. Classification between the first and second heart sound of scalograms produced by the Morse analytic wavelet was compared for CNN, support vector machine (SVM), and knearest neighbours (kNN) classifiers. Samples of the first and second heart sound were extracted from a publicly available dataset of normal and abnormal heart sound recordings, and magnitude scalograms were calculated for each sample. These scalograms were used to train and test CNNs. Classification using features extracted from a fully connected layer of the network was compared with linear binary pattern features. The CNN achieved an average classification accuracy of 86% when distinguishing between the first and second heart sound. Features extracted from the CNN and classified using a SVM achieved similar results (85.9%). Classification of the CNN features outperformed LBP features using both SVM and kNN classifiers. The results indicate that there is significant potential for the use of CWT and CNN in the analysis of heart sounds.