Any kind of visual information is encoded in terms of patterns of neural activity occurring inside the brain. Decoding neural patterns or its classification is a challenging task. Functional magnetic resonance imaging (fMRI) and Electroencephalography (EEG) are non-invasive neuroimaging modalities to capture the brain activity pattern in term of images and electric potential respectively. To get higher spatiotemporal resolution of human brain from these two complementary neuroimaging modalities, simultaneous EEG-fMRI can be helpful. In this paper, we proposed a framework for classifying the brain activity patterns with simultaneous EEG-fMRI. We have acquired five human participants' data with simultaneous EEG-fMRI by showing different object categories. Further, combined analysis of EEG and fMRI data was carried out. Extracted information through combine analysis is passed to support vector machine (SVM) classifier for classification purpose. We have achieved better classification accuracy using simultaneous EEG-fMRI i.e., 81.8% as compared to fMRI data standalone. This shows that multimodal neuroimaging can improve the classification accuracy of brain activity patterns as compared to individual modalities reported in literature.