It has been argued that unipolar major depressive disorder (MDD) and bipolar disorder (BD) exist on a continuous spectrum, given their overlapping symptomatology and genetic diatheses. The Bipolarity Index (BI) is a scale that considers bipolarity as a continuous construct and was developed to assess confidence in bipolar diagnosis. Here we investigated whether BI scores correlate with gray matter volume (GMV) in a sample of unmedicated unipolar and bipolar depressed individuals. 158 subjects (139 with MDD, 19 with BD) in a major depressive episode at time of scan were assigned BI scores. T1-weighted Magnetic Resonance Imaging scans were obtained and processed with Voxel-Based Morphometry using SPM12 (CAT12 toolbox) to assess GMV. Regression was performed at the voxel level to identify clusters of voxels whose GMV was associated with BI score, (p<0.001, family-wise error-corrected cluster-level p<0.05), with age, sex and total intracranial volume as covariates. GMV was inversely correlated with BI score in four clusters located in left lateral occipital cortex, bilateral angular gyri and right frontal pole. Clusters were no longer significant after controlling for diagnosis. GMV was not correlated with BI score within the MDD cohort alone. Incomplete clinical data required use of a modified BI scale. BI scores were inversely correlated with GMV in unmedicated subjects with MDD and BD, but these correlations appeared driven by categorical diagnosis. Future work will examine other imaging modalities and focus on elements of the BI scale most likely to be related to brain structure and function.