Quantitative in vitro receptor autoradiography was used to characterize the [3H]5-hydroxytryptamine binding sites which are not sensitive to 8-hydroxy-2-(di-N-propylamino)tetralin, mesulergine and serotonin-5-O-carboxy-methyl-glycyl-tyrosinamide, in a non-5-hydroxytryptamine1A/1B/1C/1D receptor population, in rat brain. Displacement of [3H]5-hydroxytryptamine [in the presence of 100 nM 8-hydroxy-2-(di-N-propyl-amino)tetralin and mesulergine, to block 5-hydroxytryptamine1A and 5-hydroxytryptamine1C sites] with (-)pindolol, 5-hydroxy-3(4-1,2,5,6-tetrahydropyridyl)-4-azaindole, sumatriptan and serotonin-5-O-carboxy-methyl-glycyl-tyrosinamide yielded complex competition curves suggesting the presence of 5-hydroxytryptamine1B and 5-hydroxytryptamine1D sites and an additional [3H]5-hydroxytryptamine-sensitive component in rat brain. The non-5-hydroxytryptamine1A/1B/1C/1D binding sites were localized in olfactory tubercle, several nuclei of the amygdala, bed nucleus of the stria terminalis, caudate-putamen, CA3 field of the hippocampus, the frontoparietal cortex (motor area) and parts of the striate cortex. All the drugs used had low affinity for the unknown recognition site, which therefore might be comparable to the [3H]5-hydroxytryptamine binding site reported to display low affinity for sumatriptan and 5-carboxamidotryptamine in the brains of various species, the so-called 5-hydroxytryptamine1E site. A comparison of the density of sites labelled with [125I]serotonin-5-O-carboxy-methyl-glycyl-tyrosinamide (representing 5-hydroxytryptamine1B and 5-hydroxytryptamine1D sites) and [3H]5-hydroxytryptamine (under the conditions mentioned above) showed the density of [3H]5-hydroxytryptamine recognition sites to be higher in some structures.(ABSTRACT TRUNCATED AT 250 WORDS)