The sulfhydryl reagents p-chloromercuribenzoate and N-ethylmaleimide (NEM) inactivate high affinity [3H]serotonin [( 3H]5-HT) binding to bovine and rat brain membranes in a concentration-dependent manner. In both species, 15-25% of total specific high affinity [3H]5-HT binding is relatively insensitive to NEM. This study examines the NEM sensitivity of the various high affinity [3H]5-HT binding subtypes, using selective ligands, tissues, and pharmacological masks to study each subtype. Reconstitution of NEM-inactivated binding by addition of GTP-binding proteins (G proteins, Gi and Go) is also described. Agonist binding to 5-HT1A, 5-HT1B, and 5-HT1D sites in rat brain and to 5-HT1A and 5-HT1D sites in bovine brain is sensitive to NEM. Binding of [3H]dihydroergotamine and [125I]iodocyanopindolol, both of which are weak partial agonists to 5-HT1B sites is relatively insensitive to NEM. Binding of [3H]5-HT to 5-HT1C sites in rat and bovine brain and choroid plexus is relatively insensitive to NEM. In the presence of spiperone to mask binding of 5-HT2 sites, binding of antagonist [( 3H]mesulergine) to 5-HT1C sites is also insensitive to NEM. Likewise, binding of the agonist [3H]4-bromo-2,5-dimethoxyphenylisopropylamine and of the antagonist [3H]ketanserin to 5-HT2 sites is not inhibited by NEM treatment of membranes. These findings suggest that agonist binding to 5-HT1A, 5-HT1B, and 5-HT1D sites is sensitive to NEM alkylation. Binding of neither agonist nor antagonist to 5-HT1C and 5-HT2 sites is sensitive to NEM. Inability of high concentrations of a variety of ligands to protect the sensitive binding sites against NEM inactivation indicates that the critical sulfhydryl group(s) are not located in the ligand binding domain of the NEM-sensitive binding sites. When membranes are treated with NEM, displacement of [125I]iodocyanopindolol by 5-HT is no longer sensitive to 5'-guanylyl imidodiphosphate (Gpp(NH)p). Gpp(NH)p sensitivity of agonist displacement of 5-HT1B binding to NEM-treated membranes is restored by addition of purified guanine nucleotide binding proteins (Gi plus Go). In addition, NEM-inactivated binding to 5-HT1A and 5-HT1D sites can be restored by addition of Gi plus Go. These data suggest that NEM exerts its effects on 5-HT1A, 5-HT1B, and 5-HT1D binding sites by inactivating the G protein(s) associated with the 5-HT receptor subtypes.