The interaction of cholinergic agonists and antagonists with smooth muscle muscarinic receptors has been investigated by measurement of displacement of the muscarinic antagonist [3H]QNB (quinuclidinyl benzilate) in membranes prepared from toad stomach. The binding of [3H]QNB was saturable, reversible and of high affinity (KD = 423 pM). The muscarinic receptor subtypes present in gastric smooth muscle were classified by determining the relative affinities for the selective antagonists pirenzepine (M1), AF-DX 116 (M2) and 4-DAMP (M3). The results from these studies indicate the presence of a heterogeneous population of muscarinic receptor subtypes, with a majority (88%) exhibiting characteristics of M3 receptors and a much smaller population (12%) exhibiting characteristics of M2 receptors. The binding curve for the displacement of [3H]QNB binding by the agonist oxotremorine was complex and was consistent with presence of two affinity states: 24% of the receptors had a high affinity (KD = 4.7 nM) for oxotremorine and 76% displayed nearly a 1,000-fold lower affinity (KD = 4.4 microM). When oxotremorine displacement of [3H]QNB binding was determined in the presence GTP gamma S, high affinity binding was abolished, indicating that high affinity agonist binding may represent receptors coupled to G proteins. Moreover, pertussis toxin pretreatment of membranes also abolished high affinity agonist binding, indicating that the muscarinic receptors are coupled to pertussis toxin-sensitive G proteins. Reaction of smooth muscle membranes with pertussis toxin in the presence [32P]NAD caused the [32P]-labelling of a 40 dD protein that may represent the alpha subunit(s) of G proteins that are known to by NAD-ribosylated by the toxin.(ABSTRACT TRUNCATED AT 250 WORDS)