The affinities of 16 5-hydroxytryptamine (5-HT) receptor agonists (indole derivatives) and 7 5-HT receptor antagonists for [3H]5-hydroxytryptamine [( 3H]5-HT) binding sites in rat cerebral cortex membranes were determined. In addition, the potencies of the agonists for inhibiting the electrically induced tritium overflow from rat brain cortex slices preincubated with [3H]5-HT and from canine saphenous veins preincubated with [3H]noradrenaline were measured. Furthermore, the potencies of the indole derivatives for inducing contractile responses of canine saphenous veins were recorded. In addition, the interaction of the antagonists with unlabelled 5-HT at the 5-HT autoreceptor was studied in rat brain cortex slices. There was a good correlation between the binding affinities of the indole derivatives for the [3H]5-HT sites of rat brain cortex membranes and their potencies for inhibiting the evoked tritium overflow from both rat brain cortex slices and strips of canine saphenous vein. Comparison of the inhibition constants derived from the overflow experiments in both tissues again revealed a high correlation coefficient while there was only weak correlation between the binding affinities in rat brain cortex and the contractile potencies of the drugs in canine saphenous vein strips. When 5-HT receptor antagonists were investigated, metitepin and metergoline showed moderate affinities for the 5-HT autoreceptors in rat brain cortex slices, whereas quipazine had only weak affinity, and ketanserin, metoclopramide, cinanserin and cyproheptadine exhibited no antagonistic property. In binding experiments, the competition curves of most 5-HT receptor antagonists were biphasic, suggesting that the [3H]5-HT binding sites are heterogeneous.(ABSTRACT TRUNCATED AT 250 WORDS)