Serotonin (5-hydroxytryptamine) has multiple cardiovascular actions. The presence of serotonin in the heart suggests it may be an endogenous source of inotropic support during physiologic or pathologic stress. Serotonin may increase cardiac contractility by augmenting release of norepinephrine at sympathetic nerve endings. Norepinephrine release is markedly elevated in patients with heart failure. To explore the role of serotonin in enhancing norepinephrine release in patients with heart failure, ketanserin, a specific serotonin antagonist, was used as a physiologic tool to examine the effect on transmyocardial norepinephrine flux. Ketanserin (10 mg bolus, 4 mg/hr infusion for +/- 40 min) was administered intravenously to nine patients with congestive heart failure (NYHA III or IV) secondary to congestive cardiomyopathy (N = 7), or ischemic heart disease (N = 2). Plasma catecholamines (norepinephrine, epinephrine, dopamine) were measured in the aorta (Ao) and the coronary sinus (CS) of patients at rest and during supine leg exercise before and after administration of ketanserin. Baseline norepinephrine levels were markedly elevated at rest and during exercise in all patients. Norepinephrine levels were significantly higher in the CS than in the Ao (rest, CS 1185 +/- 235, Ao 878 +/- 381 pg/mL, P less than .05; exercise, CS 2239 +/- 697, Ao 1453 +/- 697 pg/mL, P less than .05). Baseline epinephrine levels were within normal limits. In contrast to norepinephrine levels, epinephrine levels were consistently higher in the Ao than in the CS, indicating unimpaired extraction or uptake across the heart. The relationship between norepinephrine and epinephrine concentration in the Ao and CS suggested a net overflow of norepinephrine in the CS.(ABSTRACT TRUNCATED AT 250 WORDS)