The role of cyclic nucleotides (cyclic 3',5'-adenosine monophosphate [cAMP] and cyclic 3',5'-guanosine monophosphate [cGMP]) in the regulation of relaxin release from large porcine luteal cells was examined by use of a reverse hemolytic plaque assay. In this assay, luteal cells are cocultured in monolayers with protein-A-coupled ovine erythrocytes. In the presence of porcine relaxin antiserum and complement, a zone of hemolysis--a plaque--develops around relaxin-releasing luteal cells. The rate of development of plaques in time-course studies has been used as an index of the rate of relaxin release, and the size of plaques formed has been employed as a record of the cumulative amount of relaxin released by each cell. Treatment of monolayers with dibutyryl cAMP (dbcAMP, 60 mM) and dibutyryl GMP (dbcGMP, 15 mM resulted in a prompt inhibition in the rate of plaque formation. In addition, dbcAMP treatment reduced the average size of plaques formed. The stimulatory effect of prostaglandin F2 alpha (PGF2 alpha 10(-6) M) on relaxin release was significantly attenuated by combined treatment with dbcAMP (60 mM). Cholera toxin treatment (500 ng/ml) effectively reduced the average size of plaques formed, but neither this agent nor the beta-adrenergic agonist, isoproterenol (up to 5 X 10(-3) M), influenced the rate of plaque formation. These results--which provide evidence to show that both basal and stimulated relaxin release by large porcine luteal cells can be inhibited by the cyclic nucleotide analogues, dbcAMP and dbcGMP--are consistent with the view that these compounds have the potential to act as a negative regulatory mechanism for relaxin release.(ABSTRACT TRUNCATED AT 250 WORDS)