The antiestrogen tamoxifen [(Z)-1(p-beta-dimethylaminoethoxy-phenyl)-1,2-diphenylbut-1-ene] is an effective anticancer agent against estrogen receptor (ER)-positive breast cancer. The alkylaminoethane side chain is essential for antiestrogenic activity, but the potency of the antiestrogen can be increased by para hydroxylation of the phenyl ring on carbon 1 of but-1-ene. This compound, 4-hydroxytamoxifen, is a metabolite of tamoxifen and has a very high binding affinity for ER [J. Endocrinol. 75:305-316 (1977)] because the hydroxyl is located in the equivalent position as the 3-phenolic hydroxyl of 17 beta-estradiol. In this study, we have examined the relationship between the relative positions of the hydroxyl and the alkyl-aminoethane side chain and the pharmacological activity of the ligand. A fixed seven-membered ring derivative of the triphenylethylene was used to prevent isomerization. All compounds were tested, with and without 17 beta-estradiol, for their effects on the growth of estrogen-responsive T47D and MCF-7 human breast cancer cells in vitro. The growth of MDA-MB-231 ER-negative breast cancer cells was not affected by any of the compounds tested, at a concentration (1 microM) that had a profound estrogenic or antiestrogenic action in ER-positive cell lines. The relative binding affinity of the compounds was determined using rat uterine ER and was found to be consistent with the observed potencies in vitro. The compounds found to be antiestrogens in vitro were antiestrogenic against estradiol (0.08 micrograms daily) in the 3-day immature rat uterine weight test. All compounds were partial agonists in vivo. In general, the estrogenic and antiestrogenic results obtained in vivo were consistent with the potency estimates obtained with the breast cancer cells in vitro. The results of this extensive structure-activity relationship study demonstrate that the substitution for 4-hydroxytamoxifen appears to be optimal to produce a potent antiestrogen; all other substitutions produced either estrogenic compounds or less potent antiestrogens. The hydroxyl group appears to be critical to locate the alkyl aminoethoxy side chain in the correct position in the steroid-binding site to block estrogen action. Novel antiestrogens were identified that could have been predicted based upon earlier drug-receptor models for the ER.