Testosterone is a powerful steroid hormone that can impact the brain and behavior in various ways, including regulating behavioral and neuroendocrine (hypothalamic-pituitary-adrenal (HPA) axis) stress responses. Early in life androgens can act to alter development of brain regions associated with stress regulation, which ultimately impacts the display of stress responses later in life. Adult circulating androgens can also influence the expression of distinct genes and proteins that regulate stress responses. These changes in the brain are hypothesized to underlie the potent effects of androgens in regulating behaviors related to stress and stress-induced activation of the HPA axis. Androgens can induce alterations in these functions through direct binding to the androgen receptor (AR) or following conversion to estrogens and subsequent binding to estrogen receptors including estrogen receptor alpha (ERα), beta (ERβ), and G protein-coupled estrogen receptor 1 (GPER1). In this review, we focus on the role of androgens in regulating behavioral and neuroendocrine stress responses at different stages of the lifespan and the sex hormone receptors involved in regulating these effects. We also review the specific brain regions and cell phenotypes upon which androgens are proposed to act to regulate stress responses with an emphasis on hypothalamic and extended amygdala subregions. This knowledge of androgen effects on these neural systems is critical for understanding how sex hormones regulate stress responses.