Stress following preterm birth can disrupt the emerging foundation of the neonatal brain. The current study examined how structural brain development is affected by a stressful early environment and whether changes in topological architecture at term-equivalent age could explain the increased vulnerability for behavioral symptoms during early childhood. Longitudinal changes in structural brain connectivity were quantified using diffusion-weighted imaging (DWI) and tractography in preterm born infants (gestational age <28 weeks), imaged at 30 and/or 40 weeks of gestation (N = 145, 43.5% female). A global index of postnatal stress was determined based on the number of invasive procedures during hospitalization (e.g., heel lance). Higher stress levels impaired structural connectivity growth in a subnetwork of 48 connections (p = 0.003), including the amygdala, insula, hippocampus, and posterior cingulate cortex. Findings were replicated in an independent validation sample (N = 123, 39.8% female, n = 91 with follow-up). Classifying infants into vulnerable and resilient based on having more or less internalizing symptoms at two to five years of age (n = 71) revealed lower connectivity in the hippocampus and amygdala for vulnerable relative to resilient infants (p < 0.001). Our findings suggest that higher stress exposure during hospital admission is associated with slower growth of structural connectivity. The preservation of global connectivity of the amygdala and hippocampus might reflect a stress-buffering or resilience-enhancing factor against a stressful early environment and early-childhood internalizing symptoms.SIGNIFICANCE STATEMENT The preterm brain is exposed to various external stimuli following birth. The effects of early chronic stress on neonatal brain networks and the remarkable degree of resilience are not well understood. The current study aims to provide an increased understanding of the impact of postnatal stress on third-trimester brain development and describe the topological architecture of a resilient brain. We observed a sparser neonatal brain network in infants exposed to higher postnatal stress. Limbic regulatory regions, including the hippocampus and amygdala, may play a key role as crucial convergence sites of protective factors. Understanding how stress-induced alterations in early brain development might lead to brain (re)organization may provide essential insights into resilient functioning.