There are increasing efforts to scale up services globally for families raising children with developmental disabilities (DDs). Existing interventions, often developed in high income, Western settings, need substantial adaptation before they can be implemented in different contexts. The aim of this study was to explore perspectives on the role that context plays in the adaptation and implementation of interventions targeting caregivers of children with DDs across settings. The study question was applied to the Caregiver Skills Training (CST) programme of the World Health Organization specifically, as well as to stakeholder experiences with caregiver interventions more broadly. Two focus group discussions (FGDs; n = 15 participants) and 25 individual semi-structured interviews were conducted. Participants were caregivers of children with DDs and professionals involved in adapting or implementing the CST across five continents and different income settings. Data were analysed thematically. Four main themes were developed: 1) Setting the scene for adaptations; 2) Integrating an intervention into local public services; 3) Understanding the reality of caregivers; 4) Challenges of sustaining an intervention. Informants thought that contextual adaptations were key for the intervention to fit in locally, even more so than cultural factors. The socio-economic context of caregivers, including poverty, was highlighted as heavily affecting service access and engagement with the intervention. Competing health priorities other than DDs, financial constraints, and management of long-term collaborations were identified as barriers. This study validates the notion that attention to contextual factors is an essential part of the adaptation of caregiver interventions for children with DDs, by providing perspectives from different geographical regions. We recommend a stronger policy and research focus on contextual adaptations of interventions and addressing unmet socio-economic needs of caregivers.