It has been suggested that in addition to genetic factors, fetal and post-natal growth influence cognition in early adulthood. However, most studies have been in developed populations, so it is unclear if the same findings would be seen in other, less developed, settings, and have used testing tools not applicable to an Australia Aboriginal population. This study investigated the relationships between cognitive function in early adulthood and birth weight and contemporary height. Simple reaction time (SRT), choice reaction time (CRT) and working memory (WM) were assessed using the CogState battery. A significant association was seen between birth weight and SRT in early adulthood, but not with the other two cognitive measures. Urban dwellers had significantly shorter SRT and CRT than their remote counterparts. Contemporary body mass index and maternal age were associated with CRT. Only fetal growth restriction was associated with WM, with greater WM in those with restricted growth. No associations were seen with contemporary height. These results suggest that fetal growth may be more important than the factors influencing post-natal growth in terms of cognition in early adulthood in this population, but that the associations may be inconsistent between cognitive outcomes. Further research is required to identify whether similar associations are seen in other, similar, populations and to assess why differences in cognitive outcome measures are seen.