The influence of chronic arsenic exposure on cognitive impairment has been explored broadly by previous studies. However, most of them focused mainly on children rather than adults. In addition, in China, studies in this field are not sufficient. To illustrate how long-term arsenic exposure affects cognitive function, we designed a cross-sectional study involving 1556 adults. All of them came from three locations around the Realgar Plant. The cognitive function of the participants was evaluated using a Chinese version of the Mini-mental state Examination (MMSE). The participants' internal arsenic exposure status (hair arsenic concentrations) and the external arsenic exposure status (the distance between the participants' location of residence and the Realgar Plant) were measured. Our research revealed that both of hair arsenic concentrations and the prevalence of arsenicosis, two important indexes, were significantly higher in the cognitive-impaired (CI) group than in the cognitive-normal (CN) group (P < 0.05). In addition, distance from the Realgar Plant was positively correlated with the MMSE scores and was negatively correlated with the prevalence of cognitive impairment. Moreover, our results demonstrated that there was a negative correlation between hair arsenic concentrations and MMSE scores. We conducted a two-level Logistic regression analysis and further confirmed that even after adjusting for potential confounding variables, arsenicosis retained a risk factor for cognitive impairment (odds ratio (OR) = 1.84, P < 0.05). Our results indicated that chronic arsenic exposure could impair adults' cognitive function in a dose-dependent manner. Additionally, arsenicosis could be an independent risk factor for cognitive impairment.