People with HIV (PWH) incur a higher risk of COVID-19-related morbidity and mortality rates, yet less is known about COVID-19 vaccine uptake and hesitancy in this group. We conducted a cross-sectional study in Freetown, Sierra Leone, from April to June 2022, using the VAX scale, a validated instrument, to assess attitudes towards COVID-19 vaccination and calculate the hesitancy (VAX) scores. We used generalized linear models to identify the factors associated with vaccine hesitancy. Overall, 490 PWH were enrolled (71.4% female, median age: 38 years, median CD4 count: 412 cells/mm3). About 17.3% received ≥1 dose of a COVID-19 vaccine. The mean VAX score was 43.14 ± 7.05, corresponding to 59.9% participants being vaccine-hesitant. A preference for natural immunity (65.8%) and concerns about profiteering (64.4%) were the commonest reasons for hesitancy, followed by a mistrust of vaccine benefits (61.4%) and worries about future effects (48.0%). In the adjusted regression analysis, being a Muslim (β = 2.563, p < 0.001) and having an urban residence (β = 1.709, p = 0.010) were associated with greater vaccine hesitancy, while testing for COVID-19 was associated with reduced vaccine hesitancy (β = -3.417, p = 0.027). These findings underscore the importance of addressing vaccine hesitancy as a critical element boosting COVID-19 vaccine uptake among PWH.