Leisure-time physical activity (LTPA) is one of the modifiable lifestyle factors that play an important role in the prevention of non-communicable (especially cardiovascular) diseases. Certain genetic factors predisposing to LTPA have been previously described, but their effects and applicability on different ethnicities are unknown. Our present study aims to investigate the genetic background of LTPA using seven single nucleotide polymorphisms (SNPs) in a sample of 330 individuals from the Hungarian general (HG) and 314 from the Roma population. The LTPA in general and three intensity categories of it (vigorous, moderate, and walking) were examined as binary outcome variables. Allele frequencies were determined, individual correlations of SNPs to LTPA, in general, were determined, and an optimized polygenetic score (oPGS) was created. Our results showed that the allele frequencies of four SNPs differed significantly between the two study groups. The C allele of rs10887741 showed a significant positive correlation with LTPA in general (OR = 1.48, 95% CI: 1.12-1.97; p = 0.006). Three SNPs (rs10887741, rs6022999, and rs7023003) were identified by the process of PGS optimization, whose cumulative effect shows a strong significant positive association with LTPA in general (OR = 1.40, 95% CI: 1.16-1.70; p < 0.001). The oPGS showed a significantly lower value in the Roma population compared with the HG population (oPGSRoma: 2.19 ± SD: 0.99 vs. oPGSHG: 2.70 ± SD: 1.06; p < 0.001). In conclusion, the coexistence of genetic factors that encourage leisure-time physical activity shows a more unfavorable picture among Roma, which may indirectly contribute to their poor health status.