Prior studies have reported inconsistent results or less well-explored associations between sex hormones and non-alcoholic fatty liver disease (NAFLD). Here, we aimed to investigate the associations of NAFLD with sex steroids and sex hormone-binding globulin (SHBG) in the population-based study and conduct a comprehensive systematic review and meta-analysis of all published observational studies. Analyses included 755 men and 1109 women with available data on sex steroids, SHBG, and ultrasound-based NAFLD from the Rotterdam Study. Multivariable regression models were used to examine the associations. Additionally, we searched five databases from inception to 1 April 2022 and performed a systematic review and meta-analysis. Random-effects (DerSimonian-Laird) method was used for meta-analysis, odds ratios (ORs) were calculated for the effect estimate, subgroup and leave-one-out sensitivity analyses were conducted, and meta-regression was performed to explore the pooled statistics with high heterogeneity. In the Rotterdam Study, lower levels of SHBG were associated with NAFLD in both sexes, while lower testosterone was associated with NAFLD only among women. Similarly, the meta-analysis of 16 studies indicated no sex-specific association between SHBG and NAFLD (men: OR = 0.37, 95%CI 0.21-0.53; women: OR = 0.40, 95%CI 0.21-0.60), yet there was a sex-specific association between testosterone and NAFLD (men: OR = 0.59, 95%CI 0.42-0.76; women: OR = 1.06, 95%CI 0.68-1.44). Moreover, men with NAFLD had lower estradiol levels than those without NAFLD. Lower SHBG levels were associated with NAFLD in both sexes, but testosterone levels were associated in a sex-specific manner. In addition, our results showed estradiol with the potential as a protective factor for NAFLD in healthy men.