Background: Inflammation is a key feature shaping the microenvironment of hepatocellular carcinoma (HCC), and liver fibrosis is associated with the prognosis of patients with HCC. In this study, we investigated whether baseline inflammation-based scores and serum fibrosis markers can help in predicting the prognosis of HCC patients treated with transarterial chemoembolization (TACE). Methods: A total of 605 consecutive patients with HCC treated by TACE were included. The systemic immune-inflammation index (SII), neutrophil−lymphocyte ratio (NLR), platelet−lymphocyte ratio (PLR), FIB-4 index, and aspartate aminotransferase-to-platelet ratio index (APRI) were analyzed regarding their associations with disease progression and survival. Results: All tested inflammation-based scores and fibrosis markers were significantly associated with tumor progression and overall survival in the univariate analyses. In the multivariate analysis, NLR (hazard ratio [HR], 1.06; p = 0.007) and FIB-4 (HR = 1.02, p = 0.008) were independent risk factors for disease progression, along with α-fetoprotein (AFP) levels, maximum tumor size and number, and presence of vascular invasion. Furthermore, NLR (HR, 1.09; p < 0.001) and FIB-4 (HR, 1.02; p = 0.02) were independent prognostic factors for survival. Conclusions: High baseline NLR and FIB-4 levels might help the prediction of disease progression and death in patients with HCC after TACE.