Background: Physical fitness (PF) is an expression of the physiological functioning of multiple body components. PF is an important prognostic factor in terms of cardiovascular mortality, cancer mortality, and all-cause mortality. PF has been related to some biomarkers in the general population but not in breast cancer survivors (BCS). Purpose: To evaluate the effects of PF on biomarkers potentially related to physical activity (PA) in a sample of BCS. Methods: Cross-sectional study. A total of 84 BCS (mean age 54) who had finished their treatment were recruited. Different components of PF were evaluated, namely body composition (anthropometry), cardiorespiratory fitness (one-mile walk test), muscular (handgrip and sit-to-stand timed test), and motor (gait speed) components. Sexual hormones, inflammation, and insulin resistance biomarkers were measured. Results: C-Reactive Protein (CRP) was associated with every component of physical fitness: cardiorespiratory fitness (p-value = 0.002), muscular (sit-to-stand timed test, p-value = 0.002) and motor (gait speed, p-value = 0.004) components, and body composition (body mass index, p-value = 0.003; waist, p-value < 0.000; and waist-to-hip index, p-value = 0.012). CRP also was associated with “poor physical condition,” a constructed variable that encompasses all components of physical fitness (p-value < 0.001). Insulin was associated with cardiorespiratory fitness and gait speed (p-values = 0.002 and 0.024, respectively). Insulin-like Growth Factor-1 was negatively associated with waist perimeter and waist-to-hip ratio. Conclusions: CRP can also be considered an indicator of poor PF in BCS. Implications for cancer survivors: in case of elevation of CRP indicating cardiovascular risk, health professionals should recommend lifestyle changes to improve BCS physical condition.