This cross-sectional study aimed to evaluate the impact of physical activity and physical fitness on the health-related quality of life (HQoL) of adult patients with Juvenile Idiopathic Arthritis (JIA). Fifty-nine JIA patients and sixty healthy individuals participated in this study. All individuals had the following evaluations performed: body composition (electrical bioimpedance), physical fitness (6 min walk test (6MWT)), physical activity level (International Physical Activity Questionnaire (IPAQ)), and HQoL (Quality of Life Questionnaire in relation to Health-Short Form (SF36)). Thirty-nine (66%) JIA patients were considered sedentary compared with 15 (25%) in the control group (p < 0.01). JIA patients had a lower HQoL compared with the control group in all variables studied (p < 0.05). JIA patients who were very physically active had better HQoL conditions in the categories of functional capacity (p = 0.001), limitations by physical aspects (p = 0.003), and emotional aspects (p = 0.002) compared with sedentary patients. JIA patients had more cardiovascular abnormalities and walked shorter distances compared with healthy controls in the 6MWT. In conclusion, we observed that HQoL was reduced in adults with JIA. A high percentage of JIA patients were sedentary with lower physical fitness, but physically active patients had a better HQoL than sedentary patients. The duration of physical activity, rather than intensity, influenced the mental aspects of HQoL.