With the rapidly aging population, taxi drivers are aging at a fast pace, and competition in the taxi industry is intensifying due to the emergence of various transportation platforms. A descriptive secondary data study was conducted (on a total of 936 subjects) to determine the factors affecting the deterioration of taxi drivers’ physical health status (PHS) according to their age group. The increased incidence of chronic diseases and cognitive decline among taxi drivers aged 55−64 years had the greatest influence on the deterioration of their PHS. Driver obesity was more likely to be related to deterioration of the PHS in the drivers aged 55−64 years (OR: 2.459, <0.001) and 35−54 years (OR: 2.133, <0.001). Among the financial factors, a driver’s income and their number of dependent family members were correlated with the deterioration of the PHS for drivers aged 55 years or over. Therefore, chronic diseases, obesity and cognitive decline were related with deterioration of the physical health status. This suggests that attention should be paid to healthcare policies not only for the elderly aged over 65 years but also those aged 50 to 64 years, i.e., middle-aged people at the beginning of the transition to old age.