Physical distancing has been an important policy to mitigate the spread of the novel coronavirus disease 2019 (COVID-19) in public settings. However, the current 1-2 m physical distancing rule is based on the physics of droplet transport and could not directly translate into infection risk. We therefore revisit the 2-m physical distancing rule by developing an infection-risk-based model for human speaking. The key modeling framework components include viral load, droplets dispersion and evaporation, deposition efficiency, viral dose-response rate and infection risk. The results suggest that the one-size-fits-all 2-m physical distancing rule derived from the pure droplet-physics-based model is not applicable under some realistic indoor settings, and may rather increase transmission probability of diseases. Especially, in thermally stratified environments, the infection risk could exhibit multiple peaks for a long distance beyond 2 m. With Sobol's sensitivity analysis, most variance of the risk is found to be significantly attributable to the variability in temperature gradient, exposure time and breathing height difference. Our study suggests there is no such magic 2 m physical distancing rule for all environments, but it needs to be used alongside other strategies, such as using face cover, reducing exposure time, and controlling the thermal stratification of indoor environment.