Many health-care workers exposed to coronavirus disease 2019 (COVID-19) are psychologically distressed. This study aimed to investigate the psychological impact of the COVID-19 pandemic on hospital workers under the emergency declaration in Japan. This cross-sectional, survey-based study collected sociodemographic data and responses to 19 stress-related questions and the Impact of Event Scale-Revised (IES-R), which measures post-traumatic stress disorder (PTSD) symptoms, from all 3217 staff members at Kobe City Medical Center General Hospital from April 16, 2020 to June 8, 2020. Exploratory factor analysis was applied to the 19 stress-related questions. Multiple regression models were used to evaluate the association of personal characteristics with each score of the four factors and the IES-R. We received 951 valid responses; 640 of these were by females, and 311 were by respondents aged in their 20s. Nurses accounted for the largest percentage of the job category. Women, those aged in their 30s-50s, nurses, and frontline workers had a high risk of experiencing stress. The prevalence of stress (IES-R ≥ 25) was 16.7%. The psychological impact was significantly greater for those aged in their 30s-50s and those who were not medical doctors. This is the first study to examine the stress of hospital workers, as measured by the IES-R, under the emergency declaration in Japan. It showed that women, those aged in their 30s-50s, nurses, and frontline workers have a high risk of experiencing stress. Health and medical institutions should pay particular attention to the physical and psychological health of these staff members.