Neoadjuvant therapy for locally advanced rectal cancer is becoming increasingly common. However, biomarkers predicting the response to neoadjuvant therapy have not been established. Tumor-infiltrating lymphocytes (TILs) have a crucial effect on tumor progression and survival outcome as the primary host immune response, and an antitumor immune effect has been reported to contribute to the response to radiotherapy and chemotherapy. We investigated the significance of TILs before and after neoadjuvant treatment and the change in the density of those TILs. Sixty-four patients who underwent radical resection after neoadjuvant treatment for locally advanced rectal cancer were enrolled. The number of TIL subsets was examined using immunohistochemical staining of pretreatment biopsy samples and post-treatment resected specimens. In both the neoadjuvant chemotherapy cohort and the neoadjuvant chemoradiotherapy cohort, a low density of CD8+ TILs in pretreatment biopsy samples was associated with a poor response, and a low density of CD8+ TILs in post-treatment resected specimens was similarly associated with a poor response. In the neoadjuvant chemoradiotherapy cohort, the density of CD8+ TILs in post-treatment resected specimens was significantly increased compared with that in pretreatment biopsy samples. We concluded that T lymphocyte-mediated immune reactions play an important role in tumor response to neoadjuvant treatment for rectal cancer, and the evaluation of TILs in pretreatment biopsy samples might be a predictor of the clinical effectiveness of neoadjuvant treatment. Furthermore, neoadjuvant therapy, especially chemoradiotherapy, could induce the activation of the local immune status.