Radiotherapy with neutrons requires a large quantity of physical information about neutrons and their interaction with matter. Primarily the absorbed dose at a reference point needs to be determined. For treatment planning the dose distribution in a phantom must be measured and algorithms for the simulation of the dose distribution in a patient must be available. As neutrons interact with matter in a more complicated way as high-energy photons and electrons commonly used in radiotherapy, biological effects based on microdosimetric data are used for treatment planning. This paper presents a brief summary of the neutron sources used in radiotherapy. The dosimetry of the clinical neutron beams is described. Special aspects of the treatment planning with fast neutrons are discussed. For further radiobiological interpretation the fundamentals of microdosimetry are described. Finally recent and future developments in the field of physics for neutron therapy are mentioned.