In this paper, we propose a new face de-identification method based on generative adversarial network (GAN) to protect visual facial privacy, which is an end-to-end method (herein, FPGAN). First, we propose FPGAN and mathematically prove its convergence. Then, a generator with an improved U-Net is used to enhance the quality of the generated image, and two discriminators with a seven-layer network architecture are designed to strengthen the feature extraction ability of FPGAN. Subsequently, we propose the pixel loss, content loss, adversarial loss functions and optimization strategy to guarantee the performance of FPGAN. In our experiments, we applied FPGAN to face de-identification in social robots and analyzed the related conditions that could affect the model. Moreover, we proposed a new face de-identification evaluation protocol to check the performance of the model. This protocol can be used for the evaluation of face de-identification and privacy protection. Finally, we tested our model and four other methods on the CelebA, MORPH, RaFD, and FBDe datasets. The results of the experiments show that FPGAN outperforms the baseline methods.