In digital holography (DH), the quality of the reconstructed images relies on the accuracy of the reconstruction distance. Existing autofocusing approaches primarily determine the reconstruction distance by evaluating the sharpness of the features in reconstructed images under different reconstruction distances. The maximum sharpness corresponds to the optimal reconstruction distance. However, the existing approaches often fail for diffuse samples and specular ones with no features. The main challenges are as follows: (1) The spatial features, including edge, contrast, and sparsity do not vary obviously by varying the reconstruction distance. (2) The spectral features do not vary obviously as well by varying the reconstruction distance. Therefore, we propose an autofocusing approach based on phase difference to tackle the above problems. The proposed approach consists of three steps: (1) introducing a phase difference into the measured sample as an artificial feature; (2) recording two holograms before and after the phase change and obtaining the phase difference by reconstruction; and (3) taking the phase difference image as the feature and determining the optimal reconstruction distance by using image sharpness evaluation algorithms. According to simulation and practical experiments, the proposed approach has successfully solved the autofocusing problem of the specular samples with no features and diffuse samples.