Compared with the traditional BB84 protocol, the counterfactual quantum key distribution (QKD) does not rely on any signal travelling in the quantum channel, and therefore can present a security advantage where Eve cannot fully access signal. However, the practical system may be damaged in a scenario where the devices are untrusted. In this paper, we analyze the security of counterfactual QKD in untrusted detectors scenario. We show that the requirement to disclose "which detector clicked" has become the main loophole in all counterfactual QKD versions. An eavesdropping scheme which is similar to the memory attack on device-independent QKD could break its security by exploiting detectors' imperfections. We consider two different counterfactual QKD protocols and analyze their security against this major loophole. One is a modified Noh09 protocol, which would be secure in untrusted detectors context. Another is a variant of counterfactual QKD with high efficiency (Phys. Rev. A 104 (2021) 022424) against a series of detectors side-channel attacks as well as against other attacks that exploit detectors imperfections.