Lead sulfide colloidal quantum dots (PbS CQDs) have shown great potential in photodetectors owing to their promising optical properties, especially their strong and tunable absorption. However, the limitation of the specific detectivity (D*) in CQD near-infrared (NIR) photodetectors remains unknown due to the ambiguous noise analysis. Therefore, a clear understanding of the noise current is critically demanded. Here, we elucidate that the noise current is the predominant factor limiting D*, and the noise is highly dependent on the trap densities in halide-passivated PbS films and the carriers injected from the Schottky contact (EDT-passivated PbS films/metal). It is found that the thickness of CQDs is proportional to their interface trap density, while it is inversely proportional to their minimal bulk trap density. A balance point can be reached at a certain thickness (136 nm) to minimize the trap density, giving rise to the improvement of D*. Utilizing thicker PbS-EDT films broadens the width of the tunneling barrier and thereby reduces the carrier injection, contributing to a further enhancement of D*. The limiting factors of D* determined in this work not only explain the physical mechanism of the influence on detection sensitivity but also give guidance to the design of high-performance CQD photodetectors.