Colloidal quantum dots (CQDs) have a unique advantage in realizing near-infrared (NIR) photodetection since their optical properties are readily tuned by the particle size, but CQD-based photodetectors (QPDs) presently show a high dark current density (Jd) and insufficient dynamic characteristics. To overcome these two problems, we synthesized and introduced two types of conjugated polymers (CPs) by replacing the p-type CQD layer in the QPDs. The low dielectric constant and insulating properties of CPs under dark conditions effectively suppressed the Jd in the QPDs. In addition, the energy-level alignment and high-hole mobility of the CPs facilitated hole transport. Therefore, both the responsivity and specific detectivity were highly enhanced in the CP-based QPDs. Notably, the dynamic characteristics of the QPDs, such as the -3 dB cut-off frequency and rising/falling response times, were significantly improved in the CP-based QPDs owing to the sizable molecular ordering and fast hole transport of the CP in the film state as well as the low trap density, well-aligned energy levels, and good interfacial contact in the CP-based devices.