BACKGROUND Photon-counting detector CT (PCD-CT) is a new CT technology that offers enhanced spatial and spectral imaging performances. As a new technology, conditioning and qualifying its precise performance can benefit from a comprehensive framework to evaluate task-generic and task-specific image qualities. OBJECTIVE To develop and validate a customizable and physics-informed simulation framework capable of modeling spatio-energetic detector responses for various PCD designs, integrate it into a virtual imaging framework, and demonstrate its applicability in clinically relevant imaging tasks. METHODS A customizable simulation model, DukeCounter, was developed to replicate real PCD-CT systems. Photon transport and crosstalk in PCDs were modeled using Monte Carlo simulations, and charge sharing was implemented using an analytical Gaussian charge cloud model. The fundamental interactions in PCDs, including photoelectric absorption, Compton and fluorescence x-ray scatterings, charge cloud formation, and charge diffusion and repulsion, were modeled. Spatio-energetic detector responses were generated for face-on CdTe-, CZT-, GaAs-, and edge-on Si-based PCDs. These responses, combined with standardized scanner parameters, were integrated into a CT simulator to create virtual DukeCounter PCD-CT scanners. The framework was benchmarked against experimental data from a clinical CdTe-based PCD-CT scanner across three dose levels. To demonstrate its utility, three pilot studies were conducted using a computational ACR phantom for task-generic image quality assessment, an XCAT model with bronchitis and emphysema for COPD biomarker extraction, and an XCAT with liver lesions for lesion detectability analysis. RESULTS The simulated charge cloud size increased with energy and was more pronounced in Si due to its low atomic number. The detector response across a 3 × 3-pixel neighborhood varied with PCD material, design, and energy threshold settings. Validation results demonstrated strong agreement between simulated and real ACR images. For the 20-keV-threshold images, the mean relative difference (MRD) in f50 of MTF was 4.15% ± 1.21 for air and 2.54% ± 2.08 for bone, and the MRD in fav of NPS was 0.83% ± 0.97. The MRDs in noise magnitude were 2.65% ± 1.68, 3.05% ± 1.97, and 2.78% ± 1.79 for the 20-keV-threshold, 65-keV-threshold, and 70-keV-VMI images, respectively. The MRDs in CT number for the same image types were 0.03% ± 0.03, 0.11% ± 0.09, 0.11% ± 0.05 for air, and 1.85% ± 0.20, 1.84% ± 0.55, 0.50% ± 0.36 for polyethylene. DukeCounter-generated images showed that task-generic and task-specific image qualities were influenced by PCD materials, designs, and energy threshold settings. GaAs-based DukeCounter exhibited the highest image noise, the largest error in COPD biomarker quantification, and the lowest performance in liver lesion detection, under consistent acquisition and reconstruction settings. CONCLUSIONS A customizable, modular simulation framework was developed to model spatio-energetic detector responses for various PCD materials and designs. The detector responses were integrated into a CT simulation pipeline to build DukeCounter PCD-CT systems. The framework's utility was demonstrated through task-specific assessments of image quality and clinical performance of DukeCounter systems using XCAT phantoms. This approach enables systematic PCD-CT design evaluation and optimization, supporting translational research in medical imaging by reducing the cost, time, and radiation burden of physical experiments.