A Hybrid Deep Learning Framework for Accurate Cell Segmentation in Whole Slide Images Using YOLOv11, StarDist, and SAM2. 2025

Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
Engineering Faculty, Electrical & Electronics Engineering Department, Dicle University, 21280 Diyarbakır, Türkiye.

Accurate segmentation of cellular structures in whole slide images (WSIs) is essential for quantitative analysis in computational pathology. However, the complexity and scale of WSIs present significant challenges for conventional segmentation methods. In this study, we propose a novel hybrid deep learning framework that integrates three complementary approaches, YOLOv11, StarDist, and Segment Anything Model v2 (SAM2), to achieve robust and precise cell segmentation. The proposed pipeline utilizes YOLOv11 as an object detector to localize regions of interest, generating bounding boxes or preliminary masks that are subsequently used either as prompts to guide SAM2 or to filter segmentation outputs. StarDist is employed to model cell and nuclear boundaries with high geometric precision using star-convex polygon representations, which are particularly effective in densely packed cellular regions. The framework was evaluated on a unique WSI dataset comprising 256 × 256 image tiles annotated with high-resolution cell-level masks. Quantitative evaluations using the Dice coefficient, intersection over union (IoU), F1-score, precision, and recall demonstrated that the proposed method significantly outperformed individual baseline models. The integration of object detection and prompt-based segmentation led to enhanced boundary accuracy, improved localization, and greater robustness across varied tissue types. This work contributes a scalable and modular solution for advancing automated histopathological image analysis.

UI MeSH Term Description Entries

Related Publications

Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
June 2021, Scientific reports,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
January 2020, NPJ digital medicine,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
April 2025, Journal of imaging,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
September 2025, Cancers,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
January 2022, Journal of pathology informatics,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
September 2025, Journal of the American Academy of Dermatology,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
December 2023, Bioengineering (Basel, Switzerland),
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
July 2019, Computers in biology and medicine,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
January 2025, Frontiers in oncology,
Julius Bamwenda, and Mehmet Siraç Özerdem, and Orhan Ayyıldız, and Veysı Akpolat
February 2026, International journal of medical informatics,
Copied contents to your clipboard!