Benchmark and performance test for AI reading breast ultrasound images
Construction of a Standardized Benchmark Evaluation System for Intelligent Breast Ultrasound Image Interpretation and Systematic Performance Assessment of Multimodal Artificial Intelligence Models Based on ACR BI-RADS v2025 Criteria
This project will test whether multimodal AI models can read breast ultrasound images to help classify normal, benign, and malignant findings for people who have had breast ultrasound.
Quick facts
| Study type | Observational |
|---|---|
| Enrollment | 1380 (estimated) |
| Ages | 18 Years to 75 Years |
| Sex | Female |
| Sponsor | Peking Union Medical College Hospital Academic / other |
| Locations | 1 site (Beijing) |
| Trial ID | NCT07500428 on ClinicalTrials.gov |
What this trial studies
This retrospective single-center project will collect de-identified B-mode breast ultrasound images from Peking Union Medical College Hospital (2018–2025) and from open-access datasets to create a standardized evaluation benchmark. Expert radiologists with junior and senior experience will independently annotate images using ACR BI-RADS v2025 criteria, with a senior arbitrator resolving disagreements. Baseline CNN (ResNet-50) and transformer (USFM) models will be trained, and multiple multimodal large language models will be tested on an evaluation set plus an out-of-distribution safety test set. Images will be stratified into difficulty tiers based on cross-architecture model consensus to characterize where models succeed or fail.
Who should consider this trial
Good fit: Ideal candidates for inclusion are cases with de-identified B-mode breast ultrasound images that have clear visualization of the lesion and a documented pathological diagnosis (or a confirmed normal status by an experienced radiologist).
Not a fit: Patients without B-mode ultrasound images, without pathology or confirmed normal status, with severely degraded images, or whose images were not shared/de-identified will not be included and therefore will not directly benefit from this project.
Why it matters
Potential benefit: If successful, the benchmark and model comparisons could improve diagnostic consistency and help models identify cancers more reliably on breast ultrasound, reducing variability between readers.
How similar studies have performed: Previous AI work using CNNs and other deep-learning approaches has shown promising performance on breast ultrasound, but multimodal large language model approaches are newer and standardized benchmarks for head-to-head comparison are still lacking.
Eligibility criteria
Show full inclusion / exclusion criteria
Inclusion Criteria: * B-mode breast ultrasound grayscale images from the institutional PACS database or from published open-access breast ultrasound datasets with documented original institutional ethics approval * Image quality adequate for clinical diagnosis with clear visualization of the region of interest * Pathological diagnosis confirmed (for benign and malignant lesion groups), or normal breast status confirmed by a senior radiologist with \>15 years of breast ultrasound experience (for the normal group) * Complete de-identification with removal of all personally identifiable information Exclusion Criteria: * Severely degraded image quality precluding meaningful BI-RADS assessment * Duplicate images from the same patient (only the most representative image retained per lesion) * Images with residual personally identifiable information after de-identification processing * Cases with ambiguous, disputed, or unavailable pathological results * Non-B-mode ultrasound images, including elastography, contrast-enhanced ultrasound, and Doppler imaging
Where this trial is running
Beijing
- Peking Union Medical College Hospital — Beijing, China (Recruiting)
Study contacts
- Principal investigator: Qingli Zhu, MD — Peking Union Medical College Hospital
- Study coordinator: Qingli Zhu, MD
- Email: zqlpumch@126.com
- Phone: +86 13621376699
How to participate
- Review the eligibility criteria above with your treating physician.
- Visit the official trial page on ClinicalTrials.gov for the most current contact information and recruitment status.
- Contact the listed study coordinator or principal investigator to request pre-screening. Pre-screening is free and never obligates you to enroll.