AzTrials
AzTrials

Breast NeoplasmsBUST-AI BenchNCT07500428

Construction of a Benchmark for Breast Ultrasound AI Interpretation and Performance Evaluation of Multimodal AI Models

An observational study of Breast Neoplasms and Breast Diseases, run by Peking Union Medical College Hospital at 1 site in China.

Recruiting

Looking for participants now. Contact the study team or a site below to ask about taking part.

Sites and contacts
Official record
Search all trials

Trial summary

This single-center, retrospective, observational study aims to construct a standardized benchmark evaluation system for intelligent breast ultrasound image interpretation and to systematically assess the diagnostic performance of current mainstream multimodal artificial intelligence (AI) models. Read more about this study

Who is it for?
Ages 18 to 75 years, women onlyHealthy volunteers can take part
Conditions the study is for
Breast Neoplasms, Breast Diseases, Ultrasonography
What is being tested?
Multimodal AI Model Diagnostic Evaluation
How many people
1,380 participants
Where
1 siteChina
Study dates
12 Mar 2026to 1 Mar 2027
Phase
Observational
Are there results?
Not yet: the study is under way

Key requirements

Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.

  • Ages 18 to 75 years
  • Women only
  • Healthy volunteers accepted
  • Adult, Older adult

Study population: De-identified breast ultrasound images from adult patients who underwent breast ultrasound examination at Peking Union Medical College Hospital between 2018 and 2025 with subsequent pathological confirmation, supplemented by images from published, ethics-approved, open-access breast ultrasound datasets (e.g., BUSI, BrEaST).

Who may be able to join

  • B-mode breast ultrasound grayscale images from the institutional PACS database or from published open-access breast ultrasound datasets with documented original institutional ethics approval
  • Image quality adequate for clinical diagnosis with clear visualization of the region of interest
  • Pathological diagnosis confirmed (for benign and malignant lesion groups), or normal breast status confirmed by a senior radiologist with >15 years of breast ultrasound experience (for the normal group)
  • Complete de-identification with removal of all personally identifiable information

Who cannot join

  • Severely degraded image quality precluding meaningful BI-RADS assessment
  • Duplicate images from the same patient (only the most representative image retained per lesion)
  • Images with residual personally identifiable information after de-identification processing
  • Cases with ambiguous, disputed, or unavailable pathological results
  • Non-B-mode ultrasound images, including elastography, contrast-enhanced ultrasound, and Doppler imaging

Trial locations and contacts

1 site in 1 country. Contact the study team or a site near you to ask about taking part.

Qingli Zhu, MD contact

+86 13621376699

zqlpumch@126.com

Yinglan Wu, MD contact

+86 15626121076

wuylan7@gmail.com

China1 site, 1 recruiting
  • Peking Union Medical College Hospital

    Beijing, 100730

    Qingli Zhu, MD, +86 13621376699, zqlpumch@126.com

    Recruiting

Other trials in China

About this study

Official title: Construction of a Standardized Benchmark Evaluation System for Intelligent Breast Ultrasound Image Interpretation and Systematic Performance Assessment of Multimodal Artificial Intelligence Models Based on ACR BI-RADS v2025 Criteria

This single-center, retrospective, observational study aims to construct a standardized benchmark evaluation system for intelligent breast ultrasound image interpretation and to systematically assess the diagnostic performance of current mainstream multimodal artificial intelligence (AI) models.

De-identified B-mode breast ultrasound images with confirmed pathological diagnoses will be retrospectively collected from the institutional archive (2018-2025) and supplemented with images from published open-access datasets. Expert radiologists with varying experience levels will independently annotate all images according to the American College of Radiology (ACR) Breast Imaging Reporting and Data System (BI-RADS) v2025 criteria, including glandular tissue composition, lesion characterization (mass vs. non-mass lesion), morphological descriptors, and final BI-RADS classification.

Read more

Baseline deep learning models (CNN-based ResNet-50 and Transformer-based USFM) will be trained to establish performance baselines and to stratify cases by diagnostic difficulty through cross-architecture consensus. Multiple multimodal large language models (MLLMs), including both general-purpose and medical-domain models, will then be evaluated via standardized API calls using BI-RADS-guided chain-of-thought prompts at temperature 0 for reproducibility.

Primary endpoints include BI-RADS classification accuracy and diagnostic AUC for benign-malignant differentiation. Model robustness and safety will be assessed through out-of-distribution rejection testing, temperature-stability experiments, and thinking-mode ablation studies. This study adheres to the FLAIR and TRIPOD-LLM reporting guidelines.

Read the detailed description

Background: Breast cancer is the most prevalent malignancy among women worldwide. Ultrasound is a first-line screening modality, particularly in Asian populations with dense breast tissue where mammographic sensitivity is limited. However, ultrasound interpretation is highly operator-dependent, with substantial inter-observer variability in BI-RADS classification, especially for category 4A-4B lesions. Multimodal large language models (MLLMs) have emerged as a promising tool for medical image analysis due to their zero-shot diagnostic capability, interpretable chain-of-thought reasoning, and structured report generation. Nevertheless, there is currently no standardized benchmark for evaluating AI performance in breast ultrasound interpretation.

Study Design: Approximately 1,380 breast ultrasound images will be curated (1,200 evaluation set + 150 out-of-distribution safety test set + 30 prompt development set), encompassing three diagnostic categories: normal breast, benign lesions (BI-RADS 2-4B), and malignant lesions (BI-RADS 3-5). Two junior radiologists (<5 years of experience) and two senior radiologists (>15 years) will independently annotate images per ACR BI-RADS v2025 with arbitration by a fifth expert for discordant cases.

Diagnostic difficulty will be stratified into three tiers using cross-architecture deep learning consensus: Tier 1 (straightforward, both models correct), Tier 2 (equivocal, one correct/one incorrect), and Tier 3 (difficult, both incorrect, with senior expert validation). MLLMs will be evaluated across multiple dimensions: classification accuracy, sensitivity, specificity, F1 score, AUC, Cohen's kappa agreement with expert consensus, expected calibration error (ECE), morphological feature description accuracy, and chain-of-thought reasoning quality.

Safety Assessment: (1) Out-of-distribution rejection test using 150 non-diagnostic images (degraded images, non-breast ultrasound, other imaging modalities); (2) Temperature-stability pre-experiment across parameter settings; (3) Thinking-mode ablation comparing standard vs. chain-of-thought reasoning modes. All experiments use fixed model snapshots, system fingerprint monitoring, and complete logging for reproducibility.

Being tested
Multimodal AI Model Diagnostic Evaluation
Keywords
Breast Ultrasound, BI-RADS, Artificial Intelligence, Multimodal Large Language Model, Benchmark, Computer-Aided Diagnosis
Medical subject headings
Breast Neoplasms, Breast Diseases

What participants receive

What participants receive. In a randomised study each person is assigned to one group by chance.

  • Multimodal AI Model Diagnostic EvaluationDiagnostic test

    Retrospective evaluation of de-identified breast ultrasound images by multiple AI systems, including baseline deep learning models (ResNet-50, USFM) and multimodal large language models, using standardized BI-RADS-guided chain-of-thought prompts via API. No patient contact or clinical decision-making is involved.

Study groups in NCT07500428
Study groupRoleWhat they receive
Normal BreastBreast ultrasound images showing normal glandular tissue across different tissue composition types, with no focal lesions identified. Confirmed by senior radiologist review.
Receives: Diagnostic Test: Multimodal AI Model Diagnostic Evaluation
Benign LesionBreast ultrasound images containing pathologically confirmed benign lesions (BI-RADS 2-4B), including fibroadenoma, cyst, lipoma, sclerosing adenosis, intraductal papilloma, and selected non-mass lesions (NML).
Receives: Diagnostic Test: Multimodal AI Model Diagnostic Evaluation
Malignant LesionBreast ultrasound images containing pathologically confirmed malignant lesions (BI-RADS 3-5), including invasive ductal carcinoma, invasive lobular carcinoma, mucinous carcinoma, and selected non-mass lesions (NML).
Receives: Diagnostic Test: Multimodal AI Model Diagnostic Evaluation

What is being measured

The primary outcome is the main question the study is designed to answer.

  1. Diagnostic Accuracy for Pathological Diagnosis

    Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score of AI models for benign-malignant classification, with histopathological diagnosis as the gold standard.

    Measured: At study completion, approximately 12 months

  2. BI-RADS Classification Accuracy

    Overall accuracy of AI models in assigning BI-RADS categories (2, 3, 4A, 4B, 4C, 5) to breast ultrasound images, compared with expert consensus annotation as the reference standard.

    Measured: At study completion, approximately 12 months

Secondary outcomes (3)
  1. Agreement with Expert Consensus (Cohen's Kappa)

    Cohen's kappa coefficient measuring agreement between each AI model's BI-RADS classification and the expert consensus annotation, reported with 95% confidence intervals.

    Measured: At study completion, approximately 12 months

  2. Out-of-Distribution Rejection Rate

    Proportion of non-diagnostic images (degraded quality, non-breast ultrasound, other imaging modalities) correctly identified and refused by AI models, evaluating domain safety.

    Measured: At study completion, approximately 12 months

  3. Sensitivity, Specificity, PPV, NPV, and F1 Score

    Standard diagnostic performance metrics for benign-malignant classification, reported for each AI model individually.

    Measured: At study completion, approximately 12 months

Timeline

From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.

  1. First posted30 Mar 2026
  2. Study start12 Mar 2026
  3. Primary completion1 Dec 2026 (estimated)
  4. Study completion1 Mar 2027 (estimated)

Resources

More details

For doctors and researchers: how the study is designed, who runs it, and its registry IDs.

Study design
Study type
Observational
Observational model
Cohort
Time perspective
Retrospective
Enrolment
1,380 participants (estimated)
IDs, dates and publications
ClinicalTrials.gov ID
NCT07500428
Sponsor's protocol ID
K10349
Other IDs
2024-I2M-CT-B-035, I-26PJ0568
Acronym
BUST-AI Bench
Status last verified
Mar 2026
First submitted
24 Mar 2026
Last updated
30 Mar 2026
Results posted
No
Data monitoring committee
No
FDA-regulated drug
No
FDA-regulated device
No
Individual data sharing
Yes: The de-identified benchmark evaluation dataset, including expert-annotated breast ultrasound images with paired BI-RADS reading reports, is planned for public release to promote academic reproducibility and collaborative research.

Publications

  • Bi WL, Hosny A, Schabath MB, Giger ML, Birkbak NJ, Mehrtash A, Allison T, Arnaout O, Abbosh C, Dunn IF, Mak RH, Tamimi RM, Tempany CM, Swanton C, Hoffmann U, Schwartz LH, Gillies RJ, Huang RY, Aerts HJWL. Artificial intelligence in cancer imaging: Clinical challenges and applications. CA Cancer J Clin. 2019 Mar;69(2):127-157. doi: 10.3322/caac.21552. Epub 2019 Feb 5.

    PubMed 30720861
  • Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations. Radiology. 2023 Jun;307(5):e230582. doi: 10.1148/radiol.230582. Epub 2023 May 16.

    PubMed 37191485
  • Clusmann J, Kolbinger FR, Muti HS, Carrero ZI, Eckardt JN, Laleh NG, Loffler CML, Schwarzkopf SC, Unger M, Veldhuizen GP, Wagner SJ, Kather JN. The future landscape of large language models in medicine. Commun Med (Lond). 2023 Oct 10;3(1):141. doi: 10.1038/s43856-023-00370-1.

    PubMed 37816837
  • Seyyed-Kalantari L, Zhang H, McDermott MBA, Chen IY, Ghassemi M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat Med. 2021 Dec;27(12):2176-2182. doi: 10.1038/s41591-021-01595-0. Epub 2021 Dec 10.

    PubMed 34893776
  • Moor M, Banerjee O, Abad ZSH, Krumholz HM, Leskovec J, Topol EJ, Rajpurkar P. Foundation models for generalist medical artificial intelligence. Nature. 2023 Apr;616(7956):259-265. doi: 10.1038/s41586-023-05881-4. Epub 2023 Apr 12.

    PubMed 37045921
  • Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin. 2021 May;71(3):209-249. doi: 10.3322/caac.21660. Epub 2021 Feb 4.

    PubMed 33538338
  • Benary M, Wang XD, Schmidt M, Soll D, Hilfenhaus G, Nassir M, Sigler C, Knodler M, Keller U, Beule D, Keilholz U, Leser U, Rieke DT. Leveraging Large Language Models for Decision Support in Personalized Oncology. JAMA Netw Open. 2023 Nov 1;6(11):e2343689. doi: 10.1001/jamanetworkopen.2023.43689.

    PubMed 37976064
  • Miaojiao S, Xia L, Xian Tao Z, Zhi Liang H, Sheng C, Songsong W. Using a Large Language Model for Breast Imaging Reporting and Data System Classification and Malignancy Prediction to Enhance Breast Ultrasound Diagnosis: Retrospective Study. JMIR Med Inform. 2025 Jun 11;13:e70924. doi: 10.2196/70924.

    PubMed 40498674
  • Jiao J, Zhou J, Li X, Xia M, Huang Y, Huang L, Wang N, Zhang X, Zhou S, Wang Y, Guo Y. USFM: A universal ultrasound foundation model generalized to tasks and organs towards label efficient image analysis. Med Image Anal. 2024 Aug;96:103202. doi: 10.1016/j.media.2024.103202. Epub 2024 May 15.

    PubMed 38788326
  • Xiang H, Wang X, Xu M, Zhang Y, Zeng S, Li C, Liu L, Deng T, Tang G, Yan C, Ou J, Lin Q, He J, Sun P, Li A, Chen H, Heng PA, Lin X. Deep Learning-assisted Diagnosis of Breast Lesions on US Images: A Multivendor, Multicenter Study. Radiol Artif Intell. 2023 Jul 12;5(5):e220185. doi: 10.1148/ryai.220185. eCollection 2023 Sep.

    PubMed 37795135
  • Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, Ghassemi M, Liu X, Reitsma JB, van Smeden M, Boulesteix AL, Camaradou JC, Celi LA, Denaxas S, Denniston AK, Glocker B, Golub RM, Harvey H, Heinze G, Hoffman MM, Kengne AP, Lam E, Lee N, Loder EW, Maier-Hein L, Mateen BA, McCradden MD, Oakden-Rayner L, Ordish J, Parnell R, Rose S, Singh K, Wynants L, Logullo P. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024 Apr 16;385:e078378. doi: 10.1136/bmj-2023-078378.

    PubMed 38626948
  • Kottlors J, Iuga AI, Bluethgen C, Bressem K, Kather JN, Moy L, Wald C, Wang W, Liu T, Ranschaert E, Dratsch T, Kleesiek J, Gertz RJ, Rajpurkar P, Bedayat A, Fink MA, Zeeck A, Chaudhari A, Alkasab T, Wu H, Nensa F, Wang B, Grosse Hokamp N, Laukamp KR, Persigehl T, Maintz D, Truhn D, Lennartz S. Guidelines for Reporting Studies on Large Language Models in Radiology: An International Delphi Expert Survey. Radiology. 2026 Feb;318(2):e250913. doi: 10.1148/radiol.250913.

    PubMed 41631991

Questions about this trial

What is this clinical trial studying?

An observational study of Breast Neoplasms and Breast Diseases, run by Peking Union Medical College Hospital at 1 site in China.

The full registered description is in the About this study section and the official record.

Who may be eligible to take part?

The sponsor lists 4 inclusion and 5 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.

Is this clinical trial currently recruiting?

The record says: looking for participants now, as of the sponsor's last update on 30 Mar 2026. Status can change between updates, so confirm with the study contact before making plans.

Where is the study taking place?

The record lists 1 site in China. Whether a site is taking part right now should be confirmed with that site. See every site.

What are the possible benefits and risks?

They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.

Are results available for this study?

This study is still under way. Results are not available in the source record yet.

How can a doctor refer a patient or contact the research team?

Contact Qingli Zhu, MD (+86 13621376699, zqlpumch@126.com), or a site near the patient. AzTrials does not enrol participants or take referrals.

What does the primary outcome measure?

Diagnostic Accuracy for Pathological Diagnosis (time frame: At study completion, approximately 12 months). The study lists 2 primary outcomes in all. It is what the study was designed to measure, not a finding.

Where can I read the original study information?

On the ClinicalTrials.gov record for NCT07500428, which also lists 12 linked publications.

Does a listing mean the treatment is approved or proven to work?

No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.

How often is this information updated?

AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 30 Mar 2026.

Free account

Log in to keep searching

You’ve used your 3 free searches. AzTrials is free: log in or create a free account to keep searching clinical trials and save the ones you want to follow.

Create a free accountLog in