Consensus DevelopmentPRISMA-AINCT05382455
Preferred Reporting Items for Systematic Reviews and Meta-Analyses - Artificial Intelligence Extension
An observational study of Consensus Development, run by University of Southern California at 1 site in United States.
Trial summary
The investigators aim to develop the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Artificial Intelligence Extension (PRISMA-AI) guideline as a stand-alone extension of the PRISMA statement, modified to reflect the particular requirements for the reporting of AI and its related topics (namely… Read more about this study
- Who is it for?
- Ages 18 years and older
- Conditions the study is for
- Consensus Development
- What is being tested?
- Delphi Questionnaire
- How many people
- 150 participants
- Where
- 1 siteUnited States
- Study dates
- 15 Jun 2022to 15 Jul 2022
- Phase
- Observational
- Are there results?
- None posted in the registry
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- No healthy volunteers
- Adult, Older adult
Study population: A team of experts in the use AI technology in medicine together with experts in PRISMA, STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI will evaluate the PRISMA-AI extension reporting guidelines.
Who may be able to join
- experts in the use AI technology in medicine
- experts in PRISMA
- leading authors of STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI
Who cannot join
- Panelists who were not able to commit to all rounds of the modified Delphi process will be excluded
Trial locations and contacts
1 site in 1 country.
United States1 site
University of Southern California
Los Angeles, California, 90005
About this study
Official title: Preferred Reporting Items for Systematic Reviews and Meta-Analyses - Artificial. Delphi Consensus
The investigators aim to develop the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Artificial Intelligence Extension (PRISMA-AI) guideline as a stand-alone extension of the PRISMA statement, modified to reflect the particular requirements for the reporting of AI and its related topics (namely machine learning, deep learning, neuronal networking) in systematic reviews.
Read the detailed description
With advances in artificial intelligence (AI) over the last two decades, enthusiasm and adoption of this technology in medicine have steadily increased. Yet despite the greater adoption of AI in medicine, the way such methodologies and results are reported varies widely and the readability of clinical studies utilizing AI can be challenging to the general clinician.
Systematic reviews of AI applications are an important area for which specific guidance is needed. An ongoing systematic review led by our team has shown that the number of systematic reviews on AI applications (with or without meta-analysis) is increasing dramatically over the time, yet the quality of reporting is still poor and heterogeneous, leading to inconsistencies in the reporting of informational details among individual studies. Consequently, the lack of these informational details may front problems for primary research and synthesis and potentially limits their usefulness for stakeholders interested in implementing AI or using the information in systematic reviews.
The criteria will derive from the consensus among multi-specialty experts (in each medical specialty) who have already published about AI applications in leading medical journals and the lead authors of PRISMA, STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI to ensure that the criteria have global applicability in all the disciplines and for each type of study which involves the AI.
The proposed PRISMA-AI extension criteria focus on standardizing the reporting of methods and results for clinical studies utilizing AI. These criteria will reflect the most relevant technical details a data scientist requires for future reproducibility, yet they focus on the ability for the clinician reader to critically follow and ascertain the relevant outcomes of such studies.
The resultant PRISMA-AI extension will
- help stakeholders interested in implementing AI or using AI-related information in systematic reviews
- create a framework for reviewers that assess publications,
- provide a tool for training researchers on Artificial Intelligence SR methodology
- help end-users of the SR such as physicians and policymakers to better evaluate SR's validity and applicability in their decision-making process.
The success of the criteria will be seen in how manuscripts are written, how peer reviewers assess them, and finally, how the general readership is able to read and digest the published studies
- Conditions
- Consensus Development
- Being tested
- Delphi Questionnaire
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
Delphi QuestionnaireOther
An invitation email, including a link to the survey, will be sent to the panel of experts in Ai in healthcare.
The Delphi questionnaire will be administered via Welphi.com. In the first survey, panel members will outline the AI reporting standards in systematic reviews and objectively identify critical aspects of reporting methodology and results.
In subsequent surveys, the expert panel will evaluate the modified criteria using a 1 to 5-point Likert scale with space provided for suggested edits and comments. Multiple rounds will be conducted until consensus is reached. After each round of Likert responses, the study team will calculate the agreement and distribution of responses. Likert responses will be dichotomized with positive values indicating agreement and neutral or negative values indicating disagreement.
For the questions that do not reach a consensus of more than 80% in the first round or need further explanation, additional rounds of the survey may be performed.
| Study group | Role | What they receive |
|---|---|---|
| Delphi Panel | A team of experts in the use AI technology in medicine together with experts in PRISMA, STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI will evaluate the PRISMA-AI extension reporting guidelines Receives: Other: Delphi Questionnaire |
What is being measured
The primary outcome is the main question the study is designed to answer.
Degree of consensus
The level of agreement for all statements achieving consensus from the expert panel; consensus is predefined as ≥ 80% of the panel rating a given statement
Measured: 3 months
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted19 May 2022
- Study start15 Jun 2022 (estimated)
- Primary completion30 Jun 2022 (estimated)
- Study completion15 Jul 2022 (estimated)
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Publications (5)Papers the sponsor has linked to this study.
- Other trials for Consensus DevelopmentStudies of the same condition, open and under way.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Observational
- Observational model
- Cohort
- Time perspective
- Prospective
- Enrolment
- 150 participants (estimated)
Sponsor and investigators
- Lead sponsor
- University of Southern California
- Sponsor type
- University, hospital or other
- Responsible party
- Principal investigator, Giovanni Cacciamani, University of Southern California
- Investigators
- Giovanni Cacciamani, MD, Study chair, University of Southern California
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT05382455
- Sponsor's protocol ID
- UP-22-00370
- Acronym
- PRISMA-AI
- Status last verified
- May 2022
- First submitted
- 2 May 2022
- Last updated
- 19 May 2022
- Results posted
- No
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- No
Publications
Ibrahim H, Liu X, Rivera SC, Moher D, Chan AW, Sydes MR, Calvert MJ, Denniston AK. Reporting guidelines for clinical trials of artificial intelligence interventions: the SPIRIT-AI and CONSORT-AI guidelines. Trials. 2021 Jan 6;22(1):11. doi: 10.1186/s13063-020-04951-6.
PubMed 33407780Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ; SPIRIT-AI and CONSORT-AI Working Group. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health. 2020 Oct;2(10):e549-e560. doi: 10.1016/S2589-7500(20)30219-3. Epub 2020 Sep 9.
PubMed 33328049Sounderajah V, Ashrafian H, Aggarwal R, De Fauw J, Denniston AK, Greaves F, Karthikesalingam A, King D, Liu X, Markar SR, McInnes MDF, Panch T, Pearson-Stuttard J, Ting DSW, Golub RM, Moher D, Bossuyt PM, Darzi A. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat Med. 2020 Jun;26(6):807-808. doi: 10.1038/s41591-020-0941-1. No abstract available.
PubMed 32514173Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, Logullo P, Beam AL, Peng L, Van Calster B, van Smeden M, Riley RD, Moons KG. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021 Jul 9;11(7):e048008. doi: 10.1136/bmjopen-2020-048008.
PubMed 34244270Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021 Mar 29;372:n71. doi: 10.1136/bmj.n71.
PubMed 33782057
Study at a glance
From the registry record, last updated by the sponsor on 19 May 2022. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Preferred Reporting Items for Systematic Reviews and Meta-Analyses - Artificial. Delphi Consensus
- ClinicalTrials.gov ID
- NCT05382455
- Acronym / sponsor ID
- PRISMA-AI / UP-22-00370
- Conditions
- Consensus Development
- Study type
- Observational
- Recruitment status
- Unknown status
- Enrolment
- 150 participants (estimated)
- Lead sponsor
- University of Southern CaliforniaUniversity, hospital or other
- Study start
- 15 Jun 2022 (estimated)
- Primary completion
- 30 Jun 2022 (estimated)
- Study completion
- 15 Jul 2022 (estimated)
- First posted
- 19 May 2022 (actual)
- Results
- Not posted in the source record
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
The investigators aim to develop the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Artificial Intelligence Extension (PRISMA-AI) guideline as a stand-alone extension of the PRISMA statement, modified to reflect the particular requirements for the reporting of AI and its related topics (namely machine learning, deep learning, neuronal networking) in systematic reviews.
Detailed description
With advances in artificial intelligence (AI) over the last two decades, enthusiasm and adoption of this technology in medicine have steadily increased. Yet despite the greater adoption of AI in medicine, the way such methodologies and results are reported varies widely and the readability of clinical studies utilizing AI can be challenging to the general clinician.
Systematic reviews of AI applications are an important area for which specific guidance is needed. An ongoing systematic review led by our team has shown that the number of systematic reviews on AI applications (with or without meta-analysis) is increasing dramatically over the time, yet the quality of reporting is still poor and heterogeneous, leading to inconsistencies in the reporting of informational details among individual studies. Consequently, the lack of these informational details may front problems for primary research and synthesis and potentially limits their usefulness for stakeholders interested in implementing AI or using the information in systematic reviews.
The criteria will derive from the consensus among multi-specialty experts (in each medical specialty) who have already published about AI applications in leading medical journals and the lead authors of PRISMA, STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI to ensure that the criteria have global applicability in all the disciplines and for each type of study which involves the AI.
The proposed PRISMA-AI extension criteria focus on standardizing the reporting of methods and results for clinical studies utilizing AI. These criteria will reflect the most relevant technical details a data scientist requires for future reproducibility, yet they focus on the ability for the clinician reader to critically follow and ascertain the relevant outcomes of such studies.
The resultant PRISMA-AI extension will
- help stakeholders interested in implementing AI or using AI-related information in systematic reviews
- create a framework for reviewers that assess publications,
- provide a tool for training researchers on Artificial Intelligence SR methodology
- help end-users of the SR such as physicians and policymakers to better evaluate SR's validity and applicability in their decision-making process.
The success of the criteria will be seen in how manuscripts are written, how peer reviewers assess them, and finally, how the general readership is able to read and digest the published studies
Study design
- Observational model
- Cohort
- Time perspective
- Prospective
- Number of arms
- 1
| Arm | Type | Receives |
|---|---|---|
| Delphi Panel | – | Other: Delphi QuestionnaireA team of experts in the use AI technology in medicine together with experts in PRISMA, STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI will evaluate the PRISMA-AI extension reporting guidelines |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Not accepted
- Age groups
- Adult, Older adult
- Study population
- A team of experts in the use AI technology in medicine together with experts in PRISMA, STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI will evaluate the PRISMA-AI extension reporting guidelines.
Key inclusion criteria (3 listed)
- experts in the use AI technology in medicine
- experts in PRISMA
- leading authors of STARD-AI, CONSORT-AI, SPIRIT-AI, TRIPOD-AI, PROBAST-AI, CLAIM-AI and DECIDE-AI
Key exclusion criteria (1 listed)
- Panelists who were not able to commit to all rounds of the modified Delphi process will be excluded
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Delphi QuestionnaireAn invitation email, including a link to the survey, will be sent to the panel of experts in Ai in healthcare. The Delphi questionnaire will be administered via Welphi.com. In the first survey, panel members will outline the AI reporting standards in systematic reviews and objectively identify critical aspects of reporting methodology and results. In subsequent surveys, the expert panel will evaluate the modified criteria using a 1 to 5-point Likert scale with space provided for suggested edits and comments. Multiple rounds will be conducted until consensus is reached. After each round of Likert responses, the study team will calculate the agreement and distribution of responses. Likert responses will be dichotomized with positive values indicating agreement and neutral or negative values indicating disagreement. For the questions that do not reach a consensus of more than 80% in the first round or need further explanation, additional rounds of the survey may be performed. | Other | Delphi Panel |
Primary outcome
Degree of consensus
Time frame: 3 months
The level of agreement for all statements achieving consensus from the expert panel; consensus is predefined as ≥ 80% of the panel rating a given statement
Results and safety
Results are not available in the source record.
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 19 May 2022.
- Background: PubMed 33407780Ibrahim H, Liu X, Rivera SC, Moher D, Chan AW, Sydes MR, Calvert MJ, Denniston AK. Reporting guidelines for clinical trials of artificial intelligence interventions: the SPIRIT-AI and CONSORT-AI guidelines. Trials. 2021 Jan 6;22(1):11. doi: 10.1186/s13063-020-04951-6.
- Background: PubMed 33328049Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ; SPIRIT-AI and CONSORT-AI Working Group. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health. 2020 Oct;2(10):e549-e560. doi: 10.1016/S2589-7500(20)30219-3. Epub 2020 Sep 9.
- Background: PubMed 32514173Sounderajah V, Ashrafian H, Aggarwal R, De Fauw J, Denniston AK, Greaves F, Karthikesalingam A, King D, Liu X, Markar SR, McInnes MDF, Panch T, Pearson-Stuttard J, Ting DSW, Golub RM, Moher D, Bossuyt PM, Darzi A. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat Med. 2020 Jun;26(6):807-808. doi: 10.1038/s41591-020-0941-1. No abstract available.
- Background: PubMed 34244270Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, Logullo P, Beam AL, Peng L, Van Calster B, van Smeden M, Riley RD, Moons KG. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021 Jul 9;11(7):e048008. doi: 10.1136/bmjopen-2020-048008.
- Results publication: PubMed 33782057Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hrobjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021 Mar 29;372:n71. doi: 10.1136/bmj.n71.
- Individual participant dataSharing planned: No
Related trials for Consensus Development
RecruitingObservational study
Cardiothoracic Critical Care as Subspecialty and Its Core Competencies
Consensus Development
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Giovanni Cacciamani, MD study chair
University of Southern California
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
An observational study of Consensus Development, run by University of Southern California at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 3 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: status not confirmed, as of the sponsor's last update on 19 May 2022. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
Results are not available in the source record.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Degree of consensus (time frame: 3 months). It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT05382455, which also lists 5 linked publications.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 19 May 2022; AzTrials last refreshed its copy on 30 Oct 2026.