Decision MakingNCT05244122
PROJECT 2 EXAMPLE: Feedback X Prevalence Using Dermatology Stimuli
A study testing Feedback and Prevalence in people with Decision Making, run by Brigham and Women's Hospital at 1 site in United States.
Trial summary
Imagine that a dermatologist spends the morning seeing patients who have been referred for suspicion of skin cancer. Many of them do, in fact, have skin lesions that require treatment. For this set of patients, disease 'prevalence' would be high. Read more about this study
- Who is it for?
- Ages 18 years and olderHealthy volunteers can take part
- Conditions the study is for
- Decision Making
- What is being tested?
- Feedback and Prevalence
- Type of treatment
- Lifestyle or therapy programme
- Could I receive a placebo?
- No, everyone receives an active treatment
- How many people
- 1,121 participants
- Where
- 1 siteUnited States
- Study dates
- 22 Jun 2021to 27 Jun 2021
- Phase
- Interventional
- Are there results?
- Yes, posted in the registrySummary in the doctors' view
Key requirements
Who can take part, as listed by the sponsor. Only the study team can confirm whether you are eligible, so take this list to your doctor.
- Ages 18 years and older
- All sexes
- Healthy volunteers accepted
- Adult, Older adult
Who may be able to join
- All welcome to enroll on line
Who cannot join
- Under 18 yrs
Trial locations and contacts
1 site in 1 country.
United States1 site
Visual Attention Lab, Brigham and Women's Hospital
Boston, Massachusetts, 02215
About this study
Official title: Prevalence Effects in Visual Search: Theoretical and Practical Implications
Imagine that a dermatologist spends the morning seeing patients who have been referred for suspicion of skin cancer. Many of them do, in fact, have skin lesions that require treatment. For this set of patients, disease 'prevalence' would be high. Suppose that the next task is to spend the afternoon giving annual screening exams to members of the general population. Here disease prevalence will be low. Would the morning's work influence decisions about patients in the afternoon? It is known from other contexts that recent history can influence current decisions and that target prevalence has an impact on decisions. In this study, decisions were decisions about skin lesions from individuals with varying degrees of expertise, using an online, medical imaging labelling app (DiagnosUs). This allowed examination of the effects of feedback history and prevalence in a single study. Blocks of trials could be of low or high prevalence, with or without feedback. Over 300,000 individual judgements were collected. (taken from Wolfe, J. M. (2022). How one block of trials influences the next: Persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. . Cognitive Research: Principles and Implications (CRPI), 7, 10. doi: https://doi.org/10.1186/s41235-022-00362-0
Read the detailed description
This description is based on a preregistration on the Open Science Framework site. Note that this is a "BESH" study. This type of research is not designed as a traditional clinical trial, but it is being reported here because of changes in NIH clinical trial reporting rules. This is one study from Project 2 of NE017001.
Levari et al (2018) found that people responded to a decrease in the prevalence of a stimulus by expanding their concept of it. Specifically, they asked observers to judge on each trial whether a dot, drawn from a blue-purple continuum, was blue or not. The results showed that observers were more likely to call ambiguous stimuli "blue" when blue items were less prevalent. In signal detection theory (SDT) terms, this is a liberal shift of response criterion. This is "prevalence induced concept change" (PICC). However, previous results obtained the opposite results in a long series of experiments on prevalence effects. The standard finding is that Os miss more targets at low prevalence. When blue is rare, they are less likely to call something blue. In SDT terms, this is a conservative criterion shift. This is the classic Low Prevalence Effect (LPE). In a round of earlier experiments, Lyu et al (2021) found that feedback is a critical variable. With trial-by-trial feedback, we get an LPE. With no feedback, the data usually show PICC results.
Do LPE and PICC effects show up when experts view stimuli in their expert domain? There is evidence for the LPE from search tasks (e.g. Evans, K. K., Birdwell, R. L., \& Wolfe, J. M. (2013). If You Don't Find It Often, You Often Don't Find It: Why Some Cancers Are Missed in Breast Cancer Screening. . PLoS ONE 8(5): e64366. , 8(5), e64366. doi: doi:10.1371/journal.pone.0064366). However, PICC evidence has not been collected and there is no data from single item decision tasks like the "Is this dot blue?" task. This is important because criterion shifts of the sort described above can have obvious health care implications.
This study will repeat the basic "Is this dot blue" experiment using dermatology stimuli (Is this melanoma or just a nevus (a mole)?)
Hypotheses:
(H1) without feedback, Os are more likely to label a spot as cancer when cancer prevalence is low (prevalence-induced-concept-change).
(H2) that with feedback, Os are less likely to label a spot as cancer when cancer prevalence is low (classic low prevalence effect)
Dependent variable
The main dependent variable is the proportion of cancer responses as a function of the cancer prevalence in the image set, but we will also record reaction times.
Conditions
How many and which conditions will participants be assigned to?
Four conditions will be run, between observers.
- 50% cancer images with feedback
- 50% cancer images without feedback
- 20% cancer images with feedback
- 20% cancer images without feedback
Observers will make a simple 2-alternative forced-choice (2AFC) cancer/no cancer decision.
Observers will be awarded points based on the correctness of the answer (more correct, more points)
There will be 200 trials in each block. That will produce 40 target present trials in the low prevalence conditions which should produce a hit rate that is not too coarse.
Stimuli will be images of moles from the ISIC archive. Each image comes with a known answer of either melanoma (cancer) or nevus (negative).
Analyses
The data will produce a continuum from not-cancer to cancer based on the observers responses in the 50% with feedback condition. This will give yield a psychometric function rising (it may be assumed) from near 0% cancer responses to near 100%.
Using that ordering, psychometric functions will be generated for the other three conditions.
To examine the effect of prevalence and the presence and absence of feedback on observers' response behavior, \run a logistic regression with prevalence and feedback as factors in a generalized mixed model will be run using jamovi software.
The data will also be used to compute the signal detection measures of sensitivity (d') and criterion (c) based on the actual truth about the images. That is, "cancer" responses will be coded as True positives if the images show cancer and as "false positives" if they do not. T-tests will be performed to examine whether d' and/or c (criterion) change significantly as a function of prevalence and feedback.
Outliers and Exclusions
N/A
Sample Size
Separate blocks of trials will be run and conditions will be compared with unpaired t-tests.
G* Power says suggests 36 observers PER GROUP or a total of 144 observers for alpha = 0.05, power = 0.80. The plan will be to attempt to run 45 Os per group, anticipating about 20% loss of Os due to the vagaries of online testing.
- Conditions
- Decision Making
- Being tested
- Feedback, Prevalence
- Keywords
- dermatology, signal detection theory, prevalence effects, base rate effects, online study, BESH
What participants receive
What participants receive. In a randomised study each person is assigned to one group by chance.
FeedbackBehavioural
presence or absence of trial by trial feedback
PrevalenceBehavioural
In some blocks, skin cancer "target" images were present on 50% of trials (high prevalence). In other blocks, disease prevalence was 20%.
| Study group | Role | What they receive |
|---|---|---|
| Feedback X Prevalence Using Dermatology Stimuli | Experimental | In this experiment, observers (Os) completed blocks of 80 trials. On each trial, they saw an image of a spot on the skin. They classified this as a melanoma (cancer) or a nevis (benign). Blocks could be of low prevalence (20% cancer cases, 16 images) or high prevalence (50%, 40 images). Os either did received trial by trial "Feedback" about their performance accuracy, or they did not. Thus, there were four types of block.
Low prevalence, No Feedback Low prevalence, Feedback High prevalence, No Feedback High prevalence, Feedback Each of these four types of block was made available to Os on each of 6 days. Os could elect to view each of the four blocks each day. Our particular interest was in the effect of performing one block on performance on an immediately subsequent block. Receives: Behavioral: Feedback; Behavioral: Prevalence |
What is being measured
The primary outcome is the main question the study is designed to answer.
Change in D' Between Pairs of Blocks.
D' (d-prime) is defined as z-transform of the true positive rate - z-transform of false positive rate. True positive is when you say that a real melanoma is a melanoma. False positive is when you say that a nevis is a melanoma.
A correction of 0.5 error is added to avoid calculation problems when z=0 or z=1.
D' of zero indicates no ability to discriminate. D' > zero indicates some ability to discriminate.
The change of interest is the D' for Block 2 when it follows Block 1 compared to the D' for Block 2 averaged across all conditions.
Measured: Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.
Change in Criterion Between Pairs of Blocks.
Criterion, c, corresponds to the position of the midpoint between the z-transformed probabilities of hits (correct yes responses) and false alarms (incorrect yes responses). It is calculated as -[z(p(h))+z(p(FA))]/2. The criterion, c, z-score quantifies the distance away from being unbiased in units of standard deviations. A Z-score of 0 is said to be unbiased. Negative values for c indicate a more relaxed criterion for saying yes. Positive numbers indicate a more strict criterion for saying yes.
Measured: Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.
Timeline
From the start of the study to its completion; the mark is today. Estimated dates are the sponsor’s plan.
- First posted17 Feb 2022
- Study start22 Jun 2021
- Primary completion27 Jun 2021
- Study completion27 Jun 2021
Resources
- Official record on ClinicalTrials.govThe registry entry this page is built from, with its full change history.
- Posted resultsParticipant flow, outcome measures and adverse events reported by the sponsor.
- Publications (1)Papers the sponsor has linked to this study.
- Other trials for Decision MakingStudies of the same condition, open and under way.
- More trials run by Brigham and Women's HospitalThe sponsor's other registered studies.
- What the trial phases meanHow a treatment moves from first tests in people to approval.
More details
For doctors and researchers: how the study is designed, who runs it, and its registry IDs.
Study design
- Study type
- Interventional
- Phase
- Phase N/A
- Primary purpose
- Basic science
- Allocation
- Not applicable
- Intervention model
- Single group
- Model description
- We tested \~800 observers on line in a study where they could participate in multiple conditions. We were looking for the effect of condition 1 on condition N+1
- Masking
- None (open label)
- Enrolment
- 1,121 participants (actual)
Sponsor and investigators
- Lead sponsor
- Brigham and Women's Hospital
- Sponsor type
- University, hospital or other
- Responsible party
- Principal investigator, Jeremy M Wolfe, PhD, Brigham and Women's Hospital
- Investigators
- Jeremy M Wolfe, PhD, Principal investigator, Brigham and Women's Hospital
IDs, dates and publications
- ClinicalTrials.gov ID
- NCT05244122
- Sponsor's protocol ID
- 2007P000646-A
- Other IDs
- R01EY017001
- Status last verified
- Oct 2022
- First submitted
- 8 Feb 2022
- Last updated
- 26 Oct 2022
- Results posted
- Yes, on ClinicalTrials.gov
- Data monitoring committee
- No
- FDA-regulated drug
- No
- FDA-regulated device
- No
- Individual data sharing
- Yes: IPD available on OSF. Other details available on request.
Publications
Wolfe JM. How one block of trials influences the next: persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. Cogn Res Princ Implic. 2022 Feb 2;7(1):10. doi: 10.1186/s41235-022-00362-0.
PubMed 35107667
Study at a glance
From the registry record, last updated by the sponsor on 26 Oct 2022. Recruitment status is as the sponsor reported it: confirm with the study team.
- Official title
- Prevalence Effects in Visual Search: Theoretical and Practical Implications
- ClinicalTrials.gov ID
- NCT05244122
- Acronym / sponsor ID
- 2007P000646-A
- Conditions
- Decision Making
- Keywords
- dermatology, signal detection theory, prevalence effects, base rate effects, online study, BESH
- Study type
- Interventional
- Recruitment status
- Completed
- Enrolment
- 1,121 participants (actual)
- Lead sponsor
- Brigham and Women's HospitalUniversity, hospital or other
- Study start
- 22 Jun 2021 (actual)
- Primary completion
- 27 Jun 2021 (actual)
- Study completion
- 27 Jun 2021 (actual)
- First posted
- 17 Feb 2022 (actual)
- Results
- Posted 26 Oct 2022
Clinical summary
The research question as the sponsor registered it. This describes what the study plans to test, not what it found.
Imagine that a dermatologist spends the morning seeing patients who have been referred for suspicion of skin cancer. Many of them do, in fact, have skin lesions that require treatment. For this set of patients, disease 'prevalence' would be high. Suppose that the next task is to spend the afternoon giving annual screening exams to members of the general population. Here disease prevalence will be low. Would the morning's work influence decisions about patients in the afternoon? It is known from other contexts that recent history can influence current decisions and that target prevalence has an impact on decisions. In this study, decisions were decisions about skin lesions from individuals with varying degrees of expertise, using an online, medical imaging labelling app (DiagnosUs). This allowed examination of the effects of feedback history and prevalence in a single study. Blocks of trials could be of low or high prevalence, with or without feedback. Over 300,000 individual judgements were collected. (taken from Wolfe, J. M. (2022). How one block of trials influences the next: Persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. . Cognitive Research: Principles and Implications (CRPI), 7, 10. doi: https://doi.org/10.1186/s41235-022-00362-0
Detailed description
This description is based on a preregistration on the Open Science Framework site. Note that this is a "BESH" study. This type of research is not designed as a traditional clinical trial, but it is being reported here because of changes in NIH clinical trial reporting rules. This is one study from Project 2 of NE017001.
Levari et al (2018) found that people responded to a decrease in the prevalence of a stimulus by expanding their concept of it. Specifically, they asked observers to judge on each trial whether a dot, drawn from a blue-purple continuum, was blue or not. The results showed that observers were more likely to call ambiguous stimuli "blue" when blue items were less prevalent. In signal detection theory (SDT) terms, this is a liberal shift of response criterion. This is "prevalence induced concept change" (PICC). However, previous results obtained the opposite results in a long series of experiments on prevalence effects. The standard finding is that Os miss more targets at low prevalence. When blue is rare, they are less likely to call something blue. In SDT terms, this is a conservative criterion shift. This is the classic Low Prevalence Effect (LPE). In a round of earlier experiments, Lyu et al (2021) found that feedback is a critical variable. With trial-by-trial feedback, we get an LPE. With no feedback, the data usually show PICC results.
Do LPE and PICC effects show up when experts view stimuli in their expert domain? There is evidence for the LPE from search tasks (e.g. Evans, K. K., Birdwell, R. L., \& Wolfe, J. M. (2013). If You Don't Find It Often, You Often Don't Find It: Why Some Cancers Are Missed in Breast Cancer Screening. . PLoS ONE 8(5): e64366. , 8(5), e64366. doi: doi:10.1371/journal.pone.0064366). However, PICC evidence has not been collected and there is no data from single item decision tasks like the "Is this dot blue?" task. This is important because criterion shifts of the sort described above can have obvious health care implications.
This study will repeat the basic "Is this dot blue" experiment using dermatology stimuli (Is this melanoma or just a nevus (a mole)?)
Hypotheses:
(H1) without feedback, Os are more likely to label a spot as cancer when cancer prevalence is low (prevalence-induced-concept-change).
(H2) that with feedback, Os are less likely to label a spot as cancer when cancer prevalence is low (classic low prevalence effect)
Dependent variable
The main dependent variable is the proportion of cancer responses as a function of the cancer prevalence in the image set, but we will also record reaction times.
Conditions
How many and which conditions will participants be assigned to?
Four conditions will be run, between observers.
- 50% cancer images with feedback
- 50% cancer images without feedback
- 20% cancer images with feedback
- 20% cancer images without feedback
Observers will make a simple 2-alternative forced-choice (2AFC) cancer/no cancer decision.
Observers will be awarded points based on the correctness of the answer (more correct, more points)
There will be 200 trials in each block. That will produce 40 target present trials in the low prevalence conditions which should produce a hit rate that is not too coarse.
Stimuli will be images of moles from the ISIC archive. Each image comes with a known answer of either melanoma (cancer) or nevus (negative).
Analyses
The data will produce a continuum from not-cancer to cancer based on the observers responses in the 50% with feedback condition. This will give yield a psychometric function rising (it may be assumed) from near 0% cancer responses to near 100%.
Using that ordering, psychometric functions will be generated for the other three conditions.
To examine the effect of prevalence and the presence and absence of feedback on observers' response behavior, \run a logistic regression with prevalence and feedback as factors in a generalized mixed model will be run using jamovi software.
The data will also be used to compute the signal detection measures of sensitivity (d') and criterion (c) based on the actual truth about the images. That is, "cancer" responses will be coded as True positives if the images show cancer and as "false positives" if they do not. T-tests will be performed to examine whether d' and/or c (criterion) change significantly as a function of prevalence and feedback.
Outliers and Exclusions
N/A
Sample Size
Separate blocks of trials will be run and conditions will be compared with unpaired t-tests.
G* Power says suggests 36 observers PER GROUP or a total of 144 observers for alpha = 0.05, power = 0.80. The plan will be to attempt to run 45 Os per group, anticipating about 20% loss of Os due to the vagaries of online testing.
Study design
- Primary purpose
- Basic science
- Allocation
- Not applicable
- Intervention model
- Single group
- Masking
- None (open label)
- Number of arms
- 1
- Model description
- We tested ~800 observers on line in a study where they could participate in multiple conditions. We were looking for the effect of condition 1 on condition N+1
| Arm | Type | Receives |
|---|---|---|
| Feedback X Prevalence Using Dermatology Stimuli | Experimental | Behavioral: Feedback, Behavioral: PrevalenceIn this experiment, observers (Os) completed blocks of 80 trials. On each trial, they saw an image of a spot on the skin. They classified this as a melanoma (cancer) or a nevis (benign). Blocks could be of low prevalence (20% cancer cases, 16 images) or high prevalence (50%, 40 images). Os either did received trial by trial "Feedback" about their performance accuracy, or they did not. Thus, there were four types of block. Low prevalence, No Feedback Low prevalence, Feedback High prevalence, No Feedback High prevalence, Feedback Each of these four types of block was made available to Os on each of 6 days. Os could elect to view each of the four blocks each day. Our particular interest was in the effect of performing one block on performance on an immediately subsequent block. |
Who may qualify
- Ages
- 18 years and older
- Sex
- All
- Healthy volunteers
- Accepted
- Age groups
- Adult, Older adult
Key inclusion criteria (1 listed)
- All welcome to enroll on line
Key exclusion criteria (1 listed)
- Under 18 yrs
Eligibility is decided by the study team after screening; this overview does not establish that a patient qualifies. Full inclusion and exclusion criteria.
Interventions
| Intervention | Type | Arms |
|---|---|---|
| Feedbackpresence or absence of trial by trial feedback | Behavioural | Feedback X Prevalence Using Dermatology Stimuli |
| PrevalenceIn some blocks, skin cancer "target" images were present on 50% of trials (high prevalence). In other blocks, disease prevalence was 20%. | Behavioural | Feedback X Prevalence Using Dermatology Stimuli |
Primary outcome
Change in D' Between Pairs of Blocks.
Time frame: Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.
D' (d-prime) is defined as z-transform of the true positive rate - z-transform of false positive rate. True positive is when you say that a real melanoma is a melanoma. False positive is when you say that a nevis is a melanoma. A correction of 0.5 error is added to avoid calculation problems when z=0 or z=1. D' of zero indicates no ability to discriminate. D' > zero indicates some ability to discriminate. The change of interest is the D' for Block 2 when it follows Block 1 compared to the D' for Block 2 averaged across all conditions.
Change in Criterion Between Pairs of Blocks.
Time frame: Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.
Criterion, c, corresponds to the position of the midpoint between the z-transformed probabilities of hits (correct yes responses) and false alarms (incorrect yes responses). It is calculated as -[z(p(h))+z(p(FA))]/2. The criterion, c, z-score quantifies the distance away from being unbiased in units of standard deviations. A Z-score of 0 is said to be unbiased. Negative values for c indicate a more relaxed criterion for saying yes. Positive numbers indicate a more strict criterion for saying yes.
Results and safety
Loading the posted results from ClinicalTrials.gov…
Research links
- Official record on ClinicalTrials.govLast updated by the sponsor 26 Oct 2022.
- Posted resultsParticipant flow, baseline, all outcome measures and adverse events.
- Results publication: PubMed 35107667Wolfe JM. How one block of trials influences the next: persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study. Cogn Res Princ Implic. 2022 Feb 2;7(1):10. doi: 10.1186/s41235-022-00362-0.
- Individual participant dataSharing planned: Yes. IPD available on OSF. Other details available on request.
Related trials for Decision Making
RecruitingEarly Phase 1
Developing Transcranial Neuromodulation Protocols for Learning and Decision-Making
Normal Physiology
Recruiting
Patient Priorities Aligned Discrete Decision Making for Older Adults With Multiple Chronic Conditions.
Hernia
RecruitingObservational study
Olfactory Function and Model-Based Behavior in People Living With HIV and SUD
HIV, Substance Use Disorder
Active, not recruiting
Developing and Testing an Online Pathway From Screening to Treatment for Depression in Oncology: iPath*D (Aim 2)
Cancer, Depression
Not yet recruiting
A Multidimensional Intervention to Improve Ethical Decision-Making Competence Among Nursing Student Interns: A Randomized Controlled Trial
Ethical Decision-Making Competence in Nursing Student Interns
Referral and contact
AzTrials does not enrol participants or take referrals. Contact the study team or a site listed below; they decide who can take part. This study is not recruiting at the moment.
Jeremy M Wolfe, PhD principal investigator
Brigham and Women's Hospital
1 site in 1 country: see every site and its status.
Questions about this trial
What is this clinical trial studying?
A study testing Feedback and Prevalence in people with Decision Making, run by Brigham and Women's Hospital at 1 site in United States.
The full registered description is in the About this study section and the official record.
Who may be eligible to take part?
The sponsor lists 1 inclusion and 1 exclusion criteria. Eligibility is decided by the study team after screening, so read the full criteria and contact a listed site.
Is this clinical trial currently recruiting?
The record says: finished, as of the sponsor's last update on 26 Oct 2022. Status can change between updates, so confirm with the study contact before making plans.
Where is the study taking place?
The record lists 1 site in United States. Whether a site is taking part right now should be confirmed with that site. See every site.
What are the possible benefits and risks?
They depend on the study and the treatment, and no study can promise a benefit. The study team explains the known risks during informed consent, before anyone joins; the record itself does not list them.
Are results available for this study?
Yes. Results are posted in the registry (first posted 26 Oct 2022). A summary is in the For doctors view, and the full results are on ClinicalTrials.gov.
How can a doctor refer a patient or contact the research team?
The record lists no contact. The official record names the sponsor, who can be asked directly. AzTrials does not enrol participants or take referrals.
What does the primary outcome measure?
Change in D' Between Pairs of Blocks. (time frame: Participants could be in the study for as little as two blocks in one day up to 24 blocks collected over 6 days.). The study lists 2 primary outcomes in all. It is what the study was designed to measure, not a finding.
Where can I read the original study information?
On the ClinicalTrials.gov record for NCT05244122, which also lists 1 linked publication.
Does a listing mean the treatment is approved or proven to work?
No. A listing means the study is registered. It does not show that the treatment is safe, works, or is approved by a regulator.
How often is this information updated?
AzTrials copies changed records from ClinicalTrials.gov every night and reloads the whole registry every week. The sponsor last updated this record on 26 Oct 2022.