"There is thus a need to improve the certification of raters, and to reduce the time between certification and clinical administration of ratings scales. There is also a need to permit certification outside of a clinical launch meeting environment.
"The Need for Improved Rater Reliability
"The need for standardized rater training has been described (Muller & Wetzel, 1998,
"Even modest gains in rater reliability can reduce result in substantial reduction in the sample size requirement, time, cost and risk of failure that can thwart development of promising therapeutic agents. For example, Perkins et al. (Biol. Psychiatry 47: 762-766 (2000)) calculated that an improvement in reliability from R=0.7 to R=0.9 could reduce sample size requirements by 22%. This may often translate into significant cost savings.
"In psychiatric trials (i.e., trials of therapy for a psychiatric disorder), where objective biological outcome measures may be lacking, reliability may be particularly poor, as investigators typically rely on rating scales completed by human clinicians. Seeking to quantify subjective experiences or behavior introduces substantial measurement error. Thus, the need for increased rater reliability is even greater in psychiatric trials.
"Problems of Ongoing Reliability
"Certification of raters under controlled conditions leaves room for error and abuse as well as simple incompetence during the actual conduct of study ratings. Many studies utilize raters operating under considerable time pressure. Experience indicates that rating scale scores are significantly correlated with the duration of the rating interview and that time allotted for the interview tends to decrease over the course of a study. Attempting to interview patients in a fixed time tends to lower the scores of symptomatic patients, reducing potential drug-placebo differences.
"Audio or video taping of interviews could effectively ameliorate this problem, but is a costly, time intensive, and intrusive methodology that requires an elaborate system of expert review, resolution of differences and remediation. Each tape must be reviewed in its entirety by an expert, or panel of experts, effectively doubling or tripling the amount of time required to obtain a particular rating. Moreover, this methodology is often unacceptable to patients and raters. Awareness of the tape recording may alter patient behavior (and the resulting ratings). For example the patient may feel more self-conscious about discussing sensitive or embarrassing topics while being recorded.
"There is thus a need for efficient monitoring of raters during the course of a trial in order to detect rater drift and variance so that remediation efforts and recertification may be instituted when necessary.
"Problems of Recertification
"Re-certification of raters refers to the process by which previously certified raters are reexamined to confirm that their ratings remain calibrated to study standards. This process aims to measure and reduce the tendency for raters to drift away from the rating norms established at study start-up. In theory, this is a relatively simple process that can be accomplished by having raters rate videotapes for which consensus or 'gold standard' ratings have been established.
Most Popular Stories
- Adam Levine Wins Big as 'The Voice' Crowns Champ
- Target Security Breach May Affect 40 Million Cardholders
- Texting With Vodka: Booze and Social Media Can Mix After All
- Tyson Foods Charged With Civil Rights Violation
- Archer Daniels Midland Moving HQ to Chicago
- How to Protect Yourself After Target Data Breach
- Mazda Leads the Pack for Fuel Efficiency
- 'Beyonce' Tops the U.S. Album Chart
- Bernanke Lets Congress Have It in Final Press Conference
- Wall Street Falls a Day After Surge