Restructured frame-of-reference training improves rating accuracy

Ming Hong Tsai, Serena Wee, Brandon Koh

Research output: Contribution to journalArticle

Abstract

The use of heuristic judgments is prevalent in organizations and negatively impacts accurate employee assessments. To minimize the negative impact of heuristic judgments (i.e., anchoring and adjustment), we aim to improve rating accuracy by restructuring frame-of-reference (FOR) training. We conducted five studies (N = 1,143) using different samples (three including participants with hiring experience), training environments (onsite and online), and rating contexts (evaluations of sales representatives, teachers, contract negotiation specialists, and retail store managers). Across the five studies, the average improvement in rating accuracy was at least twice as large for restructured FOR (vs. control) training as it was for typical FOR (vs. control) training; the difference in rating accuracy between restructured and typical FOR training was statistically significant. Furthermore, minimizing the anchoring effect rather than increasing opportunities for rating adjustments improved rating accuracy (Study 4). Finally, restructured FOR training achieved higher criterion validity (i.e., a higher strength of the association between ratings regarding a target and the target's objective performance) than did typical FOR training (Studies 3 and 5). We discuss implications for improving the effectiveness of diverse training programs and the accuracy of judgments in organizations.

Original languageEnglish
JournalJournal of Organizational Behavior
DOIs
Publication statusE-pub ahead of print - 10 Apr 2019

Fingerprint

Social Adjustment
rating
Organizations
Negotiating
Contracts
heuristics
Education
sales representative
hiring
Rating
Frame of reference
training program
restructuring
employee
manager
Heuristics
teacher
evaluation
performance
experience

Cite this

@article{dbdbbb6c373b4a239ca2795e9e3f134f,
title = "Restructured frame-of-reference training improves rating accuracy",
abstract = "The use of heuristic judgments is prevalent in organizations and negatively impacts accurate employee assessments. To minimize the negative impact of heuristic judgments (i.e., anchoring and adjustment), we aim to improve rating accuracy by restructuring frame-of-reference (FOR) training. We conducted five studies (N = 1,143) using different samples (three including participants with hiring experience), training environments (onsite and online), and rating contexts (evaluations of sales representatives, teachers, contract negotiation specialists, and retail store managers). Across the five studies, the average improvement in rating accuracy was at least twice as large for restructured FOR (vs. control) training as it was for typical FOR (vs. control) training; the difference in rating accuracy between restructured and typical FOR training was statistically significant. Furthermore, minimizing the anchoring effect rather than increasing opportunities for rating adjustments improved rating accuracy (Study 4). Finally, restructured FOR training achieved higher criterion validity (i.e., a higher strength of the association between ratings regarding a target and the target's objective performance) than did typical FOR training (Studies 3 and 5). We discuss implications for improving the effectiveness of diverse training programs and the accuracy of judgments in organizations.",
keywords = "anchoring and adjustment heuristic, frame-of-reference, judgment, rating accuracy, subjective evaluation",
author = "Tsai, {Ming Hong} and Serena Wee and Brandon Koh",
year = "2019",
month = "4",
day = "10",
doi = "10.1002/job.2368",
language = "English",
journal = "Journal of Organizational Behavior",
issn = "0894-3796",
publisher = "John Wiley & Sons",

}

Restructured frame-of-reference training improves rating accuracy. / Tsai, Ming Hong; Wee, Serena; Koh, Brandon.

In: Journal of Organizational Behavior, 10.04.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Restructured frame-of-reference training improves rating accuracy

AU - Tsai, Ming Hong

AU - Wee, Serena

AU - Koh, Brandon

PY - 2019/4/10

Y1 - 2019/4/10

N2 - The use of heuristic judgments is prevalent in organizations and negatively impacts accurate employee assessments. To minimize the negative impact of heuristic judgments (i.e., anchoring and adjustment), we aim to improve rating accuracy by restructuring frame-of-reference (FOR) training. We conducted five studies (N = 1,143) using different samples (three including participants with hiring experience), training environments (onsite and online), and rating contexts (evaluations of sales representatives, teachers, contract negotiation specialists, and retail store managers). Across the five studies, the average improvement in rating accuracy was at least twice as large for restructured FOR (vs. control) training as it was for typical FOR (vs. control) training; the difference in rating accuracy between restructured and typical FOR training was statistically significant. Furthermore, minimizing the anchoring effect rather than increasing opportunities for rating adjustments improved rating accuracy (Study 4). Finally, restructured FOR training achieved higher criterion validity (i.e., a higher strength of the association between ratings regarding a target and the target's objective performance) than did typical FOR training (Studies 3 and 5). We discuss implications for improving the effectiveness of diverse training programs and the accuracy of judgments in organizations.

AB - The use of heuristic judgments is prevalent in organizations and negatively impacts accurate employee assessments. To minimize the negative impact of heuristic judgments (i.e., anchoring and adjustment), we aim to improve rating accuracy by restructuring frame-of-reference (FOR) training. We conducted five studies (N = 1,143) using different samples (three including participants with hiring experience), training environments (onsite and online), and rating contexts (evaluations of sales representatives, teachers, contract negotiation specialists, and retail store managers). Across the five studies, the average improvement in rating accuracy was at least twice as large for restructured FOR (vs. control) training as it was for typical FOR (vs. control) training; the difference in rating accuracy between restructured and typical FOR training was statistically significant. Furthermore, minimizing the anchoring effect rather than increasing opportunities for rating adjustments improved rating accuracy (Study 4). Finally, restructured FOR training achieved higher criterion validity (i.e., a higher strength of the association between ratings regarding a target and the target's objective performance) than did typical FOR training (Studies 3 and 5). We discuss implications for improving the effectiveness of diverse training programs and the accuracy of judgments in organizations.

KW - anchoring and adjustment heuristic

KW - frame-of-reference

KW - judgment

KW - rating accuracy

KW - subjective evaluation

UR - http://www.scopus.com/inward/record.url?scp=85065159732&partnerID=8YFLogxK

U2 - 10.1002/job.2368

DO - 10.1002/job.2368

M3 - Article

JO - Journal of Organizational Behavior

JF - Journal of Organizational Behavior

SN - 0894-3796

ER -