Validity of Topgrading
Q: Is Topgrading valid?
A: Of course. When I completed my Ph.D. in 1970, I had read every study on interviewing validity, and the results were pathetic — interviews did not predict job success. In preparation for my 1983 book, research continued to show interviews to be invalid; they still did not predict job success. And yet, I was sure I had created a magic bullet.1 I was sure that the Topgrading Interview could achieve 90‑95 percent "good hires." Beginning around the early 1980s, research began confirming that interviews not only can be valid, but done properly can be the most valid predictor of job performance. Research, for example, showed job analysis is essential. During World War II armies (literally) of psychologists took job analysis to stratospheric heights. At lower levels of organizations, job analysts, industrial engineers, and systems professionals joined with compensation specialists to promote efficiency and productivity. Years later job evaluation specialists such as Hay Associates, who became quite proficient at assigning pay "points" to job "grades," found themselves competing with McKinsey MBAs looking to reengineer processes. But, somehow, job analysis rarely crept into the most senior management positions. It is still a casual undertaking in most companies. Nonetheless, all current research supports the importance of pinning down exactly what the job is (through job analysis).
"State of the art" today is commonly a process consisting of a job analysis, behaviorally anchored competencies, and then some sort of semi-structured interview format, so that questions focus on what is important to do the job. Topgrading offers even better approaches, but in most companies assessment continues to consist solely of invalid, unstructured interviews. Unstructured interviews include one or more of the following characteristics: lack a question format, short (less than an hour), casual questioning ("Tell me about yourself"), unplanned (no job analysis, no job description, no written competencies), and no systematic analysis of data (the hire/no hire conclusion is made in minutes). In 1988 Weisner and Cronshaw2 found structured interviews over three times more valid than unstructured, in a review of 150 studies. Structured interviews are most valid when interviewers are trained, according to Pulakos et al.3 Psychological testing is less valid than structured interviews, according to Van Clieaf.4
A study by McDaniel et al.,5 analyzing 86,000 interviews by leading researchers in 1994 concluded, "Structured interviews were found to have higher interview validity than unstructured interviews." My graduate school colleague Frank Schmidt was an author of this study; he has done many meta-analytic studies of interviewing and tells me that the jury is definitely in: Interviews must be structured if they are to predict job performance. Perhaps you "wing it" in interviews because a tightly structured interview seems unbecoming, not as collegial as top‑level interviews should be. Trouble is, the "wing it" interviews can’t address all of the competencies, so 250 of such interviews might be necessary for valid conclusions about a single interviewee!
Let me cite two more studies. Pulakos and Schmitt6 found historical, experience‑based questions ("What were your accomplishments and failures in that job?") to be better predictors of job performance than hypothetical situational questions ("How would you restructure the finance department here?"). The Topgrading Interview approach does both, and why not? Campion, Campion and Hudson7 report a respectable validity coefficient (.50), with a 30‑item interview, half‑historical, half‑future questions. An interview consisting of 30 questions could take half an hour. Add half an hour to "sell" the candidate, and the interview would take an hour, which is the length of time scheduled for 90 percent of all management interviews. The one‑hour time frames exist because interviewers don’t know how to interview and because lawyers have frightened managers into avoiding so many questions. The Topgrading Interview approach, originated over 30 years ago and described meticulously in the two earlier versions of this book8 can easily ask 200 questions. In this book I argue that there are more than four dozen competencies necessary (not just desirable) in any management job, so asking a lot more than 30 questions is necessary.
The tandem interview has yet to be researched to any degree. Having discussed the tandem approach, my opinion is that a "solo" Topgrading Interviewer who is very experienced is apt to be more valid than a tandem of moderately experienced interviewers. Pulakos et. al. found multiple interviewers to reduce harmful effects of interviewer bias, but only if the interviewers did not share the same biases.
1 One book on interviewing influenced me early‑on: The Evaluation Interview, Fourth Edition, by R. A. Fear and R. J. Chiron, McGraw‑Hill, 1990. It’s still a classic, but seriously flawed. For example, whereas I might recommend a four‑hour interview, Fear says one and one‑half hours is too long, producing "a lot of unnecessary and irrelevant information" (p. 71). Furthermore, by discouraging asking about performance appraisals for every job (p. 107), Fear misses the most powerful lever for understanding negative – the TORC technique.
2 W. H. Wiesner and S. F. Cronshaw, "A Meta‑Analytic Investigation of the Impact of Interview Format and Degree of Structure on the Validity of the Employment Interview," Journal of Occupational Psychology 61(4) (1988), pp. 270‑290
3 E. D. Pulakos, N. Schmitt, D. Whitney, and M. Smith. "Individual Differences in Interviewer Ratings: The Impact of Standardization, Consensus Discussion, and Sampling Error on the Validity of a Structured Interview," Personnel Psychology 49(1) (1996), pp. 85‑102.
4 E. D. Pulakos, N. Schmitt, D. Whitney, and M. Smith. "Individual Differences in Interviewer Ratings: The Impact of Standardization, Consensus Discussion, and Sampling Error on the Validity of a Structured Interview," Personnel Psychology 49(1) (1996), pp. 85‑102.
5 E. D. Pulakos, N. Schmitt, D. Whitney, and M. Smith. "Individual Differences in Interviewer Ratings: The Impact of Standardization, Consensus Discussion, and Sampling Error on the Validity of a Structured Interview," Personnel Psychology 49(1) (1996), pp. 85‑102.
6 Pulakos, Schmitt, Whitney, and Smith, op cit.
7 M. A. Campion, J. A. Campion, and J. P. Hudson, "Structured Interviewing: A Note on incremental Validity and Alternative Question Types," Journal of Applied Psychology 79(6) (1984), pp. 998‑1002.
8 Bradford D. Smart, Selection Interviewing: A Management Psychologist’s Recommended Approach (John Wiley & Sons, 1983), and Bradford D. Smart, The Smart Interviewer.’ Tools and Techniques for Hiring the Best (John Wiley & Sons, 1989).