A Study on Applicant Perceptions of Selection Procedures
FrontLine National Case Study
Back to Newsletter
Multiple-Choice Promotional Exam Case Study
Paper and pencil multiple-choice promotional exams, designed to measure knowledge of job pertinent reference material, are often a source of contention. Often referred to as "book tests", they are frequently challenged by candidates who feel they are either unfair or irrelevant to the work they perform every day. However, book referenced exams do not need to cause controversy. The below case study highlights how Ergometrics' approach to exam development can successfully increase perception of fairness and acceptance by candidates.
A fire department in a large city on the East Coast had used outside consultants for many years, per union contract, to develop and administer their promotional exams. Over the past couple of years, there had been increasing dissatisfaction with the written, multiple-choice examinations, both on the part of the candidates taking the test and the city human resource department. Candidates felt there were too many "trick questions" in which the correct answer was differentiated from the incorrect answers only by trivial wording changes. The human resources department was dissatisfied because more and more candidates were appealing their test results through the formal appeals process provided to them costing the city human resources a lot of staff time. Additionally, dissatisfied candidates frequently complained to their fire chief and union officials, who in turn sought out the city human resource department to discern the reasons for the high rate of candidate dissatisfaction. The resulting acrimony resulted in a general lack of confidence in the process by the candidates, the department and the city human resources.
Following a detailed job analysis, Ergometrics developed a book referenced exam designed to measure important aspects of the job and designed to be fair for all candidates. Ergometrics' approach ensured all items were directly linked to source material and the source material selected was relevant. This approach included a high level of participation from experienced fire personnel in choosing exam content. Ergometrics' detailed item writing and review techniques implemented by experienced item writers ensured the exam was technically sound.
Immediately following the administration of the examination, anecdotal evidence of candidate support of Ergometrics' exam was obvious. The City's Manager of Personnel Services received no immediate complaints from candidates, whereas in previous years he was swamped with complaints immediately following exam administration. Additionally, in previous years, approximately 100-200 appeals were filed, with nearly all of the 100 items receiving at least one protest (appeal) from one or more candidates. By comparison, Ergometrics' exam received only 35 total protests from 19 of the overall 116 candidates. This represents a 65-87% reduction in candidate appeals. In addition, the mandatory appeal hearing, which typically lasted 3 to 5 hours in previous years, lasted only 45 minutes. This represents a 75 to 85% reduction in staff time.