The critical role of subjectivity at the item level in a test of spoken English: variability in rater estimations
Loading...
Citations
Altmetric:
Altmetric::
Associated Records
Authors
File Licence
Editors
Date
2011
Source Title
Journal Title
Journal ISSN
Volume Title
Publisher
Melbourne Authors
Affiliation
Melbourne Graduate School of Education
Document Type
PhD thesis
Subject Code
Citation
KeshavarzMehr, N. (2011). The critical role of subjectivity at the item level in a test of spoken English: variability in rater estimations. PhD thesis, Melbourne Graduate School of Education, The University of Melbourne.
Access Status
Open Access
DOI
Access URL
Open Access URL
Open Access at PMC
Linked Resource URL
NHMRC Grant Code
ARC Grant Code
Description
© 2011 Dr. Negar KeshavarzMehr
Abstract
In assessing oral language proficiency, the scores given to interviewees may be influenced by not only their performance, but also non-linguistic factors such as the judgment of the interviewer or rater. Rater effect exists in most performance assessments. Such effects are considered as being among the most influential factors in rating a candidate’s oral proficiency. In contrast to previous studies, in which rater effects were analysed based on general measures of rater stringency, this study is an exploration of rater stringency across items and candidates.
A multi-faceted Rasch model was used to compare ratings obtained from rater scorings. The results show that rater-item and rater-candidate interactions lead to rater inconsistencies which are not detectable in overall stringency estimations and can undermine the reliability of rater scorings of candidate ability.
Significantly, the results challenge the general perception of rater stringency as an invariant property across items and persons. Evidence is presented to illustrate how this effect is manifested. Facet analysis assumes that the rater is harsh or lenient but this study has shown that it is neither the case over items nor over all ability groups. In fact, due to variations in rater scorings across items, and variations among raters within an item and among candidates, raters’ overall estimations cannot be considered as consistent estimations of their stringency. They therefore cannot be used in adjusting the candidates ability without considering the item difficulty and candidates’ ability.
The results have practical implications for quality control in test improvement, for rater training and for developing new methods of providing feedback which targets each rater with specific information on his or her source of bias.
Notes
Terms of Use
Keywords
speaking test, interview, rater, item, consistency, rater-item interaction, item resposnse theory