Development and Validation of a Diagnostic Rating Scale for Formative Assessment in a Thai EFL University Writing Classroom: A Mixed Methods Study

Abstract

Aimed at identifying learners’ strengths and weaknesses on specific skills or contents, diagnostic assessment can provide fine-grained information to formatively promote teaching, learning, and language development in an ongoing language classroom (Alderson, et al., 2015, Elder, 2017; Jang, 2012; Knoch & Macqueen, 2017; Lee, 2015). While much research has developed diagnostic tools for large-scale standardised assessment, few have constructed diagnostic instruments for low-stakes formative classroom assessment. To contribute to the existing knowledge of diagnostic language assessment (e.g., Alderson et al., 2015; Jang, 2012; Knoch, 2007, 2009a, 2009b, 2011; Lee, 2015), this PhD research aimed to (1) develop a diagnostic rating scale for a formative diagnostic assessment to diagnose students’ strengths and weaknesses in academic writing products and support ongoing teaching and learning in an EFL university classroom, and (2) explore the validity of the assessment claims following an argument-based approach to validation (Chapelle et al., 2008, 2010; Kane, 1992, 2006, 2011, 2012, 2013, 2016a, 2016b; Knoch & Chapelle, 2018). To this end, this research employed a multistage exploratory sequential mixed-methods design (Creswell & Plano Clark, 2018) to undertake the scale development and validation over three study stages: scale construction, scale trialling, and scale implementation. Following the line of a multisource-driven approach to scale development (e.g., Banerjee et al., 2015; Knoch, 2007, 2009b; Montee & Malone, 2014), the scale was constructed and revised on the basis of theories of L2 writing ability, existing scales, expert intuition, and classroom curriculum. The scale was operationally implemented over the course of one semester in four writing classrooms, in which 80 English-major undergraduates used the scale to write, self-diagnose, and revise their assignment essays, and five teachers applied the scale to diagnose the students’ essays and use diagnostic results to support teaching and learning. The teachers and twenty students were interviewed regarding their perceptions of the scale and assessment. The diagnostic scores were analysed using Classical Test Theory, Many-Facets Rasch, correlation, regression, and ANOVA statistics, and the perception protocols were analysed following a qualitative content analysis. Overall, findings offered reasonable support for the overarching validity argument for the scale-driven assessment system. Yet, the different writing tasks to which the scale was applied over the course of instruction made it difficult to reliably gauge student progress, highlighting the need for stronger evidence relating to the consequence inference. This limits the usefulness of a measurement-driven assessment approach in detecting learning progression over the course. In addition, the current validation framework, driven by Kane’s argument-based approach, appeared not to well capture the dynamic and varying evidentiary sources of learning and writing development in the classroom assessment. The present study provides implications for developing a diagnostic rating scale for diagnostic purposes in a formative assessment, and examining the validity of the assessment within the context of EFL language classroom.