School of Languages and Linguistics - Research Publications
Permanent URI for this collection
Search Results
1 - 10 of 18
-
ItemThe challenges of providing expert advice in policy contexts(Association for Language Testing and Assessment of Australia and New Zealand (ALTAANZ-University of Melbourne), 2021-01-01)Language testers can have various roles in relation to the use of language tests and policy. One such role may be to provide expert advice in policy formation or policy review contexts. Such instances are often not documented systematically, as advice may be provided in informal or closed meetings, or confidential documents, which are not available to the public. The challenges associated with providing such advice are also rarely recorded. In this paper, I describe three instances in which the Language Testing Research Centre (LTRC) at the University of Melbourne was invited to provide external policy advice. I specifically reflect on what prompted the invitations, what advice we were asked to provide, the complexity of providing advice in each instance, and whether the advice was taken up by the policy makers. The paper reflects on some of the common threads and concludes with implications for training new language testers.
-
ItemTrust the test: Score-user perspectives on the roles of language tests in professional registration and skilled migration(Association for Language Testing and Assessment of Australia and New Zealand (ALTAANZ-University of Melbourne), 2021-01-01)Englishlanguage proficiency is a deciding factor in the life opportunities of many thousands of applicants for Australian skilled migration every year. This paper focuses on the perspectives of professional bodies that use English language tests in their decisions. Taking an interpretative approach, we explore the meanings that policy makers from these organisations ascribe (as score users) to test standards (cut-scores) so that we can better understand the uses of test scores in migration policy. The policy narratives we observed around the use of test scores describe the need to manage large numbers of applicants, to assure a level of English proficiency for high-risk professional communications, to provide an objective assessment that is separate from any assessment of professional competence and to maintain consistency of standards with other bodies. These views are contextualised with other relevant information, particularly that available from test providers, who are key players in the test-using interpretive community. We observe that particular tests and their standards become trusted and entrenched in policy, using the apparently simple semiotics of scores. Concomitantly, trust in tests is nurtured by test marketing. These tendencies warrant attention from test researchers, providers and score users.
-
ItemNo Preview AvailableRevisiting rating scale development for rater-mediated language performance assessments: Modelling construct and contextual choices made by scale developers(SAGE PUBLICATIONS LTD, 2021-10)Rating scale development in the field of language assessment is often considered in dichotomous ways: It is assumed to be guided either by expert intuition or by drawing on performance data. Even though quite a few authors have argued that rating scale development is rarely so easily classifiable, this dyadic view has dominated language testing research for over a decade. In this paper we refine the dominant model of rating scale development by drawing on a corpus of 36 studies identified in a systematic review. We present a model showing the different sources of scale construct in the corpus. In the discussion, we argue that rating scale designers, just like test developers more broadly, need to start by determining the purpose of the test, the relevant policies that guide test development and score use, and the intended score use when considering the design choices available to them. These include considering the impact of such sources on the generalizability of the scores, the precision of the post-test predictions that can be made about test takers’ future performances and scoring reliability. The most important contributions of the model are that it gives rating scale developers a framework to consider prior to starting scale development and validation activities.
-
ItemThe Effect of Sustained Teacher Feedback on CAF, Content and Organization in EFL Writing(Urmia University, 2020-06-01)Despite teachers' mainstream practices in L2 writing classrooms addressing different dimensions of writing over time, much of the research on feedback in recent years has been of relative short duration and has mainly focused on accuracy. The current longitudinal study investigated the influence of sustained teacher written feedback on accuracy, syntactic complexity, fluency, content, and organization in an EFL context. Ninety-two learners were divided into four groups, receiving written corrective feedback, feedback on content and organization, multilateral feedback (i.e., on grammatical accuracy, content, and organization), and no feedback over a 3½-month period. They completed a pre-test, post-test, and delayed post-test and wrote and revised eight interim expository compositions on a weekly basis. Results showed that the three treatment groups significantly improved in the dimensions on which they received feedback. However, only the groups who received feedback on content and organization improved in fluency. Importantly, the multilateral group improved in accuracy as well as fluency, content and organization. Theoretically, the findings endorse the language learning potentials of sustained writing as long as it is guided by teacher feedback. The findings provide empirical support for the influence of sustained feedback on expanding and consolidating learners’ explicit knowledge of L2 writing.
-
ItemThe impact of national standardized literacy and numeracy testing on children and teaching staff in remote Australian Indigenous communities(Sage Publications, 2019)All educational testing is intended to have consequences, which are assumed to be beneficial, but tests may also have unintended, negative consequences (Messick, 1989). The issue is particularly important in the case of large-scale standardized tests, such as Australia’s National Assessment Program - Literacy and Numeracy (NAPLAN), the intended benefits of which are increased accountability and improved educational outcomes. The NAPLAN purpose is comparable to that of other state and national ‘core skills’ testing programs, which evaluate cross-sections of populations in order to compare results between population sub-groupings. Such comparisons underpin ‘accountability’ in the era of population-level testing. This study investigates the impact of NAPLAN testing on one population grouping that is prominent in the NAPLAN results’ comparisons and public reporting: children in remote Indigenous communities. A series of interviews with principals and teachers documents informants’ first-hand experiences of the use and effects of NAPLAN in schools. In the views of most participants, the language and content of the test instruments, the nature of the test engagement, and the test washback have negative impacts on students and staff, with little benefit in terms of the usefulness of the test data. The primary issue is the fact that meaningful participation in the tests depends critically on proficiency in Standard Australian English (SAE) as a first language. This study contributes to the broader discussion of how reform-targeted standardized testing for national populations affects sub-groups who are not treated equitably by the test instrument or reporting for accountability purposes. It highlights a conflict between consequential validity and the notion of accountability that drives reform-targeted testing.
-
ItemCulturally and linguistically diverse populations in medical research: perceptions and experiences of older Italians, their families, ethics administrators and researchers(SAGE PUBLICATIONS INC, 2016)BACKGROUND: Low-participation of culturally and linguistically diverse (CALD) patients in medical research remains a problem in migrant and refugee destination countries such as Australia. The aims of this study were to explore i) CALD persons' perceptions and experiences of the medical system and medical research, in this case, older Italian Australians; and ii) the views of research professionals on CALD patient participation in medical research. DESIGN AND METHODS: A qualitative study was conducted in Melbourne, Australia, in 2015 utilising in-depth interviews and focus groups with four stakeholder groups: older Italian Australians (n=21); adult children of older Italian Australians (n=10); hospital Human Research Ethics Committee administrators (n=4); and clinical researchers (n=4). The data were analysed for content and thematic analysis. RESULTS: Themes for the CALD and family group were getting by in medical interactions; receptivity to medical research: testing the waters; and, receptivity to technology for support: passive versus active. Themes for the researcher and HREC groups about CALD patient participation in research were: exclusion; cultural factors; and e-consent. CONCLUSIONS: Our findings from four stakeholder perspectives and experiences confirm that there were considerable cultural, linguistic, and resourcing barriers hindering the participation of older Italian-Australians in medical research. Furthermore, our findings showed that in this study setting there were few enabling strategies in place to address these barriers despite the national ethics guidelines for equitable participation in research. The findings informed the creation of a multimedia tool whose purpose is to address and improve representation of CALD groups in clinical research. Significance for public healthMany people from culturally and linguistically diverse (CALD) backgrounds remain excluded from medical research such as clinical trials due to a range of language and cultural factors that can be amplified when this population is ageing. This exclusion has implications for the ability of CALD populations to benefit from participating in medical research and for applying research findings to CALD populations. It is essential to develop and implement strategies to include CALD communities in medical research and to uphold the ethical obligation of obtaining informed consent to research. The findings of this study have guided the development of a tablet-based resource which can be used in clinical and community contexts to raise awareness about the purpose of medical research. The resource has been carefully designed to be appropriate for participants' cultural background as well as their preferred language and literacy level. Such a resource has potential to address some of the cultural and linguistic barriers to clinical trial participation of CALD populations.
-
ItemAn evaluation of an online rater training program for the speaking and writing sub-tests of the Aptis test(ALTAANZ-ASSOC LANGUAGE TESTING & ASSESSMENT AUSTRALIA, 2016)Many large scale proficiency assessments that use human raters as part of their scoring procedures struggle with the realities of being able to offer regular face-to-face rater training workshops for new raters in different locations in the world. A number of these testing agencies have therefore introduced online rater training systems in order to access raters in a larger number of locations as well as from different contexts. Potential raters have more flexibility to complete the training in their own time and at their own pace. This paper describes the collaborative evaluation of a new online rater training module developed for a large scale international language assessment. The longitudinal evaluation focussed on two key points in the development process of the new program. The first, involving scrutiny of the online program, took place when the site was close to completion and the second, an empirical evaluation, followed the training of the first trial cohort of raters. The main purpose of this paper is to detail some of the complexities of completing such an evaluation within the operational demands of rolling out a new system and to comment on the advantages of the collaborative nature of such a project.
-
ItemLanguage test as boundary object: Perspectives from test users in the healthcare domain(SAGE PUBLICATIONS LTD, 2016-04)Objects that sit between intersecting social worlds, such as Language for Specific Purposes (LSP) tests, are boundary objects – dynamic, historically derived mechanisms which maintain coherence between worlds (Star & Griesemer, 1989). They emerge initially from sociopolitical mandates, such as the need to ensure a safe and efficient workforce or to control immigration, and they develop into standards (i.e. stabilized classifying mechanisms). In this article, we explore the concept of LSP test as boundary object through a qualitative case study of the Occupational English Test (OET), a test which assesses the English proficiency of healthcare professionals who wish to practise in English-speaking healthcare contexts. Stakeholders with different types of vested interest in the test were interviewed (practising doctors and nurses who have taken the test, management staff, professional board representatives) to capture multiple perspectives of both the test-taking experience and the relevance of the test to the workplace. The themes arising from the accumulated stakeholder perceptions depict a ‘boundary object’ that encompasses a work-readiness level of language proficiency on the one hand and aspects of communication skills for patient-centred care on the other. We argue that the boundary object metaphor is useful in that it represents a negotiation over the adequacy and effects of a test standard for all vested social worlds. Moreover, the test should benefit the worlds it interconnects, not just in terms of the impact on the learning opportunities it offers candidates, but also the impact such learning carries into key social sites, such as healthcare workplaces.
-
ItemDoes the writing of undergraduate ESL students develop after one year of study in an English-medium university?(ELSEVIER SCI LTD, 2014-07)
-
ItemNo Preview AvailableWritten corrective feedback studies: Approximate replication of Bitchener & Knoch (2010a) and Van Beuningen, De Jong & Kuiken (2012)(CAMBRIDGE UNIV PRESS, 2015-07)The question of whether written corrective feedback (CF) has a role to play in second language (L2) development has been controversial since Truscott (1996) published an article in Language Learning calling for the abandonment of the practice on theoretical, empirical and pedagogical grounds. As a result of his claims, an on-going debate about the efficacy of the practice has ensued while a number of dedicated researchers have focused their attention on empirically investigating whether learners benefit from the practice in terms of significantly improving their accuracy in subsequent texts over time. Responding not only to Truscott's doubts about the overall effectiveness of written CF for L2 learning and his specific doubts about whether certain types of provision could ever have a meaningful and enduring effect on acquisition, further questions about the potential impact of different variables were central to the thinking and research that followed his claims. Such variables include the linguistic focus of the feedback and the relative merits of targeting a limited number of error categories (focused feedback) rather than a more comprehensive range of error categories (unfocused feedback). While a growing body of research has begun investigating these issues over the last 15–20 years, the field is in need of replication studies before firm conclusions can be reached.