School of Languages and Linguistics - Theses
Permanent URI for this collection
Search Results
1 - 4 of 4
-
ItemApplying the context-adaptive model: evaluating a DEET funded English Language Program( 1995)Financially able governments around the world are embarking on major projects to retrain the growing numbers of unemployed. Education systems now dominated by 'market economy' -thinking government bodies holding the reigns on policy making and funding. It would appear from the writing of Bell and Goldstein (1995:21) that the situation in Australia is parallel to that of Canada. It is summarised in the words: "Many workers who have permanently lost their jobs in t:llls current economic recession have been advised to upgrade their educational credentials and obtain new work skills. In these changing economic times, upgrading, training and 'lifelong learning' are seen by many to be the key to finding and keeping a good job." This statement could easily be made about Australia, where the Federal Government is funding many types of training programs for the unemployed. The one being evaluated here is an English as Second Language (ESL) program funded by the Department of Employment Education and Training (DEET) for retrenched workers from the Textile Clothing and Footwear (TCF) industry. The program to be discussed is located at Victoria College, a registered private provider of education and training in Melbourne. Initially, the college offered English Language Intensive Courses for Overseas Students (ELICOS) accredited by an industry body, the National ELICOS Accreditation Scheme (NEAS). It has now broadened its scope to offer business and DEET funded courses. The evaluator has been closely connected to the program in the capacity of teacher, coordinator and (DEET) liaison officer. The first chapter presents the historical background of language program evaluations. Reports on outcomes from closely related areas, are presented next, as relevant background literature. The model chosen for the framework of this evaluation is the Context-Adaptive Model (CAM) (Lynch 1990). The second chapter leads to an evaluation design by adapting steps of the model to the evaluation context. It takes into consideration "such issues as the social and political basis and motivation for the language learning and teaching" (Lynch In press 94 13) which are important background to the evaluation. The data collection design is presented in the third chapter with the thematic framework for the evaluation. The design has quantitative and qualitative data collected for separate audience goals. The fourth chapter shows how qualitative and quantitative data is collected from various sources. The qualitative data consists of post-course questionnaires; case studies and interviews. Quantitative data consists of Australian Second Language Proficiency Rating (ASLPR) results in the form of precourse and post-course proficiency ratings for all the students and as well as a two-year charting of the four macro-skills for the case studies. In the fifth chapter, the results are discussed and arguments for the validation of the data and methods are put forward in the sixth chapter. The evaluation conclusions can then be drawn from the different perspectives presented in the last chapter.
-
ItemThe importance and effectiveness of moderation training on the reliability of teacher assessments of ESL writing samples( 1993)This thesis reports the findings of a study of the inter-rater reliability of assessment of ESL Writing by teachers in the Australian Adult Migrant Education Program, using the ASLPR, a language proficiency scale used throughout the program. The study investigates the individual ratings assigned to 15 writing samples by 83 teachers, both before and after training aimed at moderation of raters' perceptions of descriptors in the scale by reference to features of other 'anchor' writing samples. The thesis argues the necessity for on-going training of assessors of ESL writing, at a time of change in the program, from assessment of language proficiency to that of language competencies, since both forms of assessments are increasingly having consequences which affect the lives of the candidates. The importance and necessity for moderation training is established by reference to the problems of validity in the scale itself and in its use in the program, and by reference to the literature of assessor-training and features of writing which influence rater-judgements. The findings indicate that training is effective in substantially increasing inter-rater reliability of the subjects, by reducing the range of levels assigned to the samples and increasing the percentages of ratings at the mode (most accurate) level and at the Mode +/- 1 level (an allowance for 'error' due to the subjective nature of the assessment), after training. The paper concludes that on-going training is effective in achieving greater consensus i.e. inter-rater reliability amongst the assessors, but suggests that variability needs to be further reduced and offers suggestions for further research aimed at other assessors and variables.
-
ItemThe role of task-based conversation in the acquisition of Japanese grammar and vocabulary( 1999)This thesis investigates the role of conversation in the acquisition of Japanese grammar and vocabulary items in non-native speaker – native speaker interaction. In particular, it examines the effect of interactional moves of a native speaking conversation partner on the NNS’s control over Japanese grammar and vocabulary items. There are two studies: one cross-sectional and one longitudinal. A quasi-experiment (pre-test, treatment, post-test and delayed post-test) design was used for both. The subjects who participated in the study were beginning learners of Japanese at a tertiary institution. In the cross-sectional study, fifty-five subjects were randomly assigned to control and experimental groups. The subjects in the control group first took pre-tests and engaged in the free conversation with a native speaking conversation partner for 30 minutes. Soon after the conversation, they were asked to do post-tests. The subjects in the experimental group first took pre-tests and then performed three communication tasks (treatment) with a native speaking conversation partner. Post-tests were administered soon after the treatment (task-based conversation) and delayed post-tests were administered one week after the treatment. The effect of task-based conversation was examined by comparing the scores on the pre-, post- and delayed post-tests across groups and within groups. The frequency and types of native-speaking partners’ interactional moves were also investigated in relation to their effects on learners’ performance on post- and delayed post-tests. The results showed that overall task-based conversation had a positive impact on the short-term development of the targeted items. A variety of native-speaking partners’ interactional moves were observed. The effect of these moves on the development of the NNSs’ control over Japanese grammar and vocabulary items varied according to the types of the targeted items. In the longitudinal study, five subjects from the cross-sectional study participated for nine months. The five subjects met the same native-speaking conversation partner weekly for an hour and did a variety of communication tasks. Of the five subjects, three subjects received intensive recast treatment in addition to the weekly treatment after week eight. Four follow-up tests were administered during the twelve weeks and two delayed post-tests were administered one month, three month and six month after the final treatment. The results showed that the continuous treatment over twelve week had a positive impact on the development of target grammar and vocabulary items for both short- and long-terms. Intensive recast treatment had a positive impact on some structures, but had a negative impact on the structures which subjects were able to produce correctly prior to the treatment. The present study has implication for classroom teaching especially in the implementation of communication tasks and grammar and vocabulary teaching.
-
ItemAssessing the second language proficiency of health professionals( 1990)This thesis reports on the development of an Australian Government English as a Second Language test for health professionals, the Occupational English Test (OET) , and its validation using Rasch Item Response Theory models. The test contains sub-tests of the four macroskills, each based on workplace communication tasks. The thesis reports on the creation of test specifications, the trial ling of test materials and the analysis of data from full test sessions. The main research issues dealt with are as follows: 1. The nature of the constructs involved in communicative language testing. The term proficiency is analysed, and its relationship to a number of models of communicative competence examined. The difficulty of incorporating into these models factors underlying test performance is identified. 2. The nature of performance tests. A distinction is introduced between strong and weak senses of the term performance test, and related to the discussion in 1 above. 3. The content validity of the OET. This is established on the basis of a questionnaire survey, interviews, examination of relevant literature, workplace observation and test data. 4. The role of classical and Rasch IRT analysis in establishing the qualities of the test. Classical and Rasch IRT analyses are used to establish the basic reliability of the OET sub-tests. The Writing sub-test is shown to be somewhat problematic for raters because of the nature of the writing task involved. Analysis of data from the Reading subtest demonstrates the superiority of the Rasch analysis in the creation of short tests with a specific screening function. 5. The role of Rasch IRT analysis in investigating the construct and content validity of the test and hence of communicatively-oriented tests in general. Rasch analysis reveals that the sub-tests are satisfactory operationalizations of the constructs 'ESL listening/ speaking/ reading/ writing ability in health professional contexts. For the Speaking and Writing sub-tests, the analysis reveals that responses of raters in categories associated with perceptions of grammatical accuracy have a more important role in the determination of the candidate's total score than was anticipated in the design of the test. This finding has implications for the validity of communicatively oriented tests in general, and illustrates the potential of IRT analysis for the investigation of the construct validity of tests. 6. The appropriateness of the use of Rasch IRT in the analysis of language tests. The nature of the debate about 'unidimensionality' in Rasch analysis is reviewed. It is argued that the issue has been substantialy misunderstood. Data from the two parts of the Listening sub-test are analysed, and statistical tests are used to confirm the unidimensionality of the data set. It is concluded that Rasch analysis is appropriate for a language test of this type. 7. The behaviour of raters in the rating of oral and written production in a second language. The findings reported in 5 above suggest that the behaviour of raters is crucial to understanding what is being measured in a communicative test of the productive language skills. The research demonstrates the value of Rasch IRT analysis in the empirical validation of communicatively oriented language tests, and the potential of large-scale test development projects for theoretical work on language testing.