Melbourne Graduate School of Education - Theses

Permanent URI for this collection

Search Results

Now showing 1 - 10 of 12
  • Item
    Thumbnail Image
    An analysis of evaluative reasoning in education program evaluations conducted in Australia between 2014 – 2019
    Meldrum, Kathryn Janet ( 2022)
    The Australian government spends millions of dollars every year on grants that support new and innovative programs in the education sector. For example, in the 2020- 2021 Australian budget, financial support for interventions in the primary and secondary school sectors equalled more than $72.9 million dollars. Usually, and in order to account for spending the money, granting bodies ask for an evaluation of the intervention. One of the key activities of evaluation is to determine the value, merit or worth of a program. This is achieved by reaching an evaluative conclusion/judgement about the educational intervention that is credible, valid, and defensible to stakeholders. The defensibility of an evaluative conclusion/judgement relies partly on legitimate and justified arguments. In evaluation, legitimate arguments are made using the logic of evaluation. Justified arguments are made using evaluative reasoning. However, the reasoning process underpinning the logic is doubly important because readers need to be convinced of the credibility, validity, and defensibility of the evaluative conclusion/judgement. This study investigated the presence of a legitimate and justified evaluative conclusion/judgement in publicly available education evaluations conducted in Australia between 2014-2019. Using the systematic quantitative analysis method and new integrated logic of evaluation and evaluative reasoning conceptual framework, this study found that only four of the 26 evaluations analysed provided a legitimate and justified evaluative conclusion/ judgement about program value. The remaining 22 ‘evaluations’ were categorised as research because while they provided descriptive facts about the intervention, they did not ascribe value to it. The findings highlight the need for more credible, valid, and defensible evaluations of educational programs, achieved in part by using evaluative reasoning, as they provide an evidence-base for decision-making and for ensuring that quality education is available to all members of society.
  • Item
    Thumbnail Image
    Just reporting : does the school have a justifiable reporting process?
    Morrison, Caroline Mary ( 2006)
    This thesis titled `Just Reporting' aimed to explore the question: Does the school have a justifiable reporting process that meets the needs of key stakeholders (parents, students and teachers)? Through a critical review of the research literature relevant to reporting, various ethical issues were noted that assisted in the construction of the questions guiding this study. These ethical issues provided the lens through which I explored the reporting practices at the research school. The title Just Reporting emphasises the justice issues surrounding reporting as a communicative action where the integrity of each individual is maintained and relationships strengthened. The research took the form of a case study involving the participation of thirty-three parents, eleven teachers and twenty-one students from the one school setting in a questionnaire that had both quantitative and qualitative questions that gathered their affective and cognitive responses to the school's written report. I also held one focus group interview with parents to clarify information from the questionnaire. An interactive inquiry with mixed methods approach was chosen as the best way to answer the research questions. The aim was to develop a theory about reporting rather than prove an existing theory. This study examines what reporting is, the audience and purpose of reporting, and the imperatives of justifiable reporting. It gathered the opinions and beliefs about reporting at the research school from key stakeholder groups and sought to discover whether the written report met their needs and fulfilled the requirements of justifiable reporting. Final analysis of the data provided understandings about the nature of reporting at the research school and revealed a number of issues that prevented the process from being fully justifiable.
  • Item
    Thumbnail Image
    Improved student learning and leadership in self-managed schools
    Wee, Julie D ( 1998)
    This thesis examines improved student learning and leadership in schools under conditions of decentralisation, in a study designated the Learning Outcomes Project. In 1993, in the State of Victoria, Australia, a set of initiatives called Schools of the Future was introduced into government schools. The operation of the school system was changed from a relatively centralised system to one where much of the decision making about operations was decentralised to the school site. The impact on teachers, principals and schools has been considerable. Changes have occurred to teachers' working conditions and the accountability functions of schools through School Charters, Annual Reports and the introduction of Curriculum and Standards Frameworks (CSF) in eight Key Learning Areas (KLA). The Cooperative Research Project, a joint venture of the Victorian Secondary Principals' Association, the Victorian Primary Principals' Association, the Victorian Department of Education and The University of Melbourne, conducted a series of investigations into the processes and outcomes of Schools of the Future, including this study. The Learning Outcomes Project sought to investigate links from the reforms associated with decentralisation in school systems to improved student learning. This is a critical and vital area for research where little concrete evidence is available to support the benefits of decentralisation in relation to improved student learning. Previous studies have found the benefits of decentralisation to be mainly operational and managerial. A qualitative methodology was adopted to seek evidence of improved student learning in schools and explore leadership function under the conditions of decentralisation. Four schools that claimed improved student learning agreed to be part of the study. A conceptual framework was developed that allowed actions at school level be traced through the reforms to improved student learning. The leadership dimensions and strategies suited to schools under conditions of decentralisation were studied with the aid of a conceptual model, using a new model of educational leadership. Causal links from the reforms of decentralisation to improved student learning were mapped. This study reinforces the difficulties of making causal links. The findings indicate that improved student learning under conditions of decentralisation is evident in certain circumstances where clearly defined pathways are established from the reforms to improved student learning. It was shown that direct pathways to student learning were consistently and directly evident from Curriculum- CSF Implementation and Professional Development in all schools and were developing from Monitoring, Assessment and Reporting. The causal maps defined pathways which could be used by schools to monitor improved student learning in Schools of the Future. Leadership suited to decentralised systems was shown to require strong and balanced leadership across the six dimensional model used in this study. It was further shown that leadership dimensions that were not strong in the principal needed to be evident in other leaders for effective curriculum implementation to occur. Useful trends which may be relevant to other schools were found in this study. However, due to the favourable conditions in which this study was conducted across a small number of schools, further research is necessary to validate these findings and caution may be needed if using the findings in other settings.
  • Item
    Thumbnail Image
    An evaluation of the National Studies Program
    Parker, Judith M ( 1999)
    This study focuses on the National Studies Program, which is one of the activities Emergency Management Australia (EMA) and, more specifically, the Australian Emergency Management Institute (AEMI) has supported and undertaken. Since its inception the National Studies Program has consisted of workshops, seminars and study periods which explored multi-disciplinary aspects of emergency management. All these activities were designed to enhance prevention, preparedness, response or recovery capabilities or to further community understanding. The National Studies Program has evolved in an unsystematic manner. An evaluative review of how it has evolved, combined with the collection of information from involved individuals has provided a means of making the program more relevant, better presented with improved outputs and outcomes. In this study, data were collected and analysed from archival records and interviews. A needs assessment was undertaken to determine the gap between what is currently the case and what should be, in order to determine a set of recommendations for improving an existing program. This evaluation was undertaken by an internal evaluator so the study should be seen as a contribution to the insider for insider category of evaluation practice. It is therefore expected that because of the evaluator having a stake in the program the recommendations will be effective and be utilized.
  • Item
    Thumbnail Image
    Program evaluation of an alternative school
    Schier, Karl ( 2003)
    This program evaluation aims to identify the strengths and the areas that require improvement in an alternative school. The alternative school involved in this study was established to cater for 'at risk' students. The design of the study utilises both qualitative and quantitative methods. The qualitative data collection involved the use of standardised semi-structured interviews and focus group interviews. Questionnaires were the research instruments used for the collection of the quantitative data. Analysis of the results of this study indicated that the strengths of the alternative school related to the school culture/ethos, support for parents and students, relationship and connectedness and, student development and learning. The areas that require improvement involved the length of time for the students in the alternative setting, the transition from the alternative school back into mainstream, and the support for students after returning to mainstream education.
  • Item
    Thumbnail Image
    A nationwide educational knowledge and information system for China : is it likely to be adopted?
    Ping, Sun ( 2000)
    The researcher contrived a new strategy for the dissemination of educational knowledge -- a conceptual model of an Educational Knowledge and Information System (EKIS), based on a literature review. The focus of the research is to find out that whether, and under what conditions, the EKIS is likely to be adopted in China. The researcher interviewed some Chinese educational policy-makers to elicit their opinions on the EKIS model and on the possibility of the EMS development. The respondents perceived the EMS as a useful system, because it would have the potential to solve some of the current information problems. Their attitudes towards the EKIS development are positive, but with caution. The findings show consistency with the literature on innovation adoption. The importance of human factors in an innovation process is explicitly reflected in the findings. The respondents also indicated some elements that could hinder the adoption, development and utilisation of the EKIS in the setting of China, such as the influence of information policies, the impacts of educators' existing information dissemination and access conventions, and the lack of coordination among different government departments.
  • Item
    Thumbnail Image
    Setting standards: a comparison of two Rasch-Model methods
    Le, Thi Kim Anh ( 1998)
    Standard setting is an important issue in education. This study has applied two Rasch model techniques, Angoffs sum of probability and Andrich's pairwise comparison, to set cutoff scores for a Tasmanian Year 9 Link Test at year 3, year 5 and year 8 level. The main focus of the study was to examine methods of setting cutoff scores in the context of the National Profiles. The study also aimed at examining the relative difficulty of two parts of the Link Test on Mathematics for Year 9 students in Tasmania. Standard setting issues were analysed in the light of criterion referenced assessment in terms of their purposes and implications in education and measurement. Review of previous standard setting methods was also done. The two methods applied in the study were Angoff's sum of probabilities and Andrich's pairwise comparison. Both are popular in the educational context for establishing cutoff scores and examining the difficulty levels of tests. The findings produced by the sum of probability method have shown that cutoff score at each level of part 1 was higher than that of part 2, so that part 2 of the Link Test was more difficult than part 1. It was consistent with the item estimates produced by the pairwise comparison method. The correlations were high and significant at 0.05 level. It was consistent to those findings by Andrich and Titmanis (1997). However, the correlations between item difficulty estimates established by the pairwise comparison and empirical data were low from 0.39. The study has shown the possibility of using the two Rasch model techniques, sum of probability and pairwise comparison to establish cutoff score and examine the relative difficulty of the tests. Pairwise comparison method could be used as a mean of checking the validity of the proposed standards. It was shown that further research in this field should be done.
  • Item
    Thumbnail Image
    Initiating formal evaluation practices in Victorian secondary schools: a meta-evaluation of whole-school and part-school evaluation strategies
    Lambert, Faye Charlotte ( 1987)
    The purpose of this meta-evaluation was to investigate the merit of an apparent shift in evaluation policy on the part of the present government from whole-school evaluation with external validation and input to internal part-school evaluation as alternative strategies for initiating formal evaluation practices in Victorian secondary schools. While the study provides an overview of the strategies and outcomes pertaining to both approaches to evaluation, it focuses specifically on the implications of the scope of evaluation for the planning process in schools, the role and impact of the use of external expertise and the significance of staff perceptions on the process of evaluation and its outcomes. Data was collected using qualitative research methods and a retrospective study of eight carefully selected case study schools was carried out. Four of these schools had completed whole-school evaluations and the remaining four had completed part-school evaluations. While informal observation and document collection constituted an important part of the research strategy, heavy reliance was placed on data emerging from one-to-one interviews with individual members of staff across different levels of the school hierarchy. This methodology was adopted because it was believed to be the most effective way of discovering the more sensitive, less tangible outcomes related to evaluations, and because the attitudes and perceptions of staff towards evaluations represented an important outcome of the evaluation in their own right. A basic premise of this research is that the effectiveness of school-based evaluation initiatives in bringing about school improvement will be largely dependent upon the willing support of the staff who are called upon to participate in the evaluation and in any change initiatives which flow from it. While caution should be exercised in generalising from the findings of a limited number of case study schools to all schools, the findings support the general trend towards initiating formal evaluation practices via part-school evaluation strategy. However, they also underline the need for schools to initiate evaluation studies in ways which will ensure that they contribute effectively to, and become an integral part of, school development. In response to this need, an alternative model or approach to evaluation is proposed.
  • Item
    Thumbnail Image
    An examination of the sustainability of school-based program initiatives: the case of 'Turning the Tide' in schools
    Harvey, Graeme Lyle ( 2005)
    This thesis examines the factors that influence the sustained use of new program initiatives mandated in schools. Using the case of the `Turning the Tide in Schools' (TTIS) drug education program (implemented in Victorian schools, Australia, since 1997) the author has sought to address the specific question: What are the key factors at the systemic and school level that appear to support the sustained use of program initiatives in schools? In order to fulfill this aim, semi-structured interviews were employed to collect information from across the school system. This involved staff from secondary school sites; Regional Drug Education Facilitators (RDEFs); and Drug Education Unit Officers responsible for the Program's development. Seven schools were chosen for study. Six of these were identified as having been successful in implementing and embedding the TTIS. Then, in order to strengthen the design, a seventh and contrasting school was selected (i.e. where the process of establishing the program had been `less successful') for `negative case analysis'. After a general examination of these schools, four were chosen to be presented as case studies. The majority of analysis was undertaken using N5 (QSR NUD*IST) software. Findings indicated a range of factors that appeared to assist program sustainability. Success was shown to be based on three domains where external and program-based factors were mediated through a third group of school-based factors. Each of the three categories comprised two major elements. Consequently, there seemed to be six key aspects influencing embedded use. These were: high program credibility; a strong prevailing political climate; supportive program structures and processes; adequate personnel and resource provision; a strong prevailing school ethos; and a high level of school engagement. While the findings confirmed many of the factors related to program sustainability already outlined in the literature, this study has revealed the importance of the explicit use of school change theory. This was evident in schools where knowledge of implementing change was seen to be important. It also appeared to be crucial at the design stage of a program. As a result, models of the specific factors related to the sustainability of TTIS and of generic conditions necessary for the sustained used of any new school initiative are put forward. Suggested actions for strengthening the capacity to foster sustained use are also outlined.
  • Item
    Thumbnail Image
    Participatory evaluation: recording one school's experience in using evaluation as a process to support development
    Harvey, Graeme L. ( 1998)
    A participatory evaluation was conducted over a period of two terms to review an Early Intervention Program in an independent school. Using the framework developed by Cousins and Leithwood (1993) an evaluation was initiated with the author as the internal evaluator. Four 'primary users' of the program were recruited as members of the Evaluation Team and collaborated with the evaluator in the evaluation design and implementation. The negotiated focus for the evaluation was the adequacy of the program design and to support this outcome the Evaluability Assessment model (Smith, 1989b) was incorporated into the evaluation design. The evaluation was conducted over a 16 week period and after the conclusion of the project the four Evaluation Team members were interviewed using a semi-structured interview format. The data indicated that the evaluation was seen as effective in initiating change and as a process of professional and personal learning for those involved. From this information, some of the factors that were seen as assisting the effectiveness of the evaluation process and in supporting the utilisation of the information generated were identified. These were used to develop a broad framework of guidelines for the operation of participatory evaluation context within the school context.