Melbourne Graduate School of Education - Theses

Permanent URI for this collection

Search Results

Now showing 1 - 3 of 3
  • Item
    Thumbnail Image
    The evolution of the OSCA-OSCE-Clinical Examination of the Royal Australasian College of Surgeons
    Serpell, Jonathan William ( 2010)
    The overall question this research aimed to address was how and why the Objective Structured Clinical Examination (OSCE) of The Royal Australasian College of Surgeons (RACS) evolved. A literature review and introduction are presented as a background to the evolution of the Objective Structured Clinical Assessment (OSCA)-OSCE-Clinical Examination of RACS. A brief history of surgery and training, an outline of the functions of RACS and a description of evolution from the apprenticeship model to formal surgical training programs is given. A background to the purpose of assessment within RACS, and formative and summative assessments precedes a description of the Part 1 Examination of RACS. By 1985 it was realised that not all objectives of basic surgical training of the RACS could be assessed in the Part 1 Examination using Multiple Choice Questions (MCQs); hence the introduction of an OSCE Clinical Examination, to assess clinical skills such as history taking, examination of patients, procedural skills and communication skills. A description of the Part 2 exit examination and the relation of RACS to universities and government are given. To undertake clinical examinations, clear definitions of clinical competence are required, and the differences between knowledge, the application of knowledge, competence and performance are considered and elucidated. These form the background to the clinical examination as a competency assessment, as opposed to a performance assessment in actual clinical practice. Then follows a detailed analysis of some important components of any examination process including: clear definition of the purpose of the assessment; blueprinting for type and content of assessment; reliability; validity; educational impact or consequential validity; cost; and feasibility and acceptability. Reliability of different clinical examination types is considered in detail, and an outline of definitions and the method of determining reliability described. Factors affecting reliability include: length of testing time; number of testing samples; number of examiners; standardised patient performance; and variation of examinees across testing stations (inter-case variability or content specificity). Validity is examined to ensure an examination is actually testing what it is intended to test. Face and content validity, alignment between the curriculum, the references and the examination, and consequential validity (the effect of the examination on learning) are highlighted as important validity components. Then follows an evaluation of rating scales for OSCE exams, using check lists or global assessments, assessor training, and methods to determine standards and pass marks for the examination. This includes relative and absolute standards, Angoff’s judgemental method, and the importance of examiner selection, standard setting meetings, and the determination of the standard. The overall question this research aimed to address was how and why the OSCE clinical examination of RACS evolved. To answer this, the mechanics of the RACS OSCE examination process were assessed. Twenty one problem areas were identified, analysed and evaluated, and the OSCE clinical examination was assessed against the known background literature on reliability, validity, educational impact, accessibility, cost, blueprinting, alignment of curriculum and resources and examinations, utility of a data base, standard setting, rating scales and global competency versus check list scores. Seven RACS-OSCE examinations were analysed in detail to elucidate the extent to which the RACS-OSCE matches the benchmark expectations in the areas outlined above. Some of the major problems identified with the original RACS-OSCE examination included: inappropriate inclusion of written questions; inability to rate overall or global performance as opposed to check list rating; lack of electronic data base questions and reliance on hardcopy exams; lack of statistical analysis of the examination; lack of consistent nomenclature; and lack of alignment of curriculum, resources and references and examination questions. It was also determined that: examiner recruitment and examination logistics required review; the role of the Clinical Committee which administers the OSCE, needed refining; reading lists needed updating; and the clinical examination needed to reflect the nine competencies of RACS recently introduced. These problems were addressed leading to changes in the practice and evaluation of the examination process by: introduction of competency scores for global assessment in the areas of counselling, procedure, examination and history taking; consistent clinical nomenclature was introduced; the 20 station (12 assessed, 8 written) examination was replaced with a 16 assessed station examination and the written questions were discontinued; the role of the administering Clinical Committee was defined in detail; the process of new question and station creation was clarified, including essential documentation for each station; recruitment and recognition of clinical examiners was instituted; the logistics of running the examination were refined; an electronic Clinical Committee data base was established; and statistical analysis of performance of the examination was undertaken. The overall reliability of the OSCE clinical examination of RACS in multiple examinations is of the order of 0.60-0.73, which is a modest level only. Removal of written questions and increasing observed clinical stations from 12 to 16 has not altered this reliability level. The most important factor affecting reliability is sample size; to deal with the major problem of content specificity or inter-case variability. This suggests an increased number of observed stations (perhaps up to 20) will be required to increase the reliability of the RACS Clinical OSCE. Differences in reliability and geographic centres have been demonstrated, suggesting that this is related to the examiners, which raises the issues of examiner performance and training. The content validity of the OSCE is good as evidenced by: the fact that surgical experts are creating, reviewing and revising the content of the OSCE exam; the use of blueprinting, and quality control by the Clinical Committee; and examination stations are statistically analysed for correlation and reliability. Evidence was found that assessment drives learning in the consequential validity analysis of the examination. The examination was found to have good face validity and authenticity. The OSCE was found to be feasible and acceptable. Standard setting still requires further development for the RACS-OSCE Clinical Examination and it is recommended a modified Angoff method be utilised. Overall, this thesis details the modification and evolution of the RACS-OSCE clinical examination over a sixteen year period, demonstrating it is robust, reliable, and valid.
  • Item
    Thumbnail Image
    Descriptive feedback: informing and improving practice in an assessment as learning context
    Christopher Damian, Dinneen ( 2010)
    This practitioner inquiry study explores the use of descriptive feedback as a means of ‘constructing the way forward’ (Tunstall & Gipps, 1996) for the learning of six students in a Year 2 classroom. The study was undertaken in a large independent school in a Victorian country town. A qualitative methodology was adopted for the study’s purpose of gaining insights into the interplay of factors that determined the students’ uptake of descriptive feedback. This included their responsiveness to ‘Thinkit-tickets’ — a self-assessment strategy developed by the researcher to promote reflective thinking, and provide evidence of the students’ affective and cognitive responses to their learning. The data collection methods involved: audio-taped semi-structured interviews; audio-taped in situ descriptive feedback conversations with students; non-participant observations conducted by an educational consultant to the school; a teacher journal and students’ written self-reflections (Thinkit-tickets). The study revealed that unambiguous immediate feedback, learning criteria and understanding the student’s individual perceptions of a given task (Muis, 2007) were key factors in determining the effectiveness of descriptive feedback. Data highlighted the connection between descriptive feedback and the importance of engendering a classroom culture based on dialogic teaching (Alexander, 2003), and reflective thinking (Ritchhart, 2002) for promoting a self-regulated, metacognitive approach to improve their learning. The study also identified the challenge of breaking down the unequal power relationship between the teacher and students in order to facilitate learning through the co-construction of next step actions on the way to achieving specified goals. In conclusion, the study reiterates the call for further empirical research on the use of descriptive feedback within formative assessment practices (Black & Wiliam, 1998, Brinko, 1993; Hattie & Timperley, 2007; Rodgers, 2006). Recommendations are subsequently made for closer investigations of engaging with dialogic teaching to generate more effective descriptive feedback practices that build learner agency.
  • Item
    Thumbnail Image
    Examining the alignment of the intended curriculum and performed curriculum in primary school mathematics and integrated curriculum
    ZIEBELL, NATASHA ( 2010)
    Curriculum alignment can be defined as the degree to which the intended curriculum (standards and teaching plans) and the performed curriculum (instruction and assessment) are in agreement with one another. Curriculum alignment research indicates that a coherent, or a well-aligned system has a positive effect on student achievement (English, 2000; Squires, 2009; Webb, 1997). This research focuses on Webb’s (1997) criteria for alignment of expectations and assessment, which provides a comprehensive framework that could be adapted for use with the current Victorian Essential Learning Standards curriculum. Webb’s (1997) criteria focus on alignment between curriculum and assessment. This research builds on Webb’s (1997) model by highlighting the importance of the inclusion of ‘instruction’ or the ‘performed curriculum’ in studies determining curricular alignment. The data for this comparative case study was collected from two Grade 3 classrooms. Data collection methods that were used were direct observation, interviews (pre and post observation), audio recording of lessons and document analysis. The data was analysed using an adaptation of six criteria for alignment focusing on ‘Content’ and ‘Pedagogical Implications’. The criteria focusing on ‘Content’ are Categorical Concurrence, Depth of Knowledge Consistency, Range of Knowledge Correspondence and Dispositional Consonance. The criteria focusing on ‘Pedagogical Implications’ are Effective Classroom Practices and Use of Technology, Materials and Tools. The findings of the study indicate that qualitative methods can be applied successfully in a study of curriculum alignment. This study found that data is readily available in the primary school setting which can be used with an adaptation of Webb’s (1997) criteria to determine the level of curriculum alignment. A key finding demonstrated that the process of planning occurs through a series of interpretations of the curriculum performed at various levels within the school, year level team or by the individual teacher. Throughout the process of planning and implementation of curriculum, the results showed that the teachers themselves customised the prescribed curriculum in response to their own priorities and the content that they felt reflected the needs of their students. It is a recommendation of this study that further research needs to be concerned with planning, pedagogical and assessment practices that effectively strengthen curriculum alignment. The benefits of further research would enable the identification of practices that improve alignment and could inform targeted, appropriate and effective professional development for practicing teachers.