Automated video difficulty assessment

Abstract

Automated assessment of text difficulty has been recognized as one method for assisting language teachers, textbook publishers, curriculum specialists, test developers, and researchers to make more informed decisions when selecting texts for use in instruction and assessment. While there is a substantial body of work on written and spoken texts, research on videotext difficulty is very scarce. Through a series of studies, the aim of this research program was twofold: to investigate what makes a videotext difficult for language learners and to develop automated measures to help predict difficulty in videotexts. Constructed to be used in this thesis, the Second Language Video Complexity (SLVC) corpus contains 320 academic lectures and 320 government advertisements which were annotated by 322 intermediate language learners.

In Study 1, the relative contribution of verbal complexity to videotext difficulty was examined. The results demonstrated that videotext difficulty was predicted by variation in pitch, lexical frequency and sophistication, and syntactic complexity. Study 2 sought to investigate the impact of visual complexity on learners’ perception of videotext difficulty. To this end, innovative computational measures to gauge visual complexity in videotexts were developed and integrated into the Automated Video Analysis (AUVANA) software. The findings of the study suggested that visual complexity contributes to videotext difficulty and their impact is on a par with that of verbal complexity. Moreover, the result of principal component analysis demonstrated that visual complexity is more likely a multifaceted and multidimensional construct, rather than a unitary construct.

While Study 1 and Study 2 looked at verbal and visual complexity independently, Study 3 focused on the integration of multimodal complexity features into ensemble machine learning models. The findings showed that ensemble multimodal models outperformed unimodal models in predicting difficulty in both video genres. Finally, Study 4 sought to develop an unsupervised approach for forecasting video segment difficulty in real-time. Through leveraging more advanced and sophisticated AI algorithms, several neural network models were trained to forecast difficulty in a corpus of 34,363 video segments. Quantitative and qualitative analyses showed that the trained model performed very well in forecasting difficulty in unseen video segments.

Taken together, this thesis makes clear contributions to the investigation of videotext difficulty assessment. In short, the findings of this thesis revealed the usefulness of automated measures for assessing and predicting videotext difficulty. Also, introduced and developed in this thesis new measures of videotext complexity and computational tools for analyzing, computing, and visualizing complexity in videotexts which may help researchers to perform fine-grain analysis of videotext complexity.