Bridging the data literacy gap for evidence-informed education policy and practice: the impact of visualization

Abstract

Data literacy comprises an important set of competencies in today’s society. Its rise in prominence can be traced to several developments: the exponential increase in data leading to unprecedented possibilities for transforming society; the global Open Data movement as a driving force in making data more accessible; and the evidence-informed policy movement. In the education sector, the latter is linked to the data-driven decision making movement, which refers to the use of data to inform education policy and practice at all levels. Because of these developments, data literacy is becoming embedded as an integral part of professional competencies for educators and education leaders.

The purpose of the study was twofold: first, to investigate whether data literacy can be measured on a single scale of increasing proficiency, and second, to investigate the effect of different data presentation formats on data literacy within the context of evidence-informed education policy and practice. A data literacy test was developed which required participants to answer multiple-choice questions based on a set of research briefs. Participants consisted mainly of graduate students enrolled in an education-related degree and education researchers. An experimental design was used in which the treatment condition was the presentation format of the research briefs. Test participants (N = 127) were randomly assigned to one of three presentation formats – text-only, text plus tabulated data, and text plus visualization – where tabulated data and visualizations were constructed from information in the text.

The findings from the test calibration supported the hypothesis of a hierarchical unidimensional data literacy scale. The interpretation of data literacy competencies along a log-linear scale replicated the hypothesized hierarchical development of data literacy levels.

It was also hypothesized that text plus visualization would lead to higher levels of data literacy compared to the other presentation formats. While previous research analysed differences in presentation formats through raw scores, this study used many-facet Rasch model analysis. Ordinal-level raw scores were transformed into linear, interval-level measures as an outcome of the interaction between three facets: person, item, and presentation format. In contrast to raw scores, Rasch model parameter estimates are sample independent, so the findings can be more objectively generalized beyond the sample and items used in the study. Rasch parameter estimates for the three presentation formats supported the hypothesis that the use of visualizations is associated with higher levels of data literacy.

Item-level analysis of the effect of presentation format, based on the theories of cognitive fit, cognitive load, and the proximity compatibility principle, suggested that data presentations which emphasize relationships between variables matching the problem context increase data literacy levels. Those that do not may lower data literacy levels by acting as extraneous cognitive load that diverts limited cognitive resources, especially if they misdirect attention and subsequent analysis.

Implications of these findings were discussed in terms of the conceptualization of a hierarchy of data literacy competencies vis-à-vis the requirements of educators and education leaders, the potential and caveats of using data presentations for communicating policy-relevant evidence, and future research on data presentation and visualization.