University Library
  • Login
A gateway to Melbourne's research publications
Minerva Access is the University's Institutional Repository. It aims to collect, preserve, and showcase the intellectual output of staff and students of the University of Melbourne for a global audience.
View Item 
  • Minerva Access
  • Medicine, Dentistry & Health Sciences
  • Medicine, Dentistry & Health Sciences Collected Works
  • Medicine, Dentistry & Health Sciences Collected Works - Research Publications
  • View Item
  • Minerva Access
  • Medicine, Dentistry & Health Sciences
  • Medicine, Dentistry & Health Sciences Collected Works
  • Medicine, Dentistry & Health Sciences Collected Works - Research Publications
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

    Can we trust the prediction model? Demonstrating the importance of external validation by investigating the COVID-19 Vulnerability (C-19) Index across an international network of observational healthcare datasets

    Thumbnail
    Download
    Published version (1.006Mb)

    Citations
    Altmetric
    Author
    Reps, J; Kim, C; Williams, R; Markus, A; Yang, C; Salles, TD; Falconer, T; Jonnagaddala, J; Williams, A; Fernández-Bertolín, S; ...
    Date
    2020-06-17
    Publisher
    Cold Spring Harbor Laboratory
    University of Melbourne Author/s
    Zhang, Lin
    Affiliation
    Medicine Dentistry & Health Sciences
    Metadata
    Show full item record
    Document Type
    Journal Article
    Citations
    Reps, J., Kim, C., Williams, R., Markus, A., Yang, C., Salles, T. D., Falconer, T., Jonnagaddala, J., Williams, A., Fernández-Bertolín, S., DuVall, S., Kostka, K., Rao, G., Shoaibi, A., Ostropolets, A., Spotnitz, M., Zhang, L., Casajust, P., Steyerberg, E. ,... Rijnbeek, P. (2020). Can we trust the prediction model? Demonstrating the importance of external validation by investigating the COVID-19 Vulnerability (C-19) Index across an international network of observational healthcare datasets. https://doi.org/10.1101/2020.06.15.20130328.
    Access Status
    Open Access
    URI
    http://hdl.handle.net/11343/253069
    DOI
    10.1101/2020.06.15.20130328
    Abstract
    <h4>Background</h4> SARS-CoV-2 is straining healthcare systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate between patients requiring hospitalization and those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision making during the pandemic. However, the model is at high risk of bias according to the Prediction model Risk Of Bias ASsessment Tool and has not been externally validated. <h4>Methods</h4> We followed the OHDSI framework for external validation to assess the reliability of the C-19 model. We evaluated the model on two different target populations: i) 41,381 patients that have SARS-CoV-2 at an outpatient or emergency room visit and ii) 9,429,285 patients that have influenza or related symptoms during an outpatient or emergency room visit, to predict their risk of hospitalization with pneumonia during the following 0 to 30 days. In total we validated the model across a network of 14 databases spanning the US, Europe, Australia and Asia. <h4>Findings</h4> The internal validation performance of the C-19 index was a c-statistic of 0.73 and calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data the model obtained c-statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US and South Korean datasets respectively. The calibration was poor with the model under-estimating risk. When validated on 12 datasets containing influenza patients across the OHDSI network the c-statistics ranged between 0.40-0.68. <h4>Interpretation</h4> The results show that the discriminative performance of the C-19 model is low for influenza cohorts, and even worse amongst COVID-19 patients in the US, Spain and South Korea. These results suggest that C-19 should not be used to aid decision making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.

    Export Reference in RIS Format     

    Endnote

    • Click on "Export Reference in RIS Format" and choose "open with... Endnote".

    Refworks

    • Click on "Export Reference in RIS Format". Login to Refworks, go to References => Import References


    Collections
    • Minerva Elements Records [45689]
    • Medicine, Dentistry & Health Sciences Collected Works - Research Publications [578]
    Minerva AccessDepositing Your Work (for University of Melbourne Staff and Students)NewsFAQs

    BrowseCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects
    My AccountLoginRegister
    StatisticsMost Popular ItemsStatistics by CountryMost Popular Authors