A machine learning-based phenotype for long COVID in children: an EHR-based study from the RECOVER program Authors
PhD Vitaly Lorman, MPH Hanieh Razzaghi, PhD Xing Song, MD, MBA Keith Morse, MD Levon Utidjian, MS Andrea J Allen, MBBS Suchitra Rao, MD, MPH Colin Rogerson, MD Tellen D Bennett, PhD Hiroki Morizono, MLIS Daniel Eckrich, MD Ravi Jhaveri, PhD, MBA Yungui Huang, MPH, MBA Daksha Ranade, MD, MS Nathan Pajor, MD, MPH Grace M Lee, MD, PhD Christopher B Forrest, MD, PhD L Charles Bailey
doi:10.1101/2022.12.22.22283791
Background As clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and hence the clinical definition evolves, it is desirable to have a method to reliably identify patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems data.
Methods and Findings In this study, we developed and validated a machine learning algorithm to classify which patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children (MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS-CoV-2 test results in pediatric health systems within the PEDSnet EHR network. Patient features included in the model were selected from conditions, procedures, performance of diagnostic testing, and medications using a tree-based scan statistic approach. We used an XGboost model, with hyperparameters selected through cross-validated grid search, and model performance was assessed using 5-fold cross-validation. Model predictions and feature importance were evaluated using Shapley Additive exPlanation (SHAP) values.
Conclusions The model provides a tool for identifying patients with PASC and an approach to characterizing PASC using diagnosis, medication, laboratory, and procedure features in health systems data. Using appropriate threshold settings, the model can be used to identify PASC patients in health systems data at higher precision for inclusion in studies or at higher recall in screening for clinical trials, especially in settings where PASC diagnosis codes are used less frequently or less reliably. Analysis of how specific features contribute to the classification process may assist in gaining a better understanding of features that are associated with PASC diagnoses.
References
Algarni, Alamri, Khayat, Alabdali, Alsubhi et al., Clinical practice guidelines in multisystem inflammatory syndrome (MIS-C) related to COVID-19: a critical review and recommendations, World J Pediatr,
doi:10.1007/s12519-021-00499-wBorch, Holm, Knudsen, Ellermann-Eriksen, Hagstroem, Long COVID symptoms and duration in SARS-CoV-2 positive children -a nationwide cohort study, Eur J Pediatr,
doi:10.1007/s00431-021-04345-zLundberg, Lee, A Unified Approach to Interpreting Model Predictions
Mahmoud, El-Kalliny, Kotby, El-Ganzoury, Fouda et al., Treatment of MIS-C in Children and Adolescents, Curr Pediatr Rep,
doi:10.1007/s40124-021-00259-4Pellegrino, Chiappini, Licari, Galli, Marseglia, Prevalence and clinical presentation of long COVID in children: a systematic review, Eur J Pediatr,
doi:10.1007/s00431-022-04600-xPfaff, Girvin, Bennett, Identifying who has long COVID in the USA: a machine learning approach using N3C data, Lancet Digit Health,
doi:10.1016/S2589-7500(22)00048-6Ramakrishnan, Kashour, Hamid, Halwani, Tleyjeh, Unraveling the Mystery Surrounding Post-Acute Sequelae of COVID-19, Front Immunol,
doi:10.3389/fimmu.2021.686029Rao, Lee, Razzaghi, Clinical features and burden of post-acute sequelae of SARS-CoV-2 infection in children and adolescents: an exploratory EHR-based cohort study from the RECOVER program, MedRxiv Prepr Serv Health Sci. Published online,
doi:10.1101/2022.05.24.22275544Reese, Blau, Bergquist, Generalizable Long COVID Subtypes: Findings from the NIH N3C and RECOVER Programs, MedRxiv Prepr Serv Health Sci. Published online,
doi:10.1101/2022.05.24.22275398Saito, Rehmsmeier, The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets, PLoS ONE,
doi:10.1371/journal.pone.0118432Thallapureddy, Thallapureddy, Zerda, Long-Term Complications of COVID-19 Infection in Adolescents and Children, Curr Pediatr Rep,
doi:10.1007/s40124-021-00260-xWang, Maro, Baro, Data Mining for Adverse Drug Events With a Propensity Scorematched Tree-based Scan Statistic, Epidemiol Camb Mass,
doi:10.1097/EDE.0000000000000907Yang, Varghese, Stephenson, Tu, Gronsbell, Machine learning approaches for electronic health records phenotyping: A methodical review, Published online,
doi:10.1101/2022.04.23.22274218{ 'institution': [{'name': 'medRxiv'}],
'indexed': { 'date-parts': [[2022, 12, 29]],
'date-time': '2022-12-29T05:55:56Z',
'timestamp': 1672293356631},
'posted': {'date-parts': [[2022, 12, 26]]},
'group-title': 'Epidemiology',
'reference-count': 21,
'publisher': 'Cold Spring Harbor Laboratory',
'content-domain': {'domain': [], 'crossmark-restriction': False},
'accepted': {'date-parts': [[2022, 12, 26]]},
'abstract': '<jats:title>Abstract</jats:title><jats:sec><jats:title>Background</jats:title><jats:p>As '
'clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and '
'hence the clinical definition evolves, it is desirable to have a method to reliably identify '
'patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems '
'data.</jats:p></jats:sec><jats:sec><jats:title>Methods and Findings</jats:title><jats:p>In '
'this study, we developed and validated a machine learning algorithm to classify which '
'patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children '
'(MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS-CoV-2 test '
'results in pediatric health systems within the PEDSnet EHR network. Patient features included '
'in the model were selected from conditions, procedures, performance of diagnostic testing, '
'and medications using a tree-based scan statistic approach. We used an XGboost model, with '
'hyperparameters selected through cross-validated grid search, and model performance was '
'assessed using 5-fold cross-validation. Model predictions and feature importance were '
'evaluated using Shapley Additive exPlanation (SHAP) '
'values.</jats:p></jats:sec><jats:sec><jats:title>Conclusions</jats:title><jats:p>The model '
'provides a tool for identifying patients with PASC and an approach to characterizing PASC '
'using diagnosis, medication, laboratory, and procedure features in health systems data. Using '
'appropriate threshold settings, the model can be used to identify PASC patients in health '
'systems data at higher precision for inclusion in studies or at higher recall in screening '
'for clinical trials, especially in settings where PASC diagnosis codes are used less '
'frequently or less reliably. Analysis of how specific features contribute to the '
'classification process may assist in gaining a better understanding of features that are '
'associated with PASC diagnoses.</jats:p></jats:sec><jats:sec><jats:title>Funding '
'Source</jats:title><jats:p>This research was funded by the National Institutes of Health '
'(NIH) Agreement OT2HL161847-01 as part of the Researching COVID to Enhance Recovery (RECOVER) '
'program of '
'research.</jats:p></jats:sec><jats:sec><jats:title>Disclaimer</jats:title><jats:p>The content '
'is solely the responsibility of the authors and does not necessarily represent the official '
'views of the RECOVER Program, the NIH or other funders.</jats:p></jats:sec>',
'DOI': '10.1101/2022.12.22.22283791',
'type': 'posted-content',
'created': { 'date-parts': [[2022, 12, 26]],
'date-time': '2022-12-26T23:30:13Z',
'timestamp': 1672097413000},
'source': 'Crossref',
'is-referenced-by-count': 0,
'title': 'A machine learning-based phenotype for long COVID in children: an EHR-based study from the '
'RECOVER program',
'prefix': '10.1101',
'author': [ { 'ORCID': 'http://orcid.org/0000-0003-0561-6587',
'authenticated-orcid': False,
'given': 'Vitaly',
'family': 'Lorman',
'sequence': 'first',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0001-5001-1590',
'authenticated-orcid': False,
'given': 'Hanieh',
'family': 'Razzaghi',
'sequence': 'additional',
'affiliation': []},
{'given': 'Xing', 'family': 'Song', 'sequence': 'additional', 'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-8307-1642',
'authenticated-orcid': False,
'given': 'Keith',
'family': 'Morse',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-2189-108X',
'authenticated-orcid': False,
'given': 'Levon',
'family': 'Utidjian',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0001-9846-0280',
'authenticated-orcid': False,
'given': 'Andrea J.',
'family': 'Allen',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-0334-6301',
'authenticated-orcid': False,
'given': 'Suchitra',
'family': 'Rao',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0001-5251-2399',
'authenticated-orcid': False,
'given': 'Colin',
'family': 'Rogerson',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0003-1483-4236',
'authenticated-orcid': False,
'given': 'Tellen D.',
'family': 'Bennett',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-9678-5564',
'authenticated-orcid': False,
'given': 'Hiroki',
'family': 'Morizono',
'sequence': 'additional',
'affiliation': []},
{'given': 'Daniel', 'family': 'Eckrich', 'sequence': 'additional', 'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0001-9921-0419',
'authenticated-orcid': False,
'given': 'Ravi',
'family': 'Jhaveri',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0003-3265-9902',
'authenticated-orcid': False,
'given': 'Yungui',
'family': 'Huang',
'sequence': 'additional',
'affiliation': []},
{'given': 'Daksha', 'family': 'Ranade', 'sequence': 'additional', 'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-1637-4942',
'authenticated-orcid': False,
'given': 'Nathan',
'family': 'Pajor',
'sequence': 'additional',
'affiliation': []},
{'given': 'Grace M.', 'family': 'Lee', 'sequence': 'additional', 'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0003-1252-068X',
'authenticated-orcid': False,
'given': 'Christopher B.',
'family': 'Forrest',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-8967-0662',
'authenticated-orcid': False,
'given': 'L. Charles',
'family': 'Bailey',
'sequence': 'additional',
'affiliation': []}],
'member': '246',
'reference': [ { 'key': '2022122802000890000_2022.12.22.22283791v1.1',
'doi-asserted-by': 'publisher',
'DOI': '10.3390/life12020285'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.2',
'doi-asserted-by': 'publisher',
'DOI': '10.1007/s40124-021-00260-x'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.3',
'doi-asserted-by': 'publisher',
'DOI': '10.1101/2022.05.24.22275544'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.4',
'doi-asserted-by': 'publisher',
'DOI': '10.1101/2022.05.24.22275398'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.5',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/S2589-7500(22)00048-6'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.6',
'doi-asserted-by': 'publisher',
'DOI': '10.1101/2022.04.23.22274218'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.7',
'unstructured': 'HAN Archive - 00432 | Health Alert Network (HAN). Published September '
'21, 2021. Accessed August 18, 2022. '
'https://emergency.cdc.gov/han/2020/han00432.asp'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.8',
'doi-asserted-by': 'publisher',
'DOI': '10.1007/s12519-021-00499-w'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.9',
'doi-asserted-by': 'publisher',
'DOI': '10.1007/s40124-021-00259-4'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.10',
'doi-asserted-by': 'publisher',
'DOI': '10.1007/s00431-021-04345-z'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.11',
'doi-asserted-by': 'publisher',
'DOI': '10.3389/fimmu.2021.686029'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.12',
'doi-asserted-by': 'publisher',
'DOI': '10.1111/1541-0420.00039'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.13',
'unstructured': 'CDC. Healthcare Workers. Centers for Disease Control and Prevention. '
'Published February 11, 2020. Accessed August 18, 2022. '
'https://www.cdc.gov/coronavirus/2019-ncov/hcp/clinical-care/post-covid-public-health-recs.html'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.14',
'unstructured': 'Coding Long COVID: Characterizing a new disease through an ICD-10 lens | '
'medRxiv. Accessed August 18, 2022. '
'https://www.medrxiv.org/content/10.1101/2022.04.18.22273968v1'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.15',
'unstructured': 'CDC Announces Approval of ICD-10 Code for Post-Acute Sequelae of '
'COVID-19. aapmr.org. Accessed August 22, 2022. '
'https://www.aapmr.org/members-publications/member-news/member-news-details/2021/07/20/cdc-announces-approval-of-icd-10-code-for-post-acute-sequelae-of-covid-19'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.16',
'doi-asserted-by': 'publisher',
'DOI': '10.1097/EDE.0000000000000907'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.17',
'doi-asserted-by': 'publisher',
'DOI': '10.1371/journal.pone.0118432'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.18',
'unstructured': 'Lundberg SM , Lee SI. A Unified Approach to Interpreting Model '
'Predictions. In: Advances in Neural Information Processing Systems. Vol '
'30. Curran Associates, Inc.; 2017. Accessed August 18, 2022. '
'https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.19',
'doi-asserted-by': 'crossref',
'unstructured': 'GPUTreeShap: massively parallel exact calculation of SHAP scores for '
'tree ensembles [PeerJ]. Accessed August 18, 2022. '
'https://peerj.com/articles/cs-880/',
'DOI': '10.7717/peerj-cs.880'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.20',
'unstructured': 'XGBoost | Proceedings of the 22nd ACM SIGKDD International Conference on '
'Knowledge Discovery and Data Mining. Accessed August 18, 2022. '
'https://dl.acm.org/doi/10.1145/2939672.2939785'},
{ 'key': '2022122802000890000_2022.12.22.22283791v1.21',
'doi-asserted-by': 'publisher',
'DOI': '10.1007/s00431-022-04600-x'}],
'container-title': [],
'original-title': [],
'link': [ { 'URL': 'https://syndication.highwire.org/content/doi/10.1101/2022.12.22.22283791',
'content-type': 'unspecified',
'content-version': 'vor',
'intended-application': 'similarity-checking'}],
'deposited': { 'date-parts': [[2022, 12, 28]],
'date-time': '2022-12-28T10:00:33Z',
'timestamp': 1672221633000},
'score': 1,
'resource': {'primary': {'URL': 'http://medrxiv.org/lookup/doi/10.1101/2022.12.22.22283791'}},
'subtitle': [],
'short-title': [],
'issued': {'date-parts': [[2022, 12, 26]]},
'references-count': 21,
'URL': 'http://dx.doi.org/10.1101/2022.12.22.22283791',
'relation': {},
'published': {'date-parts': [[2022, 12, 26]]},
'subtype': 'preprint'}
