Zhuoyu Sun1,2,3†Yi’an Guo4†Wei He5†Shiyue Chen1,2,3Changqing Sun6Hong Zhu1,2,3Jing Li1,2,3
Yongjie Chen1,2,3Yue Du2,3,7Guangshun Wang8Xilin Yang1,2,3*
Hongjun Su9*- 1Department of Epidemiology and Biostatistics, School of Public Health, Tianjin Medical University, Tianjin, China
- 2Tianjin Key Laboratory of Environment, Nutrition and Public Health, Tianjin, China
- 3Tianjin Center for International Collaborative Research in Environment, Nutrition and Public Health, Tianjin, China
- 4Department of Radiotherapy, Baodi Clinical College of Tianjin Medical University, Tianjin, China
- 5Department of Ophthalmology, Baodi Clinical College of Tianjin Medical University, Tianjin, China
- 6Department of Neurosurgery, Baodi Clinical College of Tianjin Medical University, Tianjin, China
- 7Department of Social Medicine and Health Service Management, School of Public Health, Tianjin Medical University, Tianjin, China
- 8Department of Tumor, Baodi Clinical College of Tianjin Medical University, Tianjin, China
- 9Department of Neurology, Baodi Clinical College of Tianjin Medical University, Tianjin, China
Objectives: To develop and internally validate two clinical risk scores to detect coronavirus disease 2019 (COVID-19) during local outbreaks.
Methods: Medical records were extracted for a retrospective cohort of 336 suspected patients admitted to Baodi hospital between 27 January to 20 February 2020. Multivariate logistic regression was applied to develop the risk-scoring models, which were internally validated using a 5-fold cross-validation method and Hosmer-Lemeshow (H-L) tests.
Results: Fifty-six cases were diagnosed from the cohort. The first model was developed based on seven significant predictors, including age, close contact with confirmed/suspected cases, same location of exposure, temperature, leukocyte counts, radiological findings of pneumonia and bilateral involvement (the mean area under the receiver operating characteristic curve [AUC]:0.88, 95% CI: 0.84–0.93). The second model had the same predictors except leukocyte and radiological findings (AUC: 0.84, 95% CI: 0.78–0.89, Z = 2.56, p = 0.01). Both were internally validated using H-L tests and showed good calibration (both p > 0.10).
Conclusion: Two clinical risk scores to detect COVID-19 in local outbreaks were developed with excellent predictive performances, using commonly measured clinical variables. Further external validations in new outbreaks are warranted.
Introduction
The outbreak of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), emerged in Wuhan, China in December 2019, and has been rapidly spreading worldwide. On 30 January 2020, the World Health Organization (WHO) declared the COVID-19 outbreak a public health emergency of international concern (PHEIC) [1] and, on 11 March 2020, characterized it as a pandemic [2]. Up to 18 July 2022, a total of 562 million cases have been reported across 228 countries and territories, resulting in over 6.37 million deaths [3]. The rapid spread of COVID-19 globally, which has significant morbidity with no proven treatment, presents a huge challenge for healthcare system worldwide.
The number of new cases has been increasing rapidly largely due to the easy transmissibility of the virus, especially the hugely infectious Omicron subvariants BA.4 and BA.5, by milder cases or asymptomatic carriers [4]. The transmission often takes place in clusters, such as families [5], working places or schools [6], and crowded public places (e.g., shopping malls or restaurant) [7–9]. Currently, diagnosis of COVID-19 is mainly dependent on specialized nucleic acid-based quantitative real-time reverse transcription polymerase chain reaction (RT-PCR) testing, which is not readily available in low-resource regions [10]. Even in well-resourced areas with local outbreaks, the mass testing to detect cases in millions of population will be a high demand on nursing and laboratory resources. Given that up to 54% of COVID-19 patients may have an initial negative RT-PCR result [11], repeated testing and longer waiting time for the results (more than 24 hours) in hospitals cause a huge burden on already overcrowded healthcare system. In order to prevent the false-negative results, a combination of RT-PCR with clinical assessment, blood tests and imaging has been recommended to help clinicians in the triage of patients at high risk of COVID-19 [12].
A clinical risk-scoring model is certainly to serve the purpose, which combines demographic and clinical variables to identify COVID-19 cases from high-risk patients who need to undergo the RT-PCR testing. At present, several clinical diagnostic models have been developed, but the quality of these models was inconclusive [13]. In a systematic review of 33 diagnostic models for prediction of COVID-19, Wynants et al. concluded that all those studies were suboptimal due to use of undefined study populations, high risk of model overfitting, inappropriate statistical analyses and poor reporting [13]. In a recent study on development and validation of a clinical risk score, the authors focused on patients confirmed with COVID-19 and the outcome of an intensive care unit admission. Their primary objective was to detect patient at high risk of deterioration, but not to develop an efficient tool to classify patients at high risk of COVID-19 [14]. Thus, the present study aimed to construct a clinical risk score based on socio-demographics, epidemiological contact history, clinical features, laboratory and radiological findings, which can identify COVID-19 patients from high-risk individuals during a local outbreak in China. Given that blood tests and CT scans can be invasive and time-consuming, second risk score was also developed without these variables. The risk score will be important for clinicians to make better clinical decisions and better allocate healthcare resource.
Methods
Data Sources
Tianjin is a municipality in Northern China, with 15.6 million inhabitants in 2019. As the fourth largest city in China, Tianjin is made up of 12 central districts, and four suburban districts, including Baodi, Jizhou, Jinghai and Ninghe. As of February 25, 2020, a total of 135 cases were diagnosed in Tianjin, and almost half of them were reported in Baodi (60/135), where a cluster outbreak of COVID-19 related to Baodi Shopping Mall occurred during local transmission of the disease in Tianjin [15]. The detailed tracing of the transmission chain of the cluster outbreak was available elsewhere [7]. Briefly, the first case, an employee of the shopping mall, had a close contact with a patient with undiagnosed infection in other province, and subsequently three other employees were infected but were still on duty after symptom onset. Consequently, six employees and 19 customers were infected, whose exposures were directly linked to Baodi Shopping Mall during 19 to 25 January 2020 (9).
Cohort Description
The present study reviewed medical records of a retrospective cohort of hospitalized patients with suspected infection admitted to Baodi Hospital, Tianjin (China) during 27 January to 20 February 2020 (the second and third stage of the local outbreak in Tianjin) [16]. Patients were not involved in the recruitment to and conduct of the study. Patients’ medical records were excluded if they were younger than 18 years old, or if their results of SARS-COV-2 nucleic acid testing were unavailable at time of data collection. This study was approved by the Institutional Ethics Review Board of Baodi Hospital, Tianjin (#BDYYLL20200011). Written informed consent was waived due to the use of anonymous data. Reporting of the present study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [17] (Supplementary TRIPOD Checklist).
Diagnosis of COVID-19
In accordance with the WHO interim guidance, the diagnosis of COVID-19 was confirmed by a positive high-throughput sequencing or real-time reverse transcription polymerase chain reaction (RT-PCR) assay for SARS-COV-2 nucleic acid from oropharyngeal specimens [18]. The laboratory work was done in the Center for Disease Control and Prevention (CDC) of Baodi, Tianjin (China).
Data Collection and Extraction
A trained team of physicians and nurses who were blind to the diagnosis of patients retrospectively reviewed medical records and extracted the required data from the electronic management system of Baodi Hospital, Tianjin. Data were extracted based on clinical knowledge, literature and data availability, which included demographics (age, sex and occupation), exposure history, clinical symptoms, laboratory findings and pulmonary computed tomography (CT) scan findings. Exposure history was collected by investigators from Tianjin Centers for Disease Control and Prevention (CDC), who conducted a face-to-face epidemiological investigation for each patient, and the contact tracking was performed to identify potential infection sources [9]. Based on the rigorous epidemiological investigation, exposure history within 14 days before hospital admission included: 1) had a close contact with confirmed or suspected cases of COVID-19; 2) same location of exposure (e.g., had visited the Baodi Shopping Mall or had a close contact with those who visited the same place); 3) had a history of travel to Wuhan or many places in China (including Wuhan).
Clinical symptoms previously reported to be associated with COVID-19 infection were extracted, involving fever, cough, sore throat, sneezing, runny nose, sputum production and digestive symptoms (diarrhea, nausea and vomiting) [15]. Total counts of leukocyte or lymphocyte were extracted and classified into three groups: 1) being within normal range, 2) increased (leukocyte count >10.0 × 109/L; lymphocyte count >4.0 × 109/L), and 3) decreased (leukocyte count<3.5 × 109/L; lymphocyte: <1.0 × 109/L). The CT imaging features reported in previous studies on COVID-19 were recorded here, including bilateral involvement, lower lobes involvement, ground-glass opacification (GGO), peripheral distribution (the lesions distributed in the outer 1/3 of the lung), multiple small plaques, interstitial changes (incl. interstitial thickening or fibrous stripes), and pulmonary nodules [19]. All CT images were reviewed independently by two radiologists. A third experienced radiologist was consulted if there was a disagreement in interpreting CT findings.
Statistical Analysis
Characteristics of study participants were summarized by mean ± standard deviation (SD) for continuous variables or by numbers and percentages for categorical variables. Differences of characteristics between patients with or without COVID-19 infection were compared using Student’s t-test, chi-square test or Fisher Exact test where appropriate. Complete-case analysis was used for missing variables (missing values <1% in the present study).
Development of the Clinical Risk Scores
Binary logistic regression analysis was used to identify potential predictors for COVID-19. Potential predictor variables were selected into the multivariate model if the variables had a p-value < 0.20 in the univariate analysis, if they were judged to be of clinical importance, or if they had been consistently reported to be associated with COVID-19 in previous studies. In the multivariate logistic regression, an enter method instead of a stepwise method was used for predictor selection to avoid model over-fitting, with statistically significant variables (p < 0.05) being retained in the final prediction model [20]. The prediction model was further translated into clinical risk scores using the equation as below, with
Internal Validation of the Developed Risk Scores
To avoid over-fitting of the constructed risk score, the shrinkage factor was calculated using (x2-k)/x2, where x2 was the likelihood ratio x2 and k was the number of the predictors in the model. A shrinkage factor <0.85 may raise the concern of over-fitting. If this is the case, regression coefficients of the predictors will be adjusted for over-optimism by multiplying the shrinkage factor [21]. The performance of the developed risk score was evaluated by calibration and discrimination measures [17]. Hosmer-Lemeshow chi-square tests and calibration plots were used to test if the observed and expected probabilities of COVID-19 were comparable over the deciles of predicted risk [22]. A p-value > 0.10 indicated similarity in the predicted and observed probability, i.e., the internal calibration being acceptable. A 5-fold cross-validation method was performed to evaluate the internal discrimination, using the mean area under the receiver operating characteristic curve (AUC) [23]. Using this approach, the cohort was partitioned into five folds of equal size, with one of them selected as the testing set and the remaining four merged as the training set. This approach was repeated five times to ensure that each fold was selected once as the testing set. An AUC is closer to 100%, indicating that the model’s discrimination is perfect. If discrimination is no better than chance, then the AUC is around 50%. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were further calculated at different cutoff points of the risk scores, for clinical application based on local medical resources. All statistical analyses were carried out using SAS 9.4 (SAS Institute, Inc., Cary, United States), SPSS 24.0 (IBM/SPSS, Inc., Chicago IL), R 3.6.3, and MedCalc 20.112. The differences were considered as statistically significant if p < 0.05.
Results
Characteristics of Patients in the Study
A total of 336 patients were enrolled in the study, and 56 patients were diagnosed with COVID-19 (Table 1). In total, 56 confirmed cases had received 105 tests for SARS-COV-2 nucleic acid, with an average of 2 tests per patient. The average age of 336 patients were 44.1 ± 15.4 years, ranging from 18–93 years. The majority of patients were females (59.5%) and farmers (56.2%). Of them, 28.9% had a close contact with confirmed or suspected cases, and 32.4% had the same location of exposure (incl. had a history of visit to Baodi Shopping Mall or had a close contact with those who visited the same place during 19 to 25 January 2020). Fever (89.6%), normal levels of leukocyte (82.1%) or lymphocyte counts (78.9%), and CT scan findings suggestive of pneumonia (80.7%) were among the most common clinical features in the patients with suspected infection.
TABLE 1. Comparisons between patients with or without COVID-19 in terms of socio-demographic characteristics, exposure history, clinical features and pulmonary CT scan findings in the retrospective cohort study, China, 2020.
COVID-19 patients were older than non-COVID-19 patients. The proportion of subjects with a history of a close contact with confirmed or suspected cases was higher in patients with COVID-19 than those without it. More patients in the COVID-19 group had reduced leukocyte or lymphocyte count. CT scan findings of pneumonia, bilateral involvement, GGO, peripheral distribution were more common among the confirmed cases than their counterparts. No significant differences were observed between the two groups in terms of sex, occupation, fever, respiratory symptoms and digestive symptoms.
Predictor Selection and Construction of the Clinical Risk Scores
Based on results from the univariate analysis and clinical relevance, 19 variables were included in the multivariate logistic regression, involving age, sex, occupation, close contact with confirmed or suspected cases, same location of exposure, fever, temperature, ≥1 respiratory symptoms, cough, sore throat, sneezing/runny nose, sputum production, ≥1 digestive symptoms, leukocyte count category, lymphocyte count category, pulmonary CT findings of pneumonia, bilateral involvement, ground-glass opacification and peripheral distribution. After predictor selection, seven variables remained to be significant predictors of COVID-19, including age, close contact with confirmed or suspected cases, same location of exposure, temperature, leukocyte count category, CT findings of pneumonia and bilateral involvement (model 1). The results of 5-fold cross-validation method showed that the same predictors were included in the model 1. Given that blood testing and CT scan imaging can be invasive and time-consuming, the model 2 was constructed using the same predictors except leukocyte counts and CT findings. Temperature was centered around its mean by subtracting the mean (i.e., 37.39) to avoid a large intercept. Parameter estimates and odds ratios of predictors in two models were demonstrated in Table 2. The final risk score of COVID-19 in model 1 and model 2 were constructed as follows:
TABLE 2. Parameter estimates and odds ratios of predictors of COVID-19 infection in the retrospective cohort study, China, 2020.
Model 1: COVID-19 risk score = 0.02×age (years) + 3.59×exposure (close contact, 1 for yes/0 for no) + 1.38×exposure (same location of exposure, 1 for yes/0 for no) + 0.93×(temperature-37.39, Celsius) - 0.67×leukocyte (increased, 0 for normal/1 for decreased/2 for increased) + 1.43×leukocyte (decreased, 0 for normal/1 for decreased/2 for increased) + 2.61×CT (pneumonia) + 0.90×CT (bilateral involvement) + 7.52
Model 2: COVID-19 risk score = 0.03×age (years) + 3.27×exposure (close contact, 1 for yes/0 for no) + 1.44×exposure (same location of exposure, 1 for yes/0 for no) + 0.97×(temperature-37.39, Celsius) + 4.95
The shrinkage factor of model 1 and model 2 were 0.93 and 0.94, which was significantly higher than the value of over-fitting criteria (λ = 0.85), indicating that the performance of two models was only overestimated by 7.5% and 5.6%, respectively. The first risk score (model 1) had a good internal calibration, with the predicted probabilities of COVID-19 being similar to the observed probabilities (x2 for Hosmer-Lemeshow test = 5.59, p = 0.69). The second risk score (model 2) had similar predicted probabilities compared with the observed probabilities (x2 for Hosmer-Lemeshow test = 11.19, p = 0.19) (Figure 1). Using a 5-fold cross-validation procedure, the mean AUC of the first risk score was 0.88 (95% CI: 0.84–0.93), suggesting an excellent discrimination of the risk score for prediction of COVID-19 in the study. After excluding laboratory and radiological findings, the mean AUC of the second risk score significantly decreased but was quite acceptable (AUC:0.84 95% CI:0.78–0.89, Z = 2.56, p = 0.01) (Figure 2).
FIGURE 1. Calibration plots for both risk scores to detect COVID-19 cases in the retrospective cohort study, China, 2020.
FIGURE 2. Area under receiver operating characteristic curve (AUC) of both risk scores in the retrospective cohort study, China, 2020. Abbreviation: Se, sensitivity; Sp, specificity.
Sensitivity, Specificity, PPV and NPV at Different Cutoff Points of the Risk Scores
Because of high transmissibility of COVID-19 and high rates of morbidity and fatality, cutoff points of 3.45 (model 1) and 1.35 (model 2) were recommended to avoid false negative results (Table 3). For instance, at the cutoff point of 3.45 in the model 1 (or 1.35 in the model 2), 100% of COVID-19 patients were identified with a specificity of 27.1% (or 22.4%). With use of these cutoff points, around 76.5%∼80.4% of patients would need to undergo the SARS-CoV-2 nucleic acid testing, without any cases being missed. If 4.40 were set as a threshold in the model 1, 45.8% of patients could avoid the tests, with the missed diagnosis rate of 5.6%. The PPV and NPV were 28.5% and 97.9%, respectively. While a cutoff point of 2.25 in the model 2 might rule out 41.1% of patients to undergo the tests, with a PPV of 26.4% and a NPV of 97.7%.
TABLE 3. Sensitivity, specificity, predictive values at different cutoff points of the risk scores in the retrospective cohort study, China, 2020.
Discussion
In this study, two clinical risk scores were developed and internally validated to identify COVID-19 cases from highly suspected patients during a cluster outbreak, based on commonly measured clinical variables. The predictive performances of both risk-scoring models were satisfactory with excellent internal calibration and discrimination. The risk scores constructed can be easily used by clinicians to identify high-risk patients for the SARS-CoV-2 nucleic acid testing, while up to 20%∼40% of them were exempted from the testing.
It is worth noting that some variables can be immediately collected upon hospital arrival, such as socio-demographics, clinical symptoms and contact tracing history; while others such as vital signs, laboratory or radiological tests are time consuming and only available at a later stage. Two parallel prediction models were constructed in our study, one with and the other without laboratory tests and imaging findings. The full model with all seven predictors produced a mean AUC of 0.88 (95% CI: 0.84–0.93), while the simplified 4-predictor model had an AUC of 0.84 (95% CI: 0.78–0.89, Z = 2.56, p = 0.01). Consistent with our results, a validation and agreement study evaluated six previously published models and found that models based only on clinical symptoms and/or risk exposures had lower but though acceptable performance values compared with those based on biological parameters and/or radiological findings, highlighting the importance of these parameters to obtain more efficient models [24]. In case of limited testing facilities, the simplified risk score was able to maintain good performance, still being useful as early screening procedure. When testing facilities are sufficient or an immediate level of risk is resulted, it would be more appropriate to perform further examinations to confirm COVID-19 positivity and minimize false negative results.
In the current study, age, contact with confirmed or suspected cases, same location of exposure, temperature, leukocyte count category, CT scan findings suggestive of pneumonia and bilateral involvement were found to be most significant predictors of COVID-19, which were in line with the guideline for COVID-19 diagnosis and literature [13, 19]. The guidelines for diagnosis and treatment of COVID-19 infection, issued by the National Health Commission of the People’s Republic of China, suggested that patients had a high risk of COVID-19 if they had at least two of the following symptoms, in addition to a history of travel to Wuhan or contact with other infected patient: 1) fever and/or respiratory symptoms; 2) imaging features of pneumonia; 3) normal or reduced leukocyte count, or reduced lymphocyte count [25]. Recent published systematic reviews of 21 studies on diagnostic models to detect COVID-19 in patients with suspected infection indicated that the recurrent significant predictors were socio-demographics (age and gender), epidemiological contact history, clinical symptoms (temperature or fever), vital signs, laboratory or biological tests (leukocytes or lymphocytes) and radiological examinations (pneumonia signs on CT scan) [13, 26]. Thirty four prediction models for diagnosis of COVID-19 were proposed based on images, and most of them used CT images [13]. Our findings provided evidence to support that imaging features of pneumonia was the typical appearance of COVID-19 radiologically [13, 25–27].
In the current study, two clinical risk scores were constructed based on a retrospective cohort of patients with suspected infection in a hospital setting during a cluster outbreak. Both risk scores were developed based on commonly measured clinical variables, with excellent internal calibration and discrimination to detect COVID-19 cases from high risk individuals who need to undergo the nucleic acid testing. Our study revealed significant predictors of COVID-19 in a cluster outbreak, which were consistent with the latest systematic reviews [13, 26]. Our findings of two risk scores developed had an implication in terms of applicability in clinical practice. When testing facilities are sufficient, a full model with laboratory and imaging findings can be more effective to detect COVID-19 cases. In case of limited testing resources with a high volume of patients, a simplified risk score can be useful to identify COVID-19 patients at first admission. Moreover, the risk scores constructed can be easily used by clinicians to decide which risk scores would produce the best cut-offs for RT-PCR testing. Hence, in well-resourced areas with good access to healthcare, a risk score of 3.45 in the model 1 (or 1.35 in the model 2) might be set as a threshold, and 76.5% of patients (or 80.4% of patients in the model 2) would receive the nucleic acid tests, with no cases being missed. On the contrary, in areas with local outbreaks or limited diagnostic resources, a cutoff point of 4.40 in the model 1 (or 2.25 in the model 2) might rule out 45.8% of patients (41.1% of patients in the model 2) to undergo the tests, as long as they could be isolated at home to minimize the transmission of COVID-19 to others.
A major limitation of the present study was that the risk score was developed using a small sample of patients in the early phase of COVID-19. Further external validations in independent samples from local outbreaks of COVID-19 are warranted. Nevertheless, our study sample was collected in the early outbreak of COVID-19 and was less likely to be confounded by the use of prevention measures, such as use of various vaccines. Using a 5-fold cross-validation procedure, the same predictors were included in the final model, and the mean AUC of both risk scores was still excellent. Additionally, seven significant predictors of COVID-19 identified in our study were consistent with previous literature [13, 25–27]. Dardenne et al. evaluated and compared diagnostic models for COVID-19 identified in a recently published systematic review, and they found similar results between the first and the second wave in terms of model performances [24]. Theoretically speaking, application of any model in the clinical practice should undergo a thorough investigation and a pilot testing phase. In that case, findings from the current study might be a good initiative to develop and refine them in other settings. Again, Some potential predictors reported in previous studies were unavailable in the current study, such as neutrophil-to-lymphocyte ratio, C reactive protein, and heart ratio [13, 27]. However, considering the practical issue in clinical application, a simple and interpretable model is often preferred. This is a single-center retrospective study, which may limit the generalizability of the risk scores in other places. Multi-center studies with a large sample of patients are urgently needed.
Conclusion
In the current study, two clinical risk scores to detect COVID-19 in highly suspected patients during a local outbreak was developed based on commonly measured clinical variables, with excellent predictive performances. The risk scores may assist clinicians in screening high-risk patients for COVID-19 with exemption of 20∼40% from further RT-PCR testing, thus better use of limited healthcare resources. However, further external validations using standardized and multi-centered datasets are needed.
Data Availability Statement
The dataset supporting the conclusion of this article is available upon reasonable request from the corresponding author.
Ethics Statement
The studies involving human participants were reviewed and approved by the Institutional Ethics Review Board of Baodi Hospital, Tianjin. The ethics committee waived the requirement of written informed consent for participation.
Author Contributions
XY, HS, and ZS had the idea and designed the study. CS and GW collected and extracted the data. ZS, SC, HZ, JL, YC, and YD performed statistical analyses. ZS, YG, and WH wrote the first draft of the manuscript. ZS, XY, and HS take full responsibility for the integrity of the data and the accuracy of the data analysis. All authors contributed to the interpretation of the findings and critically commented the manuscript. All authors approved the final manuscript.
Funding
This study was supported by the National Natural Science Foundation of China (grant number 82003440) and the National Social Science Foundation of China (grant number 19BGL247). The funder was not involved in the study design, collection, analysis, interpretation of data, the writing of this article or the decision to submit it for publication.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We are grateful to Drs. Changping Li and Zhuang Cui from the Department of Epidemiology and Biostatistics at Tianjin Medical University for assistance with model validation, and to medical staff at Baodi hospital, Tianjin, for their invaluable assistance with data collection in the study.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.ssph-journal.org/articles/10.3389/ijph.2022.1604794/full#supplementary-material
References
1.Statement on the second meeting of the International Health Regulations. Emergency Committee Regarding the Outbreak of Novel Coronavirus (2019-nCoV). Geneva: World Health Organization (2005). Website: Available at: https://www.who.int/news-room/detail/30-01-2020-statement-on-the-second-meeting-of-the-international-health-regulations-(2005)-emergency-committee-regarding-the-outbreak-of-novel-coronavirus-(2019-ncov (Accessed on July 8, 2020).
2.WHO. WHO Director-General's Opening Remarks at the media Briefing on COVID-19. Geneva: World Health Organization (2005). Website: Available at: https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020 (Accessed on July 8, 2020).
3.COVID-19 Dashboard by the Center for Systems Science and Engineering at Johns Hopkins University. ArcGIS. Johns Hopkins University. Website: Available at: https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6 (Accessed on July 8, 2020).
4. Wu, Z, and McGoogan, JM. Characteristics of and Important Lessons from the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72314 Cases from the Chinese Center for Disease Control and Prevention. JAMA (2020) 323:1239–42. doi:10.1001/jama.2020.2648
5. Chan, JF, Yuan, S, Kok, KH, To, KK, Chu, H, Yang, J, et al. A Familial Cluster of Pneumonia Associated with the 2019 Novel Coronavirus Indicating Person-To-Person Transmission: a Study of a Family Cluster. Lancet (2020) 395(10223):514–23. doi:10.1016/S0140-6736(20)30154-9
6. Rothe, C, Schunk, M, Sothmann, P, Bretzel, G, Froeschl, G, Wallrauch, C, et al. Transmission of 2019-nCoV Infection from an Asymptomatic Contact in Germany. N Engl J Med (2020) 382(10):970–1. doi:10.1056/NEJMc2001468
7. Liu, YF, Li, JM, Zhou, PH, Liu, J, Dong, XC, Lyu, J, et al. Analysis on Cluster Cases of COVID-19 in Tianjin. Zhonghua Liu Xing Bing Xue Za Zhi (2020) 41(5):653–6. doi:10.3760/cma.j.cn112338-20200225-00165
8. Lu, J, Gu, J, Li, K, Xu, C, Su, W, Lai, Z, et al. COVID-19 Outbreak Associated with Air Conditioning in Restaurant, Guangzhou, China, 2020. Emerg Infect Dis (2020) 26(7):1628–31. doi:10.3201/eid2607.200764
9. Wu, WS, Li, YG, Wei, ZF, Zhou, PH, Lyu, LK, Zhang, GP, et al. Investigation and Analysis on Characteristics of a Cluster of COVID-19 Associated with Exposure in a Department Store in Tianjin. Zhonghua Liu Xing Bing Xue Za Zhi (2020) 41(4):489–93. doi:10.3760/cma.j.cn112338-20200221-00139
10. Pang, J, Wang, MX, Ang, IYH, Tan, SHX, Lewis, RF, Chen, JI, et al. Potential Rapid Diagnostics, Vaccine and Therapeutics for 2019 Novel Coronavirus (2019-nCoV): A Systematic Review. J Clin Med (2020) 9(3):E623. doi:10.3390/jcm9030623
11. Arevalo-Rodriguez, I, Buitrago-Garcia, D, Simancas-Racines, D, Zambrano-Achig, P, Del Campo, R, Ciapponi, A, et al. False-negative Results of Initial RT-PCR Assays for COVID-19: A Systematic Review. PLoS One (2020) 15(12):e0242958. doi:10.1371/journal.pone.0242958
12. Soedarsono, S, Febriani, A, Hasan, H, and Widyoningroem, A. Management of Severe COVID-19 Patient with Negative RT-PCR for SARS-CoV-2: Role of Clinical, Radiological, and Serological Diagnosis. Radiol Case Rep (2021) 16(6):1405–9. doi:10.1016/j.radcr.2021.03.049
13. Wynants, L, Van Calster, B, Collins, GS, Riley, RD, Heinze, G, Schuit, E, et al. Prediction Models for Diagnosis and Prognosis of Covid-19 Infection: Systematic Review and Critical Appraisal. BMJ (2020) 369:m1328. doi:10.1136/bmj.m1328
14. Lacap, EM, Varghese, A, Khamis, F, Al Bahrani, M, Al Naamani, H, Kolamban, S, et al. Development and Validation of the CoV19-OM Intensive Care Unit Score: An Early ICU Identification for Laboratory-Confirmed SARS-CoV-2 Patients from a Retrospective Cohort Study in Oman. Int J Infect Dis (2022) 117:241–6. doi:10.1016/j.ijid.2021.04.069
15. Dong, XC, Li, JM, Bai, JY, Liu, ZQ, Zhou, PH, Gao, L, et al. Epidemiological Characteristics of Confirmed COVID-19 Cases in Tianjin. Zhonghua Liu Xing Bing Xue Za Zhi (2020) 41(5):638–41. doi:10.3760/cma.j.cn112338-20200221-00146
16. Chen, T, Guo, S, and Zhong, P. Epidemic Characteristics of the COVID-19 Outbreak in Tianjin, a Well-Developed City in China. Am J Infect Control (2020) 48:1068–73. doi:10.1016/j.ajic.2020.06.006
17. Collins, GS, Reitsma, JB, Altman, DG, and Moons, KG. Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD Statement. Br J Surg (2015) 102(3):148–58. doi:10.1002/bjs.9736
18.WHO. Laboratory Testing for 2019 Novel Coronavirus (2019-nCoV) in Suspected Human Cases. Geneva: World Health Organization (2020). Website: Available at: https://www.who.int/publications/i/item/laboratory-testing-of-2019-novel-coronavirus-(-2019-ncov)-in-suspected-human-cases-interim-guidance-17-january-2020 July 8, 2020).
19. Salehi, S, Abedi, A, Balakrishnan, S, and Gholamrezanezhad, A. Coronavirus Disease 2019 (COVID-19): A Systematic Review of Imaging Findings in 919 Patients. AJR Am J Roentgenol (2020) 215(1):87–93. doi:10.2214/AJR.20.23034
20. Gao, S, Leng, J, Liu, H, Wang, S, Li, W, Wang, Y, et al. Development and Validation of an Early Pregnancy Risk Score for the Prediction of Gestational Diabetes Mellitus in Chinese Pregnant Women. BMJ Open Diabetes Res Care (2020) 8(1):e000909. doi:10.1136/bmjdrc-2019-000909
21. Harrell, FE, Lee, KL, and Mark, DB. Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors. Stat Med (1996) 15(4):361–87. doi:10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4
22. van Houwelingen, HC. Validation, Calibration, Revision and Combination of Prognostic Survival Models. Stat Med (2000) 19(24):3401–15. doi:10.1002/1097-0258(20001230)19:24<3401::aid-sim554>3.0.co;2-2
23. Bradley, AE. The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms. Pattern Recognition (1997) 30(7):1145–59. doi:10.1016/s0031-3203(96)00142-2
24. Dardenne, N, Locquet, M, Diep, AN, Gilbert, A, Delrez, S, Beaudart, C, et al. Clinical Prediction Models for Diagnosis of COVID-19 Among Adult Patients: a Validation and Agreement Study. BMC Infect Dis (2022) 22(1):464. doi:10.1186/s12879-022-07420-4
25.National Health Office Medical Letter. New Coronavirus Pneumonia Prevention and Control Guideline (Trial Version 7th). Beijing: National Health Commission of the People’s Republic of China (2020). Available at: http://www.nhc.gov.cn/yzygj/s7653p/202003/46c9294a7dfe4cef80dc7f5912eb1989.shtml (Accessed on July 9, 2020).
26. Locquet, M, Diep, AN, Beaudart, C, Dardenne, N, Brabant, C, Bruyere, O, et al. A Systematic Review of Prediction Models to Diagnose COVID-19 in Adults Admitted to Healthcare Centers. Arch Public Health (2021) 79(1):105. doi:10.1186/s13690-021-00630-3
Keywords: COVID-19, risk score, local outbreaks, clinical variables, retrospective cohort study
Citation: Sun Z, Guo Y, He W, Chen S, Sun C, Zhu H, Li J, Chen Y, Du Y, Wang G, Yang X and Su H (2022) Development of Clinical Risk Scores for Detection of COVID-19 in Suspected Patients During a Local Outbreak in China: A Retrospective Cohort Study. Int J Public Health 67:1604794. doi: 10.3389/ijph.2022.1604794
Received: 25 January 2022; Accepted: 15 August 2022;
Published: 06 September 2022.
Edited by:
Calogero Guccio, University of Catania, ItalyReviewed by:
Abraham Varghese, University of technology and applied sciences, OmanCopyright © 2022 Sun, Guo, He, Chen, Sun, Zhu, Li, Chen, Du, Wang, Yang and Su. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Xilin Yang, yxl@hotmail.com; Hongjun Su, redarmy007@163.com
†These authors have contributed equally to this work and share first authorship