Constructing and evaluating alternative prediction models for identifying clinically relevant cases in an STD clinic

Background: Many STD clinics have incorporated “express visits” – testing-only visits without a clinician evaluation. To identify which patients can safely receive express care at the Public Health–Seattle & King County (PHSKC) STD Clinic, we constructed and evaluated alternative triage algorithms based on computer assisted self-interview (CASI) responses. Methods: We evaluated the performance of the current triage algorithm, constructed optimized algorithms, and compared the performance of the current and optimized algorithms to a simpler, and potentially easier to implement, algorithm. We used CASI responses from all new problem visits between October 2010 and June 2015 to reconstruct a triage status using the current algorithm, which considers age, gender, symptoms, contact to STD, and health service needs. The outcome measure, need for a standard visit, included report of key symptoms, receipt of empiric treatment, or diagnosis with an infection that could have been diagnosed and treated at the visit. We estimated the sensitivity, specificity, and area under the receiver operating curve (AUC) of the current algorithm, by gender, to appropriately triage patients. We used Classification and Regression Tree (CART) analysis to construct and validate gender-specific optimized triage algorithms, considering 11 potential predictors of the outcome. We compared the sensitivity, specificity, and AUC of the current algorithm, optimized algorithms, and a simple algorithm based only on symptoms and contact to STD. Results: Between October 2010 and June 2015, patients completed the CASI at 32,113 visits, including 7,639 women (23.8%) and 24,474 men (76.2%). The current algorithm appropriately triaged 6,259 women (81.9%) and 21,337 men (87.2%). For women, the current triage algorithm had 97.9% sensitivity, 33.0% specificity, and AUC=0.65 (95%CI=0.64-0.67). For men, the current triage algorithm had 94.6% sensitivity, 71.9% specificity, and AUC=0.83 (95%CI=0.83-0.84). In the validation sample of 2,342 women and 6,984 men, the optimized algorithm appropriately triaged 2,136 women (91.2%) and 6,282 men (89.9%), and the simple algorithm appropriately triaged 2,123 women (90.6%) and 6,150 men (88.1%). For women, the optimized algorithm had 93.2% sensitivity, 86.4% specificity, and AUC=0.90 (95%CI=0.88-0.91). For men, the optimized algorithm had 93.5% sensitivity, 82.9 specificity, and AUC=0.88 (95%CI=0.87-0.89). For women, the simple algorithm had 92.9% sensitivity, 85.0% specificity, and AUC=0.89 (95%CI=0.87-0.90). For men, the simple algorithm had 90.8% sensitivity, 82.6% specificity, and AUC=0.87 (95%CI=0.86-0.88). The simple and optimized algorithms triaged more patients to express care (optimized=31.4%, simple=32.6%) than the current algorithm (23.3%). Conclusions: The sensitivity of the current triage algorithm was very high for both men and women; however, the specificity was low, leading to reduced efficiency. In most settings, the simple algorithm would be preferred over the optimized algorithm due to its simplicity and comparable performance. The current algorithm can be implemented to maximize disease detection, while the simple algorithm can be implemented to maximize clinic efficiency.