Measuring Appropriate Antimicrobial Use: Attempts at Opening the Black Box

Introduction

Unnecessary antimicrobial use places patients at risk of Clostridium difficile infection, infection with resistant organisms and drug toxicities. Roughly 50% of patients in healthcare facilities are prescribed antimicrobials, with 30–50% deemed inappropriate and/or unnecessary [1–3]. The ability to identify and stop inappropriate antimicrobial use is essential to slowing the emergence and spread of antimicrobial resistant organisms. The United States’ (U.S.) National Strategy for Combating Antibiotic-Resistant Bacteria calls for reductions in inappropriate prescribing by 20% in hospitals and 50% in outpatient settings by 2020 [4]. These goals have lent new urgency to efforts to develop more standardized measures of appropriate antimicrobial use.

Efforts to define appropriate antimicrobial use are complicated by the array of clinical scenarios for which antimicrobials are given. In some situations, use of an antimicrobial can clearly be deemed inappropriate, for example when a patient is receiving an antibiotic not treating the bacteria recovered in cultures. Other situations are more subjective. Some experts consider the use of a broad-spectrum agent to treat a susceptible bacterium to be “inappropriate”, while others might classify this therapy as “appropriate, but sub-optimal”. Many studies use these terms interchangeably. Additionally, some antimicrobial use is designated as unnecessary, referring to use for non-infectious syndromes or non-bacterial infections such as viral upper respiratory tract infections.

We sought to review existing literature on systematic approaches to define and measure appropriate antimicrobial use in hospitals. Similar efforts undertaken in outpatient settings will not be addressed. While not exhaustive in scope, this review focuses on the various steps in antimicrobial prescribing that have been evaluated, definitions or measures of appropriateness used, and recent progress in standardizing and implementing assessment tools for evaluating antimicrobial prescribing quality. Finally, we describe a collaborative effort to develop standardized audit tools to assess the appropriateness of antimicrobial prescribing and examples of these tools put into practice.

Steps in Antimicrobial Prescribing Evaluated

Initial assessments of antimicrobial use focused on drug selection and dose, with some evaluation of the accuracy of the diagnosis ascribed to the patient; however, details of methods to determine diagnostic appropriateness are not described [5–7]. Most published assessments of antimicrobial appropriateness (Table 1) come from inpatient settings and simply evaluate empiric and/or definitive drug selection [8–16], with a few evaluations of dosing [17,18] and duration of therapy [19–28].

More recent reports focus on assessing appropriateness of diagnostic evaluations and resultant diagnostic accuracy [15,21,24,29–33], understanding inaccurate diagnosis leads to inappropriate antimicrobial use. Most studies define overall diagnostic accuracy based on expert case review, without detailing specific aberrations in the diagnostic process. A prospective evaluation of antimicrobial use among intensive care unit patients found disagreement concerning antimicrobial choice between two independent ID physicians; however, less disagreement with regards to diagnostic accuracy and antimicrobial indication with 27% (reviewer 1) and 33% (reviewer 2) of patients without diagnostic evidence of infection [15].

More comprehensive descriptions of diagnostic errors come from long-term care facilities (LTCFs) where diagnostic evaluations are frequently inadequate and asymptomatic bacteriuria (ASB) accounted for half of unnecessary antimicrobial regimens, likely fueled by lack of knowledge regarding indications for treatment and misinterpretation of results leading to diagnostic errors [30,32]. CDC’s Emerging Infections Program (EIP) conducted an assessment of antibiotic prescribing among inpatients in 36 U.S. hospitals and found significant rates of diagnostic errors among patients treated for urinary tract infections (UTIs) and those prescribed vancomycin [3]. Sixteen percent of patients prescribed antibiotics for UTIs and 9% of patients prescribed vancomycin lacked a diagnostic culture prior to antibiotics [3]. Interpretation of diagnostic tests was also problematic with 23% of patients treated for a UTI lacking symptoms and/or having negative urine cultures [3]. Similarly, 22% of patients receiving vancomycin had cultures without growth of Gram-positive organisms [3]. Finally, an evaluation of 500 antimicrobial courses in one Veterans Affairs hospital found 36% of diagnoses were incorrect or indeterminate [33]. Sixty-two percent of antimicrobial courses were appropriate when the diagnosis was correct as compared to 5% with an incorrect diagnosis. The majority of antimicrobial courses (84%) prescribed for incorrect diagnoses were not indicated, highlighting the profound impact of diagnostic inaccuracy on antimicrobial use.

There are many opportunities for error along the antimicrobial prescribing pathway, leading to unnecessary antimicrobial use. Due to lack of standardized assessments, quantifying when, why and to what degree errors occur is difficult. However, the aforementioned studies provide insights and the initial framework for developing systematic measurements of antimicrobial prescribing quality.

How to Define Appropriate

Studies evaluating empiric and definitive drug selection often use in vitro antimicrobial susceptibilities; defining inappropriate therapy as absence of antimicrobial agents to which an organism is susceptible or use of an agent to which the organism is resistant [8–11,13,14,16]. Studies applying this definition most commonly evaluate antimicrobial therapy for known bacteremia or critically ill patients with septic shock [9,10,13,14]. This method is one of the least subjective and most clinically important assessments of appropriate therapy because it identifies patients at high risk of treatment failure. Because this definition requires a positive culture, it limits the extent of antimicrobial use that can be assessed by excluding many infectious syndromes lacking culture results. In addition, while bug-drug mismatches are identified by this definition, opportunities for de-escalation may be overlooked.

A retrospective analysis of U.S. hospital administrative data assessing the incidence and economic impact of redundant antimicrobial therapy, defined as use of two agents with activity against the same organism, found avoidable redundant antimicrobial therapy in 78% (394/505) of hospitals [34]. Dual anti-anaerobic therapy accounted for 70%, totaling 148,589 antibiotic days and $12 million in potential cost savings [34]. This definition of inappropriate therapy is relatively objective and well suited as a standard definition applied across the healthcare spectrum. However, the impact of minimizing this type of inappropriate antimicrobial use on antimicrobial resistant infections is likely minimal.

Most studies assessing appropriateness of antimicrobial therapy have used the more subjective method of expert opinion based definitions of appropriate therapy. These assessments often expand from drug selection and evaluate additional steps in the antimicrobial prescribing pathway (e.g., diagnostic work-up, drug route and duration of therapy); however, methods of evaluation are subjective and definitions used often lack detail, leading to lack of reproducibility outside single centers [5,12,15,17,18,20,23,27,30,31,35]. Despite limitations, expert opinion is commonly used in the literature, often yielding important information about variability in antimicrobial prescribing quality.

Attempts have been made to reduce the subjectivity of this expert review approach. Gyssens and colleagues modified the criteria developed by Kunin to provide detailed explanations (e.g. diagnosis, dose, route, duration, or spectrum) for inappropriateness based on expert review [7,19]. This modification provides a quantifiable and reproducible means to assess appropriateness. Willemsen and colleagues used this algorithm in successive point prevalence surveys and expanded its application to evaluate patients who did not receive antimicrobial therapy, finding only 0.6% of infections requiring treatment were inappropriately not treated [22]. Compliance with local guidelines as the standard for appropriate therapy is increasingly used to reduce subjectivity and provide a reproducible method of measurement for large-scale evaluations across facilities that share similar treatment guidelines.

An approach based on expert review defining appropriateness based on local guidelines and standard coding was used in an evaluation of antimicrobial prescribing in neonatal units in Australia [24]. The European Surveillance of Antimicrobial Consumption (ESAC) project conducted hospital antimicrobial point prevalence surveys across Europe incorporating adherence with guidelines as a measure of appropriate therapy [36]. This approach yielded useful targets for quality improvement such as reducing durations of therapy for surgical prophylaxis [36]. Although still requiring expert chart review, these systematic evaluations created standard reporting mechanisms and robust data allowing for comparison across various healthcare settings, facilitating both local evaluations and national goals of benchmarking judicious antimicrobial use.

In an observational, retrospective cohort study conducted at four sites across the U.S. [37], investigators retrospectively applied four definitions of appropriateness for antimicrobials used for suspected or documented infections with methicillin-resistant Staphylococcus aureus, vancomycin-resistant enterococci and Pseudomonas aeruginosa. Appropriateness ranged from 79% based on expert opinion to 94% based on susceptibility data [37]. Rates of appropriate use based on expert opinion differed significantly from all other definitions, whether evaluating antimicrobial or indication. These data highlight the variability in rates of appropriateness depending on definition applied. Expert opinion introduces subjectivity that although clinically credible, reduces value in comparisons between facilities. Definitions based on antimicrobial susceptibility data and drug information resources likely lack discriminatory power for identifying inappropriate antimicrobial use given inability to account for complex factors requiring clinical judgment. However, ID physicians disagree on appropriate antimicrobial therapy 30% of the time [15]. These findings underscore the difficulty in developing a standard definition of appropriateness that is discriminatory, yet also objective and reproducible. In an attempt to standardize nomenclature, we propose definitions of unnecessary, inappropriate and sub-optimal antimicrobial use applied to days of therapy of particular agents (Table 2). Perhaps the path to the ideal definition of appropriateness lies in establishing the correlation between various definitions and quantitative antimicrobial use data and clinical outcomes, bringing us closer to meaningful quality measure development and antimicrobial benchmarking.

From Definition to Measurement: Development of Audit Tools and Quality Indicators

Antimicrobial point prevalence surveys often incorporate a comprehensive evaluation of prescribing quality based on expert chart review. These audits provide targets for quality improvement initiatives and ongoing antimicrobial stewardship program (ASP) evaluations. Various audit tools have been used to compare antimicrobial prescribing quality within and across institutions, in addition to international evaluations [12,22,24,36,38]. However, a lack of standardization of audit tools and design for primary use by ID physicians or pharmacists limits their applicability. Ideally, audit tools should facilitate large-scale data collection without the need for expert review and permit comparative analysis between hospital types.

The Australian Commission on Safety and Quality in Health Care (ACSQHC) required ASPs be established in all Australian hospitals in 2013 and that antimicrobial prescribing be audited and monitored [28,39]. A multidisciplinary group within Melbourne Health, Victoria, designed an antimicrobial prescribing survey (APS) tool suitable for use in all Australian hospitals [28]. The tool was developed using knowledge gained from prior audit tools, local and national treatment guidelines, expert opinion and clinician feedback; it was designed to be used by auditors of varying expertise ranging from ID physicians and pharmacists to infection control practitioners, nurses and microbiologists. Areas of interest for benchmarking were documentation of antimicrobial indication, compliance with national prescribing guidelines and duration of surgical prophylaxis [28]. Auditors were asked to assess appropriateness based on compliance with published national prescribing guidelines, with use of a multidisciplinary team to assess appropriateness if warranted. In contrast to the U.S., Australian antimicrobial prescribing guidelines are written with consensus from multiple disciplines and the intention of supporting antimicrobial stewardship efforts. Therefore, they provide an agreed upon objective metric that can be collected by auditors of varying expertise. The audit tool was piloted in two successive years and revised based on end user feedback. All participating sites were provided a toolkit and webinar as well as training scenarios on assessing appropriateness. Although the majority of auditors felt the survey had the right amount of detail and committed to future participation, 58% reported staffing issues and lack of expert advice for complex cases were a continued barrier [28].

The Australian National Antimicrobial Prescribing Survey (NAPS) began using the audit tool in 2011. The NAPS helps individual healthcare facilities understand their quantity and quality of antimicrobial prescribing, but also provides information on national antimicrobial prescribing trends. Participation is voluntary and fulfills reporting requirements. A total of 248 hospitals participated in the 2014 NAPS, up 64% from 2013 and participating hospitals represented 44% of all public hospital beds [40]. In 2014, the prevalence of antimicrobial use among inpatients was 38%, with nearly a quarter (23%) considered inappropriate. Prolonged use for surgical prophylaxis and use for respiratory tract infections were deemed of significant concern, with 37–70% of these prescriptions inappropriate [40].

The development of consensus-based antimicrobial prescribing quality indicators (QIs) and audit tools to measure antimicrobial prescribing quality at a national level represent monumental strides in the effort to improve antimicrobial use. Although resource constraints and lack of access to individuals with antimicrobial expertise remain an issue, these tools and surveys demonstrate large-scale assessments of antimicrobial use are possible, and provide the framework for further national and international evaluations.

Development of CDC Antimicrobial Prescribing Audit Tools

To begin identifying key opportunities to improve antibiotic use in U.S. hospitals, the CDC in collaboration with external experts developed audit tools (Appendix A) to assess appropriate antimicrobial use. The tools were intended to be as objective as possible, to highlight situations where providers may be deviating from best practices to identify targets for improvement, and to be useable by auditors with varying clinical expertise.

Four tools were drafted by a working group of three ID physicians, one ID pharmacist and a CDC representative. Development of the tools was based on review of existing treatment guidelines, scientific literature and expert opinion. Two focus on UTI and community-acquired pneumonia (CAP), known key drivers of suboptimal hospital antibiotic use. A third addresses use of agents used to treat resistant Gram-positive organisms. These agents are commonly used in hospitals and present a good target for assessment since therapy can often be stopped on the basis of culture results. The fourth is a general antimicrobial use form, evaluating accepted best practices in antimicrobial prescribing. Each tool includes questions aimed at assessing the steps involved in antimicrobial prescribing involving diagnosis, use of empiric therapy, adjunctive non-antimicrobial therapy, de-escalation, and appropriate duration of therapy. Rather than provide a prescriptive interpretation of appropriateness for each tool, we addressed various steps in antimicrobial prescribing where providers may stray from best practices that can individually or collectively be defined as inappropriate prescribing. The tools are intended to not only identify inappropriate antimicrobial use, but also uncover drivers that are meaningful targets for stewardship intervention.

CDC subsequently convened an in-person meeting of 21 external experts to discuss the proposed tools and provide expert consensus. Those in attendance included adult and pediatric ID physicians, hospitalists, general internists, epidemiologists, surgeons, emergency room physicians and clinical pharmacists representing academia and the private sector and specializing in antimicrobial stewardship. Feedback from the in-person meeting was used to revise the tools and two subsequent modifications were made via electronic communication until final consensus was reached.

The UTI tool was utilized by a citywide antimicrobial stewardship collaborative aimed at evaluating prescribing practices for UTIs in acute care settings. Use of the assessment tool in a single day point prevalence survey across four hospitals facilitated prompt identification of high rates of treatment of ASB. Subsequent implementation of a multicenter educational intervention resulted in the percentage of patients treated for ASB falling from 23% in 2014 to 9% in 2015 (Personal communication with Elizabeth Dodds Ashley). Stewardship pharmacists performed chart reviews, taking 15–45 minutes per case to review depending on complexity. This group now performs an annual audit utilizing the UTI tool, and a manuscript is in progress detailing their findings.

As discussed previously, the tools were a foundation for efforts to assess appropriate use as part of the CDC’s EIP Healthcare Associated Infection and Antimicrobial Use Point Prevalence Survey [1,3], which found prescribing could be improved in 37.2% of cases (39.5% of 111 UTI cases and 35.7% of 185 vancomycin cases). Trained chart reviewers used the tools to collect information for review by a physician at the local health department. While this methodology may not be feasible in some settings, having a reviewer abstract data for clinician review might be practical and sustainable for hospitals with chart reviews done by quality improvement and infection control staff. The next step for the audit tools is a full-scale assessment of appropriate use for 2 infections (UTIs and CAP) and 2 antibiotics (fluoroquinolones and vancomycin). This provides the first opportunity for large-scale assessment of antibiotic use in the U.S. and could potentially be repeated in future years to assess changes in prescribing.

While further study is necessary, we expect with expert consensus on audit tool development and expansion of electronic medical records (EMRs), assessments using the tools will prove valid and reproducible. Additionally, real-world use of the audit tools incorporated into daily ASP reviews could provide a means to perform larger scale assessments of appropriateness, as well as inform future modifications of audit tools and opportunities for automated data extraction.

Next Steps in Assessing Appropriate Antimicrobial Therapy

Currently, manual review of clinical data is the primary way to assess antimicrobial use. Little is known about the association between measurements of appropriateness from point prevalence surveys and longitudinal trends in antimicrobial prescribing. Future research defining this relationship is needed in order to determine if more frequent analyses are necessary, or if intermittent in depth point prevalence surveys provide sufficient information to follow trends in antimicrobial use. The growth and sophistication of EMRs will help advance this endeavor. For example, many of the variables required to assess appropriate use in the CDC audit tools can be directly extracted from some electronic records, which would permit an “electronic audit”. More sophisticated algorithms could identify patients receiving antimicrobials who have no clear evidence of infection (e.g., no fever, normal white blood cell count and negative cultures) whose cases could then be flagged for further review. Such an automated review could greatly facilitate and considerably expand efforts to review antimicrobial prescribing to identify targets for improvements and are being explored by CDC supported efforts.

Conclusion

Many have attempted to measure inappropriate antimicrobial use in various settings with varying definitions applied; however, no established reference standard exists. Most evaluations of prescribing have relied on expert review to define and judge appropriate use. More recently, efforts have focused on defining QIs and methods for measuring antimicrobial prescribing quality based on objective criteria including diagnostic evaluation and adherence of antimicrobial choice and duration with existing literature and treatment guidelines. Some of these qualitative assessment tools have been used in large-scale surveys with success, revealing valuable information necessary to improve antimicrobial use. Evaluation of appropriate antibiotic use using current methods should expand while more advanced, electronic assessments are developed.