http://orcid.org/0000-0003-1339-6465
Figures
Abstract
Introduction
Extensive antibiotic use makes the intensive care unit (ICU) an important focus for antibiotic stewardship programs. The aim of this study was to develop a set of actionable quality indicators for appropriate antibiotic use at ICUs and an implementation toolbox, which can be used to assess and improve the appropriateness of antibiotic use in the treatment of adult patients at an ICU.
Methods
A four round modified-RAND Delphi procedure was used. Potential indicators were identified by a multidisciplinary panel of 15 Dutch experts, from international literature and guidelines. Using an online survey, the identified indicators were rated on three criteria: relevance, actionability and feasibility. Experts discussed and rated the indicators for the second time during a face-to-face consensus meeting. During a final consensus meeting the toolbox was developed, containing potential barriers and improvement strategies which were identified using a validated checklist by Flottorp et al., and if available also containing supporting material.
Results
The first round resulted in 24 potential indicators. After the final meeting a set of three process indicators, one structure indicator and one quantity metric remained: 1) perform at least two sets of blood cultures before start of empirical systemic therapy; 2) perform therapeutic drug monitoring in patients treated with vancomycin or aminoglycosides; 3) perform surveillance cultures if selective digestive or oropharyngeal decontamination is applied at the ICU; 4) biannual face-to-face meetings between ICU and microbiology staff in which local resistance rates are discussed; and 5) quantitative antibiotic use at the ICU expressed in days of therapy (DOT). The toolbox contains 24 unique barriers and 37 improvement strategies.
Citation: Kallen MC, Roos-Blom M-J, Dongelmans DA, Schouten JA, Gude WT, de Jonge E, et al. (2018) Development of actionable quality indicators and an action implementation toolbox for appropriate antibiotic use at intensive care units: A modified-RAND Delphi study. PLoS ONE 13(11): e0207991. https://doi.org/10.1371/journal.pone.0207991
Editor: Valérie Pittet, Institute of Social and Preventive Medicine, SWITZERLAND
Received: November 24, 2017; Accepted: November 10, 2018; Published: November 29, 2018
Copyright: © 2018 Kallen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was supported by the Stichting Kwaliteitsgelden Medisch Specialisten (SKMS, Dutch Foundation for Quality Funding for Medical Specialists), grant number [29762944]. Grant recipient is professor Nicolette de Keizer.
Competing interests: The National Intensive Care Evaluation (NICE) foundation pays the department of Medical Informatics for processing, checking and maintaining the Dutch quality registry. M.J. Roos-Blom and N.F. de Keizer are employees of the department of medical informatics and work for the NICE registry. N.F. de Keizer, D.A. Dongelmans, and E. de Jonge are members of the NICE board. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Abbreviations: AMR, antimicrobial resistance; AB, antibiotic; QI, quality indicator; ICU, Intensive Care Unit; ASP, Antimicrobial Stewardship Program; DOT, days on therapy; SDD, selective digestive decontamination; SOD, selective oropharyngeal decontamination; HER, electronic health record; PDMS, patient data management system
Introduction
Infection is a major cause of morbidity and mortality in critically ill patients, resulting in a high percentage of patients using antibiotics in intensive care units (ICUs), up to 71% [1]. Inappropriate use of antibiotics is the main driving force in the emergence and spread of resistant microorganisms, which makes the intensive care an important focus for antibiotic stewardship programs (ASPs) [2]. ASPs are coordinated programs designed to improve the appropriate use of antibiotics at an institutional level [3]. Integration of ASPs in the ICU is essential in order to pursue optimal antibiotic use in critically ill patients.
A requirement for an effective stewardship program is the ability to measure the appropriateness of antibiotic use. Quality indicators are defined as measurable elements designed to evaluate aspects of quality of care [4]. Previous studies have developed indicators to measure appropriate antibiotic use in hospitalized non-ICU adults. These studies showed large variation in quality of antibiotic use between hospitals and thus considerable room for improvement [5–8]. In addition, studies illustrated that the use of quality indicators improves the appropriateness of antibiotic use and is associated with a decreased length of hospital stay [9, 10]. The Organisation for Economic Cooperation and Development (OECD) and the Agency for Healthcare Research and Quality (AHRQ) have defined criteria for good quality indicators: a good indicator should be relevant, actionable, reliable, show room for improvement and data collection should be feasible [11, 12]. Literature points out that actionability, meaning that the indicator offers clear direction to improve performance in daily practice, specifically contributes to the success of quality improvement [13, 14].
Indicator scores can be used to develop tailored interventions. Tailored interventions are designed to achieve improvements in healthcare, based on the assessment of local barriers in clinical practice [15]. Systematic tailoring of improvement strategies entails three key steps: 1) identification of the barriers of practice (e.g. factors that hinder the performance of recommended appropriate antibiotic use), 2) designing interventions targeted at these barriers, and 3) application and assessment of the effects of the interventions [15, 16]. However, several factors such as time and resource constraints, lack of knowledge on how to improve, and insufficient involvement of staff members hamper health care professionals to develop and execute improvement strategies [14, 17]. Current literature suggests that providing a pre-established list with barriers that might hinder performance on a quality indicator and a list with suggested improvement strategies to overcome these barriers can support health care providers during the process of quality improvement [18].
The aim of this study was to develop a set of actionable quality indicators and an implementation toolbox, which can be used to assess and improve the appropriateness of antibiotic use in the treatment of adult patients at an ICU.
Materials and methods
We performed a modified-RAND Delphi procedure, consisting of four rounds, to develop quality indicators and a toolbox for appropriate antibiotic use in the treatment of critically ill patients at an adult ICU [19–21]. An overview of the RAND-modified Delphi procedure is shown in Fig 1.
Round 1: Identification of potential indicators
First, we performed an inventory of potential indicators among a panel of Dutch experts. The multidisciplinary panel consisted of fifteen experts: three anesthesiologists-intensivists, three internist-intensivists, one intensivist-infectious diseases physician, three internists-ID (infectious diseases) physicians, two clinical microbiologists and three clinical pharmacists. The panelists were invited based on their (inter)national involvement in guideline development, working groups or societies regarding antibiotic use or intensive care (e.g. the National Intensive Care Evaluation (NICE) foundation, the Dutch Working Party on Antibiotic Policy (SWAB) or the European Society of Clinical Microbiology and Infectious Diseases (ESCMID)). They represent nine university and non-university hospitals. The experts were instructed to individually propose potential quality indicators representing appropriate antibiotic use, based on their knowledge and expertise in the field. In addition, we performed a systematic literature search in Medline to identify already available indicators on appropriate antibiotic use. The search included all articles available in Medline up to January 2014. The search was updated in April 2017, which showed no additional indicators relating to antibiotic use at the ICU since 2014. The search strategy is listed in Table 1. Two authors (MK and MJB) independently screened title and abstract in order to identify studies that described (the development of) indicators for appropriate antibiotic use at the ICU. We included all articles that concerned antibiotic use, quality indicators and critical care. Antibiotics were defined as antibacterial agents or antimicrobial agents. Quality indicators were defined as quality measures, performances measures or criteria. Critical care was defined as critical care unit or intensive care unit. Articles were excluded if they were case reports, were not written in English or Dutch, did not concern antibiotic use, critical care, or performance measures, or if they concerned only a specific subgroup of patients. Duplicate studies were removed. We reviewed potentially relevant articles, including references, in full-text format. Third, we made an expert-based selection of national and international guidelines regarding antibiotic use in critically ill patients, from which we extracted all potential indicators (see S1 Appendix). Finally, a list with potential indicators was composed, for which indicators were rephrased if needed and duplicate indicators were removed.
Limits: adults, English, Dutch.
Round 2: Online survey
In the second round we converted the list with potential quality indicators into an online survey and instructed experts to appraise each indicator. Based on a 9-point Likert scale (1 = totally disagree, 9 = totally agree), the experts independently rated the potential indicators on two criteria: 1) relevance, the impact of the indicator on disease or on healthcare expenditure and 2) actionability, the extent to which an indicator offers direction for improvement in clinical practice. Indicators with a median score between 4–9 on both relevance and actionability were defined as potentially suitable and indicators with a median score between 1–4 were defined as not suitable [21]. We selected all potentially suitable indicators for the expert panel meeting (see Table 2).
Round 3: Expert panel consensus meeting
During a face-to-face consensus meeting held on October 1st 2015, we presented the results of the second round to the expert panel. All 15 experts from round 2 were invited, and 10 of them were available at the selected meeting date. The indicators were discussed, rephrased, merged and rated for the second time based on three criteria: 1) relevance, 2) actionability and 3) feasibility of data collection, the ability to use routinely collected electronic data. In addition, the experts appraised each indicator with a validity score, reflecting whether the indicator is associated with and appears to measure quality of antibiotic use in clinical practice (i.e. face and content validity). A blinded survey tool was used to support independent rating. After the second rating all indicators with median scores of 7–9 on relevance, actionability and feasibility without disagreement (i.e. 80% of the assessment rates were within the range of the median scores of 7–9), together with a validity score in the highest quartile of all validity scores, were selected (see Table 2) [21]. The selected indicators were described in detail according to the AIRE instrument, including definitions, in-and exclusion criteria, target values, important case-mix variables and subgroups [4]. If an indicator reflected the degree in which antibiotic use was appropriate and was accompanied by a clear target value, it was defined as a quality indicator. If an indicator reflected the volume of antibiotic use, and its outcome only gained value when comparing it among ICUs, it was defined a quantity metric. Results of the consensus meeting were send to all 15 panel members for their final approval.
Round 4: Action implementation toolbox
Flottorp et al. developed a checklist with all possible barriers for practice and improvement strategies to overcome these barriers, by performing an extensive systematic review of frameworks of determinants of practice followed by a consensus procedure [15]. Based on this validated checklist and expert opinion we defined per selected indicator a list with all possible barriers that can lead to poor performance on that indicator, and a list with improvement strategies to overcome these specific barriers. The list with barriers and improvement strategies was discussed during a second face-to-face meeting. All 15 experts from round 2 were invited, and 5 of them were available at the selected meeting date. The experts adjusted the predefined list and suggested additional barriers and improvement strategies. Again, results of the consensus meeting were send to all 15 panel members for approval and additional suggestions. The final list with barriers and improvement strategies were grouped into four out of the seven predefined categories that are considered relevant in the antibiotic care process [15]. If available, literature or materials, e.g. posters, that could support implementation of the improvement strategies was provided (see S2 Appendix). The barriers, improvement strategies and supporting materials together form “the action implementation toolbox”.
The Medical Ethics Research Committee of the Academic Medical Center confirmed that the Medical Research Involving Human Subjects Acts (WMO) did not apply to this study, since the study subjects were to receive treatment according to standard care and had no burden of the study (August 2016).
Results
Round 1: Identification of potential indicators
The expert panel proposed in the first round 31 unique quality indicators for appropriate antibiotic use, based on their clinical experience. The literature search resulted in 133 scientific articles. Based on predefined exclusion criteria 105 articles were excluded. The remaining 28 articles were selected for full-text screening. Two additional articles were selected from literature references and added to the final list. Thirteen indicators were extracted from these studies. Seventy-eight indicators were extracted from six guidelines regarding antibiotic therapy in critically ill patients (S1 Appendix). After de-duplication and rephrasing, a set of 24 potential indicators remained (Fig 1).
Round 2: Online survey
The panel members rated the 24 potential indicators on their relevance and actionability. All indicators were considered potentially suitable and were therefore selected for the next round (Fig 1 and Table 2).
Round 3: Expert panel consensus meeting
During a three-hours consensus meeting all 24 indicators were discussed. As a result, three indicators were rejected, eleven indicators were rephrased and seven interrelated indicators were merged into three indicators. One indicator was considered a quantity metric rather than a quality indicator, as this metric provides relevant context to the other indicators despite the absence of a target value. Sixteen resulting unique indicators and the quantity metric were rated for the second time. Thirteen indicators and the quantity metric received a median score of 7 or higher on relevance, actionability and feasibility, of which four indicators and the quantity metric received a validity score in the highest quartile (Fig 1, Table 2).
This resulted in a final set of three process indicators, one structure indicator and one quantity metric: 1) perform at least two sets of blood cultures before start of empirical systemic therapy; 2) perform therapeutic drug monitoring in patients treated with vancomycin or aminoglycosides, within 48 hours after start of antibiotic therapy; 3) perform surveillance cultures if selective digestive or oropharyngeal decontamination is applied at the ICU; 4) biannual face-to -face meetings between IC staff and microbiology staff in which local resistance rates and trends in the ICU population are discussed; and 5) quantitative antibiotic use at the ICU expressed in days of therapy (DOT; one day of therapy represents the administration of a single agent on a given day regardless of the number of doses administered or dosage strength) [22]. Targets for indicator 1, 2 and 3 were set at 100%, which is a theoretical optimum, meaning that it is not necessarily realistic but ICUs should aim to achieve this optimum. The target for indicator 4 was based on expert opinion and set at a minimum of two meetings per year. The quantity metric provides context for the indicators and is suitable for benchmarking ICUs. All 15 panel members approved the detailed description of the selected indicators. The final set of quality indicators and quantity metric is presented in Table 3.
Round 4: Action implementation toolbox
The implementation toolbox shows for each indicator a list of potential barriers in the antibiotic care process, and associated improvement strategies to overcome these barriers. Barriers can be applicable to more than one indicators and improvement strategies can be applicable to more than one barrier. A total of 24 unique barriers and 37 unique improvement strategies were identified and grouped into four categories: A) barriers related to the guidelines, e.g. the local antibiotic guidelines are inadequate or incomplete; B) barriers related to the individual health care professional, e.g. health care professionals are not (sufficiently) familiar with the antibiotic protocol; C) barriers related to professional interactions, e.g. there is insufficient communication within ICU teams; D) barriers related to incentives and resources, e.g. there is a lack of culture media bottles at the ICU ward. Supporting materials were provided for eight of the unique improvement strategies (S2 Appendix).
Discussion
In this study we used a modified-RAND Delphi procedure to systematically develop a set of four actionable quality indicators and one quantity metric for appropriate antibiotic use in adult ICUs. In addition, we developed an implementation toolbox, containing a list with possible barriers that lead to poor performance on the selected indicators, and a list with improvement strategies to overcome these specific barriers, which can be used to support stewardship actions to increase performance on antibiotic use.
A quantity metric for total antibiotic use was selected next to the four quality indicators, as the experts found it important to include this metric that enables benchmarking and provides context to the other indicators despite the absence of a specific target value.
Previous studies have described several quality indicators for ICUs [23–26]. De Vos et al. developed a general set of indicators for quality of care at the ICU and used a method comparable to ours, including established criteria in their selection process of indicators [24]. Nevertheless, they did not use the criterion actionability, resulting in a set that did not contain clues for quality improvement in daily practice: the set consisted mainly of structure indicators, indicators lacked clear definitions and target values, and little or no variation in outcome of indicators was seen between different hospitals. Furthermore, none of these indicators were related to antibiotic use in the ICU. Berenholtz et al. also developed quality indicators focusing on care provided in the ICU, but they focused on the whole range of treatment specifically in sepsis patients [26]. A recent study by van den Bosch et al. was most similar to our study, as it developed a set of indicators focusing on antibiotic treatment. Their set consists of outcome and process indicators with clear definitions, however, they too did not take into account the actionability of indicators. Moreover, they focused on patients with sepsis on a general medical ward or ICU, while we specifically aimed to develop a set of indicators for the ICU [23]. We have taken into account the limitations of previous studies and focused on the importance of actionability of indicators, in order to develop ICU-specific indicators that give clear direction to improve quality of care [13, 14].
The use of quality indicators to measure the appropriateness of antibiotic use requires extensive data collection from patient records, which is labor intensive [24]. Since the implementation of the NICE registry database [27], the majority of Dutch ICUs upload their patient level data from their local electronic health records (EHRs) or patient data management system (PDMS) through automatic data extractions, which reduces the workload significantly. Our set of indicators had to be suitable for electronic data extraction, which therefore was an important prerequisite to score high on ‘feasibility of data collection’. We realize that this might be different in other countries or settings.
Not every clinical setting needs the same level of improvement. Improvement strategies should therefore be tailored, depending on the local barriers in clinical practice. [15, 28] Literature shows that systematic tailoring of improvement strategies can improve quality of care [29]. To our knowledge we are the first to provide an implementation toolbox that supports ICU health care providers in the process of tailoring improvement strategies, as they can select those potential barriers from the toolbox that are relevant to their own context, after which the toolbox displays the suggested improvement strategies associated with the selected barriers. We aim to continuously expand and improve the toolbox with new or revised barriers, strategies and supporting materials. The validated indicators, together with the quality improvement toolbox, will be implemented in an online quality dashboard as part of the Dutch National Intensive Care Evaluation (NICE) registry [27] and evaluated in future work. The online dashboard will be offered to the participating ICUs, providing for each ICU their performance scores and benchmark information on the antibiotic indicators (e.g. the median score of all participating ICUs and the average score of the top 10% best performing ICUs) accompanied by a list of potential barriers in the antibiotic care process and the associated improvement strategies to improve the performance on these indicators [30].
Our study has several strengths. First, we used a modified-RAND Delphi approach, a systematic rigorous procedure in which scientific evidence and expert opinions are combined. [5, 6, 9, 19, 20]. Our study was consistent with the validated method, nevertheless, we made a valuable modification and extension: in contrary to previous studies, at the start of the procedure we instructed the experts to propose indicators reflecting appropriate antibiotic use without providing them a predefined list of potential indicators from the literature. By this, we ensured that the clinical experts raised those indicators that were most relevant in daily practice, also when these were not (yet) described in the literature. Second, we used a multidisciplinary expert panel, consisting of different specialties relevant for antibiotic use in the ICU. Third, to our knowledge, we are the first to provide an additional implementation toolbox. This tool enables health care professionals to systematically select tailored improvement strategies based on local barriers.
Our study also has some limitations. First, for pragmatic reasons the literature search was performed only in Medline. However, we used a wide range of search terms regarding quality measurements and antibiotics and we therefore assume that the terms should identify those studies reporting on quality indicators. Second, the Medline search was performed in 2014 when the project started. The search was therefore updated on April 5th 2017, which showed no additional indicators relating to antibiotic use at the ICU since 2014. Third, we did not involve ICU nurses or patients in the expert panel. In the Netherlands antibiotic agents are solely prescribed by doctors. We realize that nurses are increasingly involved in antimicrobial stewardship and should be included in future expert panels. Patients were not involved in our expert team as this was logistically challenging, and also given the heterogeneity of reasons for ICU admission. Fourth, while some of our indicators have been described before and are applicable to general wards as well as to the ICU, they were selected because they were considered particularly relevant to the ICU setting based on literature and expert opinion, because they offered direction for improvement in clinical practice and because data collection was considered feasible. Moreover, indicator 3 and 4 are specific for the ICU setting only and have not been described as quality indicators before. Finally, although our indicators have been developed in a national setting, our method of development was according to current international standards and as the indicators were based on international literature, guidelines and consensus of a multidisciplinary team of experts with international experience in their field, we believe that the resulting indicators are generalizable to other ICU settings. Furthermore, one is able to select indicators tailored to one’s own healthcare setting (e.g. some countries do not apply SDD).
The results of our study are a first step towards the use of a new set of actionable quality indicators, together with an action implementation toolbox, to monitor and improve the appropriateness of antibiotic use in adult ICUs. The toolbox will support professionals in selecting improvement strategies that work best in their setting, based on careful assessment of local barriers. Clinimetric properties of the indicators and feasibility in daily practice of electronic data reuse from the EHR or PDMS will be tested during an evaluation study using time series analysis.
Supporting information
S1 Appendix. Literature and guidelines selected in the RAND modified Delphi procedure.
https://doi.org/10.1371/journal.pone.0207991.s001
(DOCX)
S2 Appendix. Quality improvement toolbox containing possible barriers and subsequent improvement actions on each quality indicator.
https://doi.org/10.1371/journal.pone.0207991.s002
(DOCX)
Acknowledgments
The authors would like to thank the panel members Dylan de Lange, Peter Pickkers, Rob Bosman, Evelien de Jong, Dennis Bergmans, Erik van Maarseveen, Roger Brüggemann, Edwin Boel, Patrick Sturm, Tom Sprong and Sander Kouwen for their contribution.
References
- 1. Vincent JL, Rello J, Marshall J, Silva E, Anzueto A, Martin CD, et al. International study of the prevalence and outcomes of infection in intensive care units. Jama. 2009;302(21):2323–9. Epub 2009/12/03. pmid:19952319.
- 2.
Centers for Disease Control and Prevention. Report: A public health action plan to combat antimicrobial resistance. Available from: http://www.cdc.gov/drugresistance/pdf/public-health-action-plan-combat-antimicrobial-resistance.pdf. 2011.
- 3. Bartlett JG. A call to arms: the imperative for antimicrobial stewardship. Clin Infect Dis. 2011;53 Suppl 1:S4–7. Epub 2011/08/04. pmid:21795727.
- 4.
de Koning J, Smulders A, Klazinga NS. Appraisal of Indicators through Research and Evaluation (AIRE), Academisch Medisch Centrum Universiteit van Amsterdam, afdeling Sociale Geneeskunde, Amsterdam. 2007.
- 5. Schouten JA, Hulscher ME, Wollersheim H, Braspennning J, Kullberg BJ, van der Meer JW, et al. Quality of antibiotic use for lower respiratory tract infections at hospitals: (how) can we measure it? Clin Infect Dis. 2005;41:450–60. Epub 2005/07/20. pmid:16028151.
- 6. Hermanides HS, Hulscher ME, Schouten JA, Prins JM, Geerlings SE. Development of quality indicators for the antibiotic treatment of complicated urinary tract infections: a first step to measure and improve care. Clin Infect Dis. 2008;46:703–11. Epub 2008/01/31. pmid:18230045.
- 7. Hulscher ME, Grol RP, van der Meer JW. Antibiotic prescribing in hospitals: a social and behavioural scientific approach. Lancet Infect Dis. 2010;10(3):167–75. Epub 2010/02/27. pmid:20185095.
- 8. van den Bosch CM, Hulscher ME, Natsch S, Wille J, Prins JM, Geerlings SE. Applicability of generic quality indicators for appropriate antibiotic use in daily hospital practice: a cross-sectional point-prevalence multicenter study. Clin Microbiol Infect. 2016;22(10):888.e1–.e9. Epub 2016/07/20. pmid:27432770.
- 9. van den Bosch CM, Geerlings SE, Natsch S, Prins JM, Hulscher MEJL. Quality indicators to measure appropriate antibiotic use in hospitalized adults. Clin Infect Dis. 2015;60:281–91. Epub 2014/10/01. pmid:25266285.
- 10. Spoorenberg V, Hulscher ME, Akkermans RP, Prins JM, Geerlings SE. Appropriate antibiotic use for patients with urinary tract infections reduces length of hospital stay. Clin Infect Dis. 2014;58:164–9. Epub 2013/10/26. pmid:24158412.
- 11.
Kelley E, Hurst J. Health Care Quality Indicators Project: OECD Publishing. 2006.
- 12.
Center for Health Policy/Center for Primary Care and Outcomes Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement. Rockville, MD: Agency for Healthcare Research and Quality. 2011.
- 13. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;(6):Cd000259. Epub 2012/06/15. pmid:22696318.
- 14. de Vos Maartje LG, van der Veer SN, Graafmans WC, de Keizer NF, Jager KJ, Westert GP, et al. Process evaluation of a tailored multifaceted feedback program to improve the quality of intensive care by using quality indicators. BMJ Qual Saf. 2013;22(3):233–41. Epub 2013/01/31. pmid:23362504.
- 15. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35. Epub 2013/03/26. pmid:23522377; PubMed Central PMCID: PMCPmc3617095.
- 16. Wensing M, Oxman A, Baker R, Godycki-Cwirko M, Flottorp S, Szecsenyi J, et al. Tailored Implementation For Chronic Diseases (TICD): a project protocol. Implement Sci. 2011;6:103. pmid:21899753; PubMed Central PMCID: PMCPMC3179734.
- 17. Gude WT, van Engen-Verheul MM, van der Veer SN, Kemps HM, Jaspers MW, de Keizer NF, et al. Effect of a web-based audit and feedback intervention with outreach visits on the clinical performance of multidisciplinary teams: a cluster-randomized trial in cardiac rehabilitation. Implement Sci. 2016;11(1):160. pmid:27938405; PubMed Central PMCID: PMCPMC5148845.
- 18. Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, et al. Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness. Annals of internal medicine. 2016;164(6):435–41. pmid:26903136.
- 19. Van Engen-Verheul M, Kemps H, Kraaijenhagen R, De Keizer N, Peek N. Modified Rand method to derive quality indicators: a case study in cardiac rehabilitation. Studies in health technology and informatics. 2011;169:88–92. pmid:21893720.
- 20. Kotter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators—a systematic review. Implement Sci. 2012;7:21. Epub 2012/03/23. pmid:22436067; PubMed Central PMCID: PMCPmc3368783.
- 21. Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lázaro P, et al. The Rand/UCLA appropriateness method user’s manual. 2001.
- 22. Polk RE, Fox C, Mahoney A, Letcavage J, MacDougall C. Measurement of adult antibacterial drug use in 130 US hospitals: comparison of defined daily dose and days of therapy. Clin Infect Dis. 2007;44(5):664–70. Epub 2007/02/06. pmid:17278056.
- 23. van den Bosch CM, Hulscher ME, Natsch S, Gyssens IC, Prins JM, Geerlings SE. Development of quality indicators for antimicrobial treatment in adults with sepsis. BMC Infect Dis. 2014;14:345. Epub 2014/06/22. pmid:24950718; PubMed Central PMCID: PMCPmc4078010.
- 24. de Vos M, Graafmans W, Keesman E, Westert G, van der Voort PH. Quality measurement at intensive care units: which indicators should we use? J Crit Care. 2007;22(4):267–74. Epub 2007/12/19. pmid:18086396.
- 25. Nguyen HB, Corbett SW, Steele R, Banta J, Clark RT, Hayes SR, et al. Implementation of a bundle of quality indicators for the early management of severe sepsis and septic shock is associated with decreased mortality. Crit Care Med. 2007;35(4):1105–12. pmid:17334251.
- 26. Berenholtz SM, Pronovost PJ, Ngo K, Barie PS, Hitt J, Kuti JL, et al. Developing quality measures for sepsis care in the ICU. Joint Commission journal on quality and patient safety. 2007;33(9):559–68. pmid:17915530.
- 27. van de Klundert N, Holman R, Dongelmans DA, de Keizer NF. Data Resource Profile: the Dutch National Intensive Care Evaluation (NICE) Registry of Admissions to Adult Intensive Care Units. Int J Epidemiol. 2015;44(6):1850–h. Epub 2015/11/29. pmid:26613713.
- 28. Hulscher M, Prins JM. Antibiotic stewardship: does it work in hospital practice? A review of the evidence base. Clin Microbiol Infect. 2017;23(11):799–805. pmid:28750920.
- 29. Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010;(3):CD005470. pmid:20238340; PubMed Central PMCID: PMCPMC4164371.
- 30. Gude WT, Roos-Blom MJ, van der Veer SN, de Jonge E, Peek N, Dongelmans DA, et al. Electronic audit and feedback intervention with action implementation toolbox to improve pain management in intensive care: protocol for a laboratory experiment and cluster randomised trial. Implement Sci. 2017;12(1):68. pmid:28545535.