No magic bullets: a systematic review of 102 trials of interventions
to improve professional practice
Andrew D. Oxman, MD, MSc; Mary Ann Thomson, BHSc(PT); David A. Davis, MD; R. Brian Haynes, MD, PhD
Canadian Medical Association Journal 1995; 153: 1423-1431
This article is the fourth in a series of six that appeared in the October, November and December issues of CMAJ.
Dr. Oxman (principal investigator) is a member of the Health Services Research Unit, National Institute of Public Health, Oslo, Norway; Ms. Thomson is with the School of Occupational and Physical Therapy, Faculty of Health Sciences, McMaster University, Hamilton, Ont.; Dr. Davis is in the Office of Continuing Education and the Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, Toronto, Ont.; and Dr. Haynes is in the departments of Clinical Epidemiology and Biostatistics and of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ont.
Paper reprints of the full text may be obtained from: Dr. Andrew D. Oxman, Health Services Research Unit, National Institute of Public Health, Geitmyrsveien 75, 0462 Oslo, Norway
Objective: To determine the effectiveness of different types of interventions in improving health professional performance and health outcomes.
Data sources: MEDLINE, SCISEARCH, CINAHL and the Research and Development Resource Base in CME were searched for trials of educational interventions in the health care professions published between 1970 and 1993 inclusive.
Study selection: Studies were selected if they provided objective measurements of health professional performance or health outcomes and employed random or quasi-random allocation methods in their study designs to assign individual subjects or groups. Interventions included such activities as conferences, outreach visits, the use of local opinion leaders, audit and feedback, and reminder systems.
Data extraction: Details extracted from the studies included the study design; the unit of allocation (e.g., patient, provider, practice, hospital); the characteristics of the targeted health care professionals, educational interventions and patients (when appropriate); and the main outcome measure.
Data synthesis: The inclusion criteria were met by 102 trials. Areas of behaviour change included general patient management, preventive services, prescribing practices, treatment of specific conditions such as hypertension or diabetes, and diagnostic service or hospital utilization. Dissemination-only strategies, such as conferences or the mailing of unsolicited materials, demonstrated little or no changes in health professional behaviour or health outcome when used alone. More complex interventions, such as the use of outreach visits or local opinion leaders, ranged from ineffective to highly effective but were most often moderately effective (resulting in reductions of 20% to 50% in the incidence of inappropriate performance).
Conclusion: There are no "magic bullets" for improving the quality of health care, but there are a wide range of interventions available that, if used appropriately, could lead to important improvements in professional practice and patient outcomes.
Objectif : Déterminer l'efficacité de différents types d'intervention destinés à améliorer le rendement des professionnels de la santé et à accroître les résultats sur la santé.
Source de données : On a cherché dans MEDLINE, SCISEARCH, CINAHL et la Base de ressources en recherche et développement sur EMC des tentatives d'interventions d'éducation auprès des professions de la santé, publiées entre 1970 et 1993 inclusivement.
Sélection des études : On a retenu les études qui présentaient une mesure objective du rendement des professionnels de la santé ou des résultats sur la santé et dont le plan d'étude recourait à des méthodes aléatoires ou quasi aléatoires de répartition des sujets individuels ou des groupes. Les interventions comprenaient diverses activités comme des conférences, des visites de liaison, le recours à des personnalités locales, des vérifications, la rétroaction et les systèmes de rappel.
Extraction des données : Parmi les détails extraits des études, mentionnons le plan d'étude, l'unité d'attribution (comme le patient, le fournisseur, la pratique ou l'hôpital), les caractéristiques des professionnels de la santé visés, des interventions d'éducation et des patients (le cas échéant), et la principale mesure de résultats.
Synthèse des données : Les critères d'inclusion sont ressortis dans 102 études. Parmi les secteurs où s'est produit un changement de comportement, notons les relations générales avec les patients, les services de prévention, les pratiques de prescription, le traitement de certaines maladies comme l'hypertension ou le diabète, et le service diagnostic ou le recours à l'hôpital. Utilisées comme critères uniques, les stratégies axées uniquement sur la diffusion, comme les conférences ou l'envoi par la poste de documents non sollicités, n'ont suscité que très peu de changements en ce qui touche le comportement des professionnels de la santé ou les résultats sur la santé. Les interventions plus complexes, comme des visites de liaison ou le recours à des personnalités locales, ont donné des résultats allant de pauvres à excellents, mais, dans la plupart des cas, elles n'ont eu qu'un effet modéré (diminution de 20 % à 50 % de l'incidence d'une performance non appropriée).
Conclusion : Il n'y a aucune «formule magique» pour accroître la qualité des soins de santé, cependant il
existe un vaste éventail d'interventions disponibles qui, si on les utilise comme il faut, pourraient donner lieu à d'importantes améliorations de la pratique des professionnels et des résultats au niveau des patients.
[Table of Contents]
Although the importance of research evidence in guiding clinical decision making is generally unquestioned intellectually, at a practical level patterns of medical practice often diverge from evidence-based recommendations, robbing patients of the benefits of medical research.(1-14) For example, Williamson and associates(14) found that one fifth to one half of primary care practitioners in the United States were not aware of or were not using new evidence about six common procedures. Similarly, in a systematic review of randomized controlled trials Lau and collaborators(15) found overwhelming evidence for the use of thrombolytic therapy in patients who had suffered myocardial infarction; however, these findings were often not reflected in the recommendations by experts in textbooks and review articles(16) or in practice.(17) These and other studies reflect shortcomings in the management of health science information by physicians and other health care professionals as well as other constraints on the use of research evidence.(18-20)
In 1992 Davis and colleagues(21) systematically reviewed 50 randomized controlled trials of the effectiveness of continuing medical education (CME), which they broadly defined as all the ways by which physician learning and clinical practice might be altered by educational or persuasive means. The objective of our study was to update and enhance that body of knowledge by (a) including nonphysician health care professionals, (b) applying explicit criteria for assessing study quality, (c) defining specific types of interventions, described later, (d) analysing other published systematic reviews of interventions targeted at a specific problem (e.g., vaccination delivery(22)) or provider (e.g., nurses(23)), or of a specific type of intervention (e.g., audit and feedback(24)), and (e) applying theoretic frameworks such as total quality management,(25) the PRECEDE (predisposing, reinforcing and enabling constructs in educational diagnosis and evaluation) health behaviour model26 and various social science and adult educational models27 to clarify the findings of trials. We present preliminary data resulting from these processes.
[Table of Contents]
We added to the search strategies described by Davis and coworkers(21) by searching the following computerized bibliographic databases for the period January 1990 to December 1993: MEDLINE, SCISEARCH, CINAHL and the Research and Development Resource Base in Continuing Medical Education.(28)
We included trials that met the following criteria: the studies involved health care providers (excluding students); outcomes were measured by objective assessments of provider performance in a health care setting or by any health outcome; and the study designs were such that individual subjects or groups were assigned to one of two or more alternative interventions or to a control group by means of random allocation or some quasi-random method of allocation (e.g., alternation). We included the following types of intervention:
The inclusion criteria were subject to a pilot study by all of us on a sample of five studies. Two of us (A.D.O. and M.A.T.) then independently applied the criteria to all potentially relevant studies for all 10 types of intervention. Disagreements were resolved by consensus.
- Educational materials: Distribution of published or printed recommendations for clinical care, including clinical practice guidelines, audiovisual materials and electronic publications.
- Conferences: Participation of health care providers in conferences, lectures, workshops or traineeships outside their practice settings.
- Outreach visits: Use of a trained person who meets with providers in their practice settings to provide information. The information given may include feedback on the provider's performance.
- Local opinion leaders: Use of providers explicitly nominated by their colleagues to be "educationally influential."
- Patient-mediated interventions: Any intervention aimed at changing the performance of health care providers for which information was sought from or given directly to patients by others (e.g., direct mailings to patients, patient counselling delivered by others, or clinical information collected directly from patients and given to the provider).
- Audit and feedback: Any summary of clinical performance of health care over a specified period, with or without recommendations for clinical action. The information may have been obtained from medical records, computerized databases or patients or by observation.
- Reminders: Any intervention (manual or computerized) that prompts the health care provider to perform a clinical action. Examples include concurrent or inter-visit reminders to professionals about desired actions such as screening or other preventive services, enhanced laboratory reports or administrative support (e.g., follow-up appointment systems or stickers on charts).
- Marketing: Use of personal interviewing, group discussion (focus groups) or a survey of targeted providers to identify barriers to change and the subsequent design of an intervention.
- Multifaceted interventions: Any intervention that includes two or more of the last six interventions described here.
- Local consensus processes: Inclusion of participating providers in discussion to ensure agreement that the chosen clinical problem is important and the approach to managing it appropriate.
Assessment of trial quality
The following criteria were used to assess the quality of the trials: concealment of random allocation (protection against selection bias); completeness of follow-up; blind outcome assessment; reliable outcome, indicated by two or more raters with at least 90% agreement or achievement of a kappa value greater than 0.8; baseline measurement of performance or health outcomes measured before intervention; and assurance that contamination was unlikely (i.e., that the control group did not receive the experimental interventions). All four of us tested these on a sample of 10 studies, and then one of us (M.A.T.) applied them to each study included in the review.
The details extracted from the studies included, but were not limited to, the following: whether the study was a randomized controlled trial (parallel groups), a quasi-randomized controlled trial such as one using alternative allocation, or a crossover study; the unit of allocation (e.g., patient, provider, practice, hospital); the characteristics of the participating providers and patients; the characteristics of the intervention, including the content, format, source, recipient, setting and timing; and a description of the main outcome measure.
We attempted to obtain sufficient information to calculate the mean proportion (per provider) of times that performance was appropriate for each patient or episode of care in each study group. We also tried to obtain the proportion of baseline performance that was adequate and the absolute and relative percent change in performance for each study group.
[Table of Contents]
We identified 102 trials(29-130) investigating one or more interventions targeted at improving the performance of health care professionals. The number of trials assessing each type of intervention is shown in Table 1. The comparison groups by type of intervention are shown in Table 2. Table 3 lists the main type of behaviour targeted in each study.
Effectiveness of interventions
- Educational materials: Most of the studies that used printed materials only(29,30) failed to demonstrate changes in performance or health outcome, a finding that has been associated also with the distribution of guidelines.(31) Conversely, Marton, Tul and Sox(32) found printed materials to be as effective as audit and feedback in altering test-ordering behaviour.
- Conferences: Reflecting the dissemination-only characteristics of unsolicited printed materials, those conferences, rounds and workshops during which no explicit effort was made to determine practice needs or to facilitate practice change failed to demonstrate change in performance or health outcome. However, other, more comprehensive strategies employing workshops as a central focus did effect changes through intra-session practice rehearsal(34) or other patient educational and practice-reinforcing strategies.(35)
- Outreach visits: Outreach visits, also known as academic detailing or counter detailing, were effective in reducing inappropriate prescribing and, to a lesser extent, increasing the delivery of preventive services. Reductions of 12% to 49% in inappropriate prescriptions as a result of counter detailing were reported in four studies.(29,36-38) Second, Cummings and coworkers(39) reported the effectiveness of detailing in smoking cessation. Dietrich and associates(40) reported small increases (5% to 27%) in the delivery of 10 preventive services, 6 of which were statistically significant (p * 0.05). Finally, Putnam and Curry(41) demonstrated a moderate, statistically significant effect of outreach visits combined with audit on essential elements in the management of five common conditions in family practice.
- Local opinion leaders: The effectiveness of opinion leaders ranged from nonsignificant to substantial. Stross and collaborators(42-44) found some improvement in the quality of care of patients with arthritis and respiratory disease. Lomas and colleagues(31) demonstrated a substantial increase in the number of trials of vaginal delivery after previous cesarean section in hospitals in which a local opinion leader was used.
- Patient-mediated interventions: These included patient educational materials used widely in the literature for smoking cessation(47,48) and diabetes mellitus.(49) Changes in performance were demonstrated in multifaceted interventions when patient-mediated interventions were combined with other interventions, such as academic detailing. Patient education effected statistically significant improvements in the management of diabetes mellitus that were probably clinically important.(49) These effects were enhanced when patient education was combined with physician education. Other patient-mediated strategies, which made use of information derived from patients through questionnaires or interviews, demonstrated mixed results.(50,51)
- Audit, feedback and reminders: The effectiveness of audit and feedback or of reminders across different types of clinical behaviour ranged from nil to moderate. The types of participants and the characteristics of the interventions and targeted behaviours varied across the studies. For example, among the trials of reminders 30 (58%) used computerized prompting, 14 (27%) relied on administrative assistance, and 8 (15%) used simple stickers or chart inserts. Using the audit/feedback strategy one study reported an absolute increase of over 40% in the median weighted rate of prescriptions for generic drugs.(45) Although another study achieved only a small change (6.7%) in the mean cost of prescriptions through this strategy, the ratio of savings to costs of the intervention was 50:1.
- Marketing: Studies have demonstrated that learning experiences based on objective practice-needs assessment or knowledge testing can alter at least some aspects of physician performance.(52,53) The evidence is less clear, however, when more subjective needs or practice assessments are performed.(54) Market research was an important component of outreach visits targeted at reducing the incidence of inappropriate prescribing, but it is not possible to separate out the effect of this component of counter detailing.
- Multifaceted interventions and local consensus processes: The use of a variety of interventions, such as audit and feedback, reminders, outreach visits, patient-mediated interventions or opinion leaders, has demonstrated changes in professional performance and, less consistently, changes in health outcomes. The importance of local consensus processes is not clear. For example, Putnam and Curry(41) demonstrated a moderate effect on performance of a local consensus process for generating criteria for optimal care but did not find a significant effect of such a process for essential care.
[Table of Contents]
Interventions to improve professional performance are complex, and any cogent interpretation of the results of these trials requires a disentangling of the variation in the characteristics of the targeted professionals, the interventions studied, the targeted behaviours and the study designs. Because of these variations and inadequacies in reporting, it is not possible to compare or make definitive conclusions about the effects of specific types of intervention, or to measure the size of the effects. Nevertheless, several interventions have been found to improve provider performance and, to a lesser degree, health outcomes. At the same time, some common forms of continuing education have either not been tested (e.g., journal reading) or have been tested and found wanting (e.g., unsolicited mailings).
It is helpful to draw an analogy between trials of interventions to improve the performance of health care professionals and drug trials. There are (arguably) no wonder drugs; often several medications are needed, along with lifestyle or environmental changes, to effect clinically important changes in health status. It is the same with the alteration of health professional performance: many interventions have modest or negligible practical effects when used alone. However, when coupled with other strategies the effects may be cumulative and significant.
Furthermore, just as there is a need for rational drug prescribing, there is a need in the area of health professional performance to include appropriate diagnostic strategies (to determine the reasons for suboptimal performance and to identify barriers to change) and to select carefully the interventions most likely to be effective in light of the diagnosed problem. For example, reminders are likely to be effective only if not having the right information at the right time is an important cause of suboptimal performance. These diagnostic strategies bear a direct relation to quality-assurance activities. Closer collaboration of researchers in the areas of health professional performance, health services and quality assurance appears to be both desirable and necessary.(21) Future efforts should aim to improve and standardize reporting of trials of interventions to improve professional performance. A broad framework should be developed for designing and selecting appropriate interventions across the wide range of professional activities in which gaps between evidence and practice are found. An important initiative in this direction is the formation of the Cochrane Collaboration on Effective Professional Practice, which aims to prepare and keep up-to-date systematic reviews of the effects of interventions to improve professional practice.(131)
There are no "magic bullets" for improving the quality of health care. There are, however, a wide range of interventions available that, if used appropriately, could lead to substantial improvements in clinical care derived from the best available evidence.
We thank the following for their financial support: the North Thames Regional Health Authority Research and Development Programme, National Health Service, London, England; the UK Cochrane Centre, Oxford, England; and Health Canada, Ottawa.
[Table of Contents]
- Sherrington AM: An annotated bibliography of studies on the flow of medical information to practitioners. Methods Inf Med 1966; 2 (5): 45-57
- Chalmers TC: The impact of controlled trials on the practice of medicine. Mount Sinai J Med 1974; 41: 753-759
- Stross JK, Harlan WR: The dissemination of new medical information. JAMA 1979; 241: 2622-2624
- Goodwin JS, Foodwin JM: Failure to recognize efficacious treatments: a history of salicylate therapy in rheumatoid arthritis. Perspect Biol Med 1981; 25: 78-92
- Schade RR, Donaldson RM: How physicians use cimetidine: a survey of hospitalized patients and published cases. N Engl J Med 1981; 304: 1281-1284
- Rockette HE, Redmond CK, Fisher B: Impact of randomized clinical trials on therapy of primary breast cancer: the NSABP overview. Control Clin Trials 1982; 3: 209-225
- Garnier HS, Flamant R, Fohanno C: Assessment of the role of randomized clinical trials in establishing treatment policies. Control Clin Trials 1982; 3: 227-234
- Fisher LD, Kennedy JW: Randomized surgical clinical trials for treatment of coronary artery disease. Control Clin Trials 1982; 3: 335-338
- Friedman L, Wenger NK, Knatterude GL: Impact of the coronary drug project findings on clinical practice. Control Clin Trials 1983; 4: 513-522
- Fineberg HV: Effects of clinical evaluation on the diffusion of medical technology. In Committee for Evaluating Medical Technologies in Clinical Use: Assessing Medical Technologies, National Academy Press, Washington, 1985: 176-210
- Lewis C: Disease prevention and health promotion practices of primary care physicians in the United States. Am J Prev Med 1988; 4: 9-16
- Bass MJ, Elford WE: Preventive practice patterns of Canadian primary care physicians. Am J Prev Med 1988; 4: 17-23
- Stross JK: Relationships between knowledge and experience in the use of disease-modifying antirheumatic agents: a study of primary care practitioners. JAMA 1989; 262: 2721-2723
- Williamson JW, German PS, Weiss R et al: Health science information management and continuing education of physicians: a survey of US primary care practitioners and their opinion leaders. Ann Intern Med 1989; 110: 151-160
- Lau JL, Antman EM, Jimenez-Silva J et al: Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med 1992; 327: 248-254
- Antman EM, Lau J, Kupelnick B et al: A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. JAMA 1992; 268: 240-248
- Ketley D, Woods KL: Impact of clinical trials on clinical practice: example of thrombolysis for acute myocardial infarction. Lancet 1993; 342: 891-894
- Lomas J, Haynes RB: A taxonomy and critical review of tested strategies for the application of clinical practice recommendations: from official to individual clinical policy. Am J Prev Med 1988; 4 (4 suppl): S77-S94
- Eisenberg JM: Doctors' Decisions and the Cost of Medical Care, Health Administration Press Perspective, Ann Arbor, Mich, 1986
- Eddy DM: Clinical policies and the quality of clinical practice. N Engl J Med 1982; 307: 343-347
- Davis DA, Thomson MA, Oxman AD et al: Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA 1992; 268: 1111-1117
- Gyorkos TW, Tannenbaum TN, Abrahamowicz A et al: Evaluation of the effectiveness of immunization delivery methods. Can J Public Health 1994; 85 (suppl 1): S37-S40
- Waddell DL: The effects of continuing education on nursing practice: a meta-analysis. J Contin Educ Nurs 1991; 22: 113-118
- Mugford M, Banfield P, O'Hanlon M: Effects of feedback of information on clinical practice: a review. [review] BMJ 1991; 303: 398-402
- Murrey KO, Gottlieb LK, Schoenbaum SC: Implementing clinical guidelines: a quality management approach to reminder systems. [review] QRB Qual Rev Bull 1992; 18: 423-433
- Green LW, Eriksen MP, Schor EL: Preventive practices by physicians: behavioral determinants and potential interventions. [review] Am J Prev Med 1988; 4 (4 suppl): S101-S107
- Lomas J: Teaching Old (and Not So Old) Docs New Tricks: Effective Ways to Implement Research Findings. No 93-4 of CHEPA Working Paper Series, McMaster University, Hamilton, Ont, 1993
- Davis DA, Fox RD (eds): The Physician as Learner: Linking Research to Practice, American Medical Association Press, Chicago, 1994: xiii
- Avorn J, Soumerai SB: Improving drug-therapy decisions through educational outreach. A randomized controlled trial of academically based detailing. N Engl J Med 1983; 308: 1457-1463
- Evans CE, Haynes RB, Birkett NJ et al: Does a mailed continuing education program improve physician performance? Results of a randomized trial in antihypertensive care. JAMA 1986; 255: 501-504
- Lomas J, Enkin M, Anderson GM et al: Opinion leaders versus audit and feedback to implement practice guidelines. JAMA 1991; 265: 2202-2207
- Marton KI, Tul V, Sox HC: Modifying test-ordering behaviour in the outpatient medical clinic. Arch Intern Med 1985; 145: 816-821
- Meyer TJ, Van Kooten D, March S et al: Reduction of polypharmacy by feedback to clinicians. J Gen Intern Med 1991; 6: 133-136
- Maiman LA, Becker MH, Liptak GS et al: Improving pediatricians' compliance-enhancing practices: a randomized trial. Am J Dis Child 1988; 142: 773-779
- Wilson DM, Taylor W, Gilbert JR et al: A randomized trial of a family physician intervention for smoking cessation. JAMA 1988; 260: 1570-1574
- McConnell TS, Cushing AH, Bankhurst AD et al: Physician behaviour modification using claims data: tetracycline for upper respiratory tract infection. West J Med 1982; 137: 448-450
- Avorn J, Soumerai SB, Everitt DE et al: A randomized trial of a program to reduce the use of psychoactive drugs in nursing homes. N Engl J Med 1992; 327: 168-173
- Soumerai SB, Salem-Schatz S, Avorn J et al: A controlled trial of educational outreach to improve blood transfusion practice. JAMA 1993; 270: 961-966
- Cummings SR, Coates TJ, Richard RJ et al: Training physicians in counselling about smoking cessation, a randomized trial of the "quit for life" program. Ann Intern Med 1989; 110: 640-647
- Dietrich AJ, O'Connor GT, Keller A et al: Cancer: improving early detection and prevention; a community practice randomised trial. BMJ 1992; 304: 687-691
- Putnam RW, Curry L: Impact of patient care appraisal on physician behaviour in the office setting. CMAJ 1985; 132: 1025-1029
- Stross JK, Hiss RG, Watts CM et al: Continuing education in pulmonary disease for primary care physicians. Am Rev Respir Dis 1983; 127: 739-746
- Stross JK, Bole GG: Evaluation of an educational program for primary care practitioners on the management of osteoarthritis. Arthritis Rheum 1985; 28: 108-111
- Stross JK, Banwell BF, Wolf FM et al: Evaluation of an education program on the management of rheumatic diseases for physical therapists. J Rheumatol 1986; 13: 374-378
- Gehlbach SH, Wilkinson WE, Hammond WE et al: Improving drug prescribing in a primary care practice. Med Care 1984; 22: 193-201
- Hershey CO, Porter DK, Breslau D et al: Influence of simple computerized feedback on prescription charges in an ambulatory clinic. Med Care 1986; 24: 472-481
- Cohen SJ, Stookey GK, Katz BP et al: Encouraging primary care physicians to help smokers quit. Ann Intern Med 1989; 110: 648-652
- Cummings SR, Richard RJ, Duncan CL et al: Training physicians about smoking cessation: a controlled trial in private practices. J Gen Intern Med 1989; 4: 482-489
- Vinicor F, Cohen SJ, Mazzuca SA et al: DIABEDS: A randomized trial of the effects of physician and/or patient education on diabetes patient outcomes. J Chronic Dis 1987; 40: 345-356
- Magruder-Habib K, Zung WK, Feussner JR: Improving physicians' recognition and treatment of depression in general medical care. Med Care 1990; 28: 239-250
- Rubenstein LV, Calkins DR, Young RT et al: Improving patient function: a randomized trial of functional disability screening. Ann Intern Med 1989; 111: 836-842
- Jennett PA, Laxdal OE, Hayton RC et al: The effects of continuing medical education on family doctor performance in office practice: a randomized control study. Med Educ 1988; 22: 139-145
- White CW, Albanese MA, Brown DD et al: The effectiveness of continuing medical education in changing the behaviour of physicians caring for patients with acute myocardial infarction. Ann Intern Med 1985; 102: 689-692
- Putnam RW, Curry L: Physicians' participation in establishing criteria for hypertension management in the office: Will patient outcomes be improved? CMAJ 1989; 140: 806-809
- Mazzuca SA, Vinicor F, Einterz RM et al: Effects of the clinical environment on physicians' response to postgraduate medical education. Am Educ Res J 1990; 27: 473-478
- Adams ID, Chan M, Clifford PC et al: Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ 1986; 293: 800-804
- Barnett GO, Winickoff RN, Morgan MM et al: A computer-based monitoring system for follow-up of elevated blood pressure. Med Care 1983; 21: 400-409
- Bass MJ, McWhinney IR, Donner A: Do family physicians need medical assistants to detect and manage hypertension? CMAJ 1986; 134: 1247-1255
- Berwick DM, Coltin KL: Feedback reduces test use in a health maintenance organization. JAMA 1986; 255: 1450-1454
- Billi JE, Hejna GF, Wolf FM et al: The effects of a cost-
education program on hospital charges. J Gen Intern Med 1987; 306-311
- Britton ML, Lurvey PL: Impact of medication profile review on prescribing in a general medicine clinic. Am J Hosp Pharm 1991; 48: 265-270
- Bulpitt CJ, Beilin LJ, Coles EC et al: Randomised controlled trial of computer-held medical records in hypertensive patients. BMJ 1976; 1: 677-679
- Buntinx F, Knottnerus JA, Crebolder HFJM et al: Reactions of doctors to various forms of feedback designed to improve the sampling quality of cervical smears. Qual Assur Health Care 1992; 4: 161-166
- Buntinx F, Knottnerus JA, Crebolder HFJM et al: Does feedback improve the quality of cervical smears? A randomized controlled trial. Br J Gen Pract 1993; 43: 194-198
- Chambers CV, Balaban DJ, Carlson BL et al: Microcomputer-generated reminders, improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract 1989; 29: 273-280
- Chase CR, Vacek PM, Shinozaki T et al: Medical information management: improving the transfer of research results to presurgical evaluation. Med Care 1983; 21: 410-424
- Chassin MR, McCue SM: A randomized trial of medical quality assurance. JAMA 1986; 256: 1012-1016
- Cheney C, Ramsdell JW: Effect of medical records' checklists on implementation of periodic health measures. Am J Med 1987; 83: 129-136
- Cohen DI, Jones P, Littenberg B et al: Does cost information availability reduce physician test usage? Med Care 1982; 20: 286-292
- Cohen SJ, Weinberger M, Hui SL et al: The impact of reading on physicians' non-adherence to recommended standards of medical care. Soc Sci Med 1985; 21: 909-914
- Cohen SJ, Christen AG, Katz BP et al: Counselling medical and dental patients about cigarette smoking: the impact of nicotine gum and chart reminders. Am J Public Health 1987; 77: 313-316
- Cowan JA, Heckerling PS, Parker JB: Effect of a fact sheet reminder on performance of the periodic health examination: a randomized controlled trial. Am J Prev Med 1992; 8: 104-109
- Davidoff F, Goodspeed R, Clive J: Changing test ordering behaviour, a randomized controlled trial comparing probabilistic reasoning with cost-containment education. Med Care 1989; 27: 45-58
- Dickinson JC, Warshaw GA, Gehlbach SH et al: Improving hypertension control: impact of computer feedback and physician education. Med Care 1981; 19: 843-854
- Everett GD, deBlois S, Chang PF et al: Effect of cost education, cost audits, and faculty chart review on the use of laboratory services. Arch Intern Med 1983; 143: 942-944
- Garett LE Jr, Hammond WE, Stead WW: The effects of computerized medical records on provider efficiency and quality of care. Methods Inf Med 1986; 25: 151-157
- Gullion DS, Tschann JM, Adamson TE et al: Management of hypertension in private practices: a randomized controlled trial in continuing medical education. J Contin Educ Health Prof 1988; 8: 239-255
- Headrick LA, Speroff T, Pelecanos HI et al: Efforts to improve compliance with the national cholesterol education program guidelines. Arch Intern Med 1992; 152: 2490-2496
- Heale J, Davis D, Norman G et al: A randomized controlled trial assessing the impact of problem-based versus didactic teaching methods in CME. Proc Res Med Educ 1988; 27: 72-77
- Hershey CO, Goldberg HI, Cohen DI: The effect of computerized feedback coupled with a newsletter upon outpatient prescribing charges. Med Care 1988; 26: 88-93
- Johnson RE, Campbell WH, Azevedo DJ et al: Studying the impact of patient drug profiles in an HMO. Med Care 1976; 14: 799-807
- Johnson RE, Campbell WH, Azevedo DJ et al: Examining physicians' drug order recording behavior. Med Care 1978; 16: 408-416
- Kopelow ML, Schnabl GK, Hassard TH et al: Assessing practicing physicians in two settings using standardized patients. Acad Med 1992; 67 (10 suppl): S19-S21
- Kottke TE, Brekke ML, Solberg LI et al: A randomized trial to increase smoking intervention by physicians. JAMA 1989; 261: 2101-2106
- Levinson W, Roter D: The effects of two continuing medical education programs on communication skills of practicing primary care physicians. J Gen Intern Med 1993; 8: 318-324
- Linn BS: Continuing medical education impact on emergency room burn care. JAMA 1980; 244: 565-570
- Litzelman DK, Dittus RS, Miller ME et al: Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols.
J Gen Intern Med 1993; 8: 311-317
- Manheim LM, Feinglass J, Hughes R et al: Training house officers to be cost conscious: effects of an educational intervention on charges and length of stay. Med Care 1990; 28: 29-42
- Martin AR, Wolf MA, Thibodeau LA et al: A trial of two strategies to modify the test-ordering behavior of medical residents. N Engl J Med 1980; 303: 1330-1336
- Mayefsky JH, Foye HR: Use of a chart audit: teaching well-child care to paediatric house officers. Med Educ 1993; 27: 170-174
- McAlister NH, Covvey HD, Tong C et al: Randomised controlled trial of computer assisted management of hypertension in primary care. BMJ 1986; 293: 670-674
- McDonald CJ: Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med 1976; 84: 162-167
- McDonald CJ: Protocol-based computer reminders, the quality of care and the non-perfectibility of man. N Engl J Med 1976; 295: 1351-1355
- McDonald CJ, Wilson GA, McCabe GP: Physician response to computer reminders. JAMA 1980; 244: 1579-1581
- McDonald CJ, Hui SL, Smith DM et al: Reminders to physicians from an introspective computer medical record, a two-year randomized trial. Ann Intern Med 1984; 100: 130-138
- McDonald CJ, Hui SL, Tierney WM: Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput 1992; 9: 304-312
- McPhee SJ, Bird JA, Jenkins CNH et al: Promoting cancer screening: a randomized controlled trial of three interventions. Arch Intern Med 1989; 149: 1866-1872
- McPhee SJ, Bird JA, Fordham D et al: Promoting cancer prevention activities by primary care physicians: results of a randomized controlled trial. JAMA 1991; 266: 538-544
- Morgan M, Studney DR, Barnett GO et al: Computerized concurrent review of prenatal care. QRB 1978; 4: 33-36
- Nattinger AB, Panzer RJ, Janus J: Improving the utilization of screening mammography in primary care settings. Arch Intern Med 1989; 149: 2087-2092
- Norton PG, Dempsey LJ: Self-audit: its effect on quality of care. J Fam Pract 1985; 21: 289-291
- Palmer RH, Louis TA, Hsu LN et al: A randomized controlled trial of quality assurance in sixteen ambulatory care practices. Med Care 1985; 23: 751-770
- Perera DR, LoGerfo JP, Shulenberger E et al: Teaching sigmoidoscopy to primary care physicians: a controlled study of continuing medical education. J Fam Pract 1983; 16: 785-788
- Pollack MM, Getson PR: Pediatric critical care cost containment: combined actuarial and clinical program. Crit Care Med 1991; 19: 12-20
- Pozen MW, D'Agostino RB, Mitchell JB et al: The usefulness of a predictive instrument to reduce inappropriate admissions to the coronary care unit. Ann Intern Med 1980; 92: 238-242
- Pozen MW, D'Agostino RB, Selker HP et al: A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multi-center clinical trial. N Engl J Med 1984; 310: 1273-1278
- Restuccia JD: The effect of concurrent feedback in reducing inappropriate hospital utilization. Med Care 1982; 20: 46-62
- Rodman JH, Jelliffe RW, Kolb E et al: Clinical studies with computer-assisted initial lidocaine therapy. Arch Intern Med 1984; 144: 703-709
- Rogers JL, Haring OM: The impact of a computerized medical record summary system on incidence and length of hospitalization. Med Care 1979; 17: 618-630
- Rogers JL, Haring OM, Wortman PM et al: Medical information systems: assessing impact in areas of hypertension, obesity and renal disease. Med Care 1982; 20: 63-74
- Rosser WW, McDowell I, Newell C: Use of reminders for preventive procedures in family medicine. CMAJ 1991; 145: 807-814
- Sanazaro PJ, Worth RM: Concurrent quality assurance in hospital care. N Engl J Med 1978; 298: 1171-1177
- Sibley JC, Sackett DL, Neufeld V et al: A randomized controlled trial of continuing medical education. N Engl J Med 1982; 306: 511-515
- Steele MA, Bess DT, Franse VL et al: Cost effectiveness of two interventions for reducing outpatient prescribing costs. DICP 1989; 23: 497-500
- Stross JK, Bole GG: Evaluation of a continuing education program in rheumatoid arthritis. Arthritis Rheum 1980; 23: 846-849
- Tape TG, Campbell JR: Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993; 94: 619-625
- Tape TG, Givner N, Wigton RS et al: Process in ambulatory care: a controlled clinical trial of computerized records. In Greenes R (ed): Proceedings of the 12th Annual Symposium on Computer Applications in Medical Care, SCAMC Inc, Washington, 1988: 749-752
- Tierney WM, Hui SL, McDonald CJ: Delayed feedback of physician performance versus immediate reminders to perform preventive care. Med Care 1986; 24: 659-666
- Tierney WM, McDonald CJ, Hui SL et al: Computer predictions of abnormal test results effects on outpatient testing. JAMA 1988; 259: 1194-1198
- Tierney WM, Miller ME, McDonald CJ: The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med 1990; 322: 1499-1504
- Turner RC, Waivers LE, O'Brien K: The effect of patient-carried reminder cards on the performance of health maintenance measures. Arch Intern Med 1990; 150: 645-647
- Weingarten MA, Bazel D, Shannon HS: Computerized protocol for preventive medicine: a controlled self-audit in family practice. Fam Pract 198; 6: 120-124
- Wexler JR, Swender PT, Tunnessen WW et al: Impact of a system of computer-aided diagnosis. Am J Dis Child 1975; 129: 203-205
- White RH, Hong R, Venook AP et al: Initiation of warfarin therapy. J Gen Intern Med 1987; 2: 141-148
- White PT, Pharoah CA, Anderson HR et al: Randomized controlled trial of small group education on the outcome of chronic asthma in general practice. J R Coll Gen Pract 1989; 39: 182-186
- Wigton RS, Zimmer JL, Wigton JH et al: Chart reminders in the diagnosis of anemia. JAMA 1981; 245: 1745-1747
- Winickoff RN, Coltin KL, Morgan MM et al: Improving physician performance through peer comparison feedback. Med Care 1984; 22: 527-534
- Wones RG: Failure of low-cost audits with feedback to reduce laboratory test utilization. Med Care 1987; 25: 78-82
- Hong SW, Ching TY, Fung JPM et al: The employment of ward opinion leaders for continuing education in the hospital. Med Teach 1990; 12: 209-217
- Raisch DW, Bootman JL, Larson LN et al: Improving antiulcer agent prescribing in a health maintenance organization. Am J Hosp Pharm 1990; 47: 1766-1773
- Freemantle N, Grilli R, Grimshaw J et al: Implementing findings of medical research: the Cochrane Collaboration on Effective Professional Practice. Q Health Care 1995; 4: 45-47
[Table of Contents]
This guideline is for reference and education only and is not intended to be a substitute for the advice of an appropriate health care professional or for independent research and judgement. The CMA relies on the source of the CPG to provide updates and to notify us if the guideline becomes outdated. The CMA assumes no responsibility or liability arising from any outdated information or from any error in or omission from the guideline or from the use of any information contained in it.
CMAJ November 15, 1995 (vol 153, no 10)
/ JAMC le 15 novembre 1995 (vol 153, no 10)