Evaluating educational programmes: AMEE Education Guide no. 29

  Here is the latest AMEE Education Guide, published in the May 2006 issue of Medical Teacher.   See also AMEE Education Guides: Extended Summaries; AMEE Education Guides

Goldie J. AMEE Education Guide no. 29: Evaluating educational programmes. Medical Teacher 2006; 28(3): 210-224.

Abstract: Evaluation has become an applied science in its own right in the last 40 years. This guide reviews the history of programme evaluation through its initial concern with methodology, giving way to concern with the context of evaluation practice and into the challenge of fitting evaluation results into highly politicized and decentralized systems. It provides a framework for potential evaluators considering undertaking evaluation. The role of the evaluator; the ethics of evaluation; choosing the questions to be asked; evaluation design, including the dimensions of evaluation and the range of evaluation approaches available to guide evaluators; interpreting and disseminating the findings; and influencing decision making are covered.

From the conclusion:
In performing evaluations, evaluation theory can help evaluators with all aspects of the process. Previously adopted approaches often present and provoke new ideas and techniques, and provide useful checklists. However, evaluators should be aware of the limitations of individual evaluation approaches and be eclectic in their choice of methods. The ‘good enough’ rule is worth remembering. As with all research findings, the validity and reliability of the data obtained are important to establish. When using quantitative and qualitative approaches in the same evaluation it is important to unify the different approaches. Recognition of the social components of evaluation knowledge and the fallibility of evaluation methodologies has led to the need for meta-evaluation …  On reviewing the results of his/her endeavour it is important for the educational evaluator to remember the lesson history teaches: that improvement, even when modest, is valuable.

Article headings: Introduction; What is evaluation?; History of evaluation; Effecting programme evaluation; Interpreting the findings; Dissemination of the findings; Influencing decision-making; Conclusions; References

The evidence-based health care debate – 2006. Where are we now?

The June 2006 issue of the Journal of Evaluation in Clinical Practice is the ninth in a series of “thematic editions” on evidence-based medicine. (See the list below for introductions to the previous issues.) Below is an excerpt from the introduction to the current issue. [full text by subscription]

Miles A, Polychronis A, Grey JE. The evidence-based health care debate – 2006. Where are we now? Journal of Evaluation in Clinical Practice 2006;12:239-247.

From the introduction:
When an academic in the History of Medicine comes to write a comprehensive account of the nature, characteristics and scale of contribution to human progress of the evidence-based health care movement, there will indeed be a great deal of considerable interest to read. Rarely before, it seems, has there been such fierce and greatly protracted polarization of both scientific and clinical positions as in the last 16 years since the coining of that now severely tired, and almost defunct neologism, ‘evidence-based medicine’ (EBM) (Evidence-Based Medicine Working Group 1992; Polychronis et al. 1996). The Journal of Evaluation in Clinical Practice, over this time, has achieved a pre-eminent reputation in directly ensuring that the necessary debates on the nature of ‘evidence’ took place and, indeed, that they continue to take place. …

We are therefore gratified to commit to the international medical literature the present Part One of the 9th Thematic Edition (Volume 12, Number 3) on the progress of the EBM movement, which augments the previous eight (Miles et al. 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004) and, at the time of writing, we are completing the editing of Part Two of the present Thematic Edition, to be published as the next issue (Volume 12, Number 4). This very large opus having been achieved, we invite contributions to the 10th Thematic Edition for publication in 2007.     DOI Link    June 2006 issue

Here is key article from this issue:

Tonelli, Mark R. Integrating evidence into clinical practice: an alternative to evidence-based approaches. Journal of Evaluation in Clinical Practice 2006; 12 (3), 248-256.

Abstract: Evidence-based medicine (EBM) has thus far failed to adequately account for the appropriate incorporation of other potential warrants for medical decision making into clinical practice. In particular, EBM has struggled with the value and integration of other kinds of medical knowledge, such as those derived from clinical experience or based on pathophysiologic rationale. The general priority given to empirical evidence derived from clinical research in all EBM approaches is not epistemically tenable. A casuistic alternative to EBM approaches recognizes that five distinct topics, 1) empirical evidence, 2) experiential evidence, 3) pathophysiologic rationale, 4) patient goals and values, and 5) system features are potentially relevant to any clinical decision. No single topic has a general priority over any other and the relative importance of a topic will depend upon the circumstances of the particular case. The skilled clinician must weigh these potentially conflicting evidentiary and non-evidentiary warrants for action, employing both practical and theoretical reasoning, in order to arrive at the best choice for an individual patient.

Previous EBM thematic editions:

Miles A., Bentley P., Polychronis A. & Grey J.E. (1997) The limits of evidence-based medicine. Journal of Evaluation in Clinical Practice 3, 83-86.
Miles A., Bentley P., Polychronis A., Grey J.E. & Price N. (1998) Recent progress in health services research: on the need for evidence-based debate. Journal of Evaluation in Clinical Practice 4, 257-265.
Miles A., Bentley P., Polychronis A., Grey J.E. & Price N. (1999) Advancing the evidence-based healthcare debate. Journal of Evaluation in Clinical Practice 5, 97-101.
Miles A., Charlton B.G., Bentley P., Polychronis A., Grey J.E. & Price N. (2000) New perspectives in the evidence-based healthcare debate. Journal of Evaluation in Clinical Practice 6, 77-84.
Miles A., Bentley P., Polychronis A., Grey J.E. & Melchiorri C. (2001) Recent developments in the evidence-based healthcare debate. Journal of Evaluation in Clinical Practice 7, 85-89.
Miles A., Grey J.E., Polychronis A. & Melchiorri C. (2002) Critical advances in the evaluation and development of clinical care. Journal of Evaluation in Clinical Practice 8, 87-102.
Miles A., Grey J.E., Polychronis A., Price N. & Melchiorri C. (2003) Current thinking in the evidence-based healthcare debate. Journal of Evaluation in Clinical Practice 9, 95-109.
Miles A., Grey J.E., Polychronis A., Price N. & Melchiorri C. (2004) Developments in the evidence-based healthcare debate – 2004. Journal of Evaluation in Clinical Practice 10, 129-142.
 

Medical Teacher May 2006; 28 (3)

See also Individual Issues of Journals; News & Selected Journals for Medical & Adult Educators
 The May 2006 issue of Medical Teacher has just been published online; full text by subscription.  Selected titles:

Teaching professionalism: general principles 

AMEE Education Guide no. 29: Evaluating educational programmes

Teaching medical students using role play: Twelve tips for successful role plays

Formal mentoring programmes for medical students and doctors – a review of the Medline literature

Portfolio as a tool to stimulate teachers' reflections

A manageable approach to integrating personal digital assistants into a family medicine clerkship

Teaching in practice: a survey of a general practice teaching network    
 

Friday Fun: Palm Mobifest: Canada’s first mobile film festival — really really short films you can watch here

 Canada's first film festival honouring films that were shot and created for a mobile phone, smart-phone or other handheld device was Palm Mobifest, held last week in Toronto. Festival winners were chosen from a pool of 60 finalists from Colombia, Argentina, Australia, India, Ireland, New Zealand, Spain, the United States and Canada. These films are around a minute long, and you can view them on the mobifest Web site. Turn up your sound!

The festival winner: Regurgitation (Can't pay that huge restaurant bill? No problem. Bring it on, and bring it up!)

The Canadian winner: Swings /Sits (.stationary .mobile .enclosed .free .the everyday experience)

Caught on Treo winner: The Trio (Three misunderstood artists try to make "The Date": a short film about two people on a date with only a piece of fruit to share between them.)

You can view all the films from here.

Academic Medicine June 2006; 81 (6) Supplement

See also Individual Issues of Journals; What's New in the Medical Education Journals; News & Selected Journals for Medical & Adult Educators
The June 2006 issue of Academic Medicine is now available online. It is a special supplement on medical education pipeline initiatives. From the preface by Ella F. Cleveland and Ann Steinecke:

We are pleased to bring you this supplement of Academic Medicine. Our hope is that it will add to the literature on medical education pipeline initiatives and highlight the legacy of the Health Professions Partnership Initiative (HPPI), which was funded by The Robert Wood Johnson and W. K. Kellogg Foundations and conducted from 1996 to 2005.

The issue opens with a foreword by Association of American Medical Colleges (AAMC) vice president for the Division of Diversity Policy and Programs, Charles Terrell, demonstrating the need and context for these partnerships. The next two articles (by Davis Patterson and Jan Carline of the University of Washington and by James Hamos of the National Science Foundation, respectively) present the current scholarship on pipeline partnerships and review the literature.

From The Future of the Health Professions Pipeline: A New Call to Action by Jordan J. Cohen:

As the articles in this supplement demonstrate, HPPI’s 26 programs have provided needed continuity for an entire generation of future physicians and scientists. The projects described on these pages—as well as the many other projects nationwide—have been essential to moving us toward a physician workforce that looks more like the United States, improving the quality of care available for underserved Americans, and helping to close medicine’s diversity gap. Additionally, HPPI’s numerous success stories are testimony to how far underrepresented minority students can go when provided both the opportunity and necessary resources.

Referral patterns and attitudes of primary care physicians towards chiropractors

  Published recently in BMC Complementary & Alternative Medicine:

Greene BR, Smith M, Allareddy V, Haas M. Referral patterns and attitudes of primary care physicians towards chiropractors. BMC Complement Altern Med 2006;6:5.:5.

BACKGROUND: Despite the increasing usage and popularity of chiropractic care, there has been limited research conducted to examine the professional relationships between conventional trained primary care physicians (PCPs) and chiropractors (DCs). The objectives of our study were to contrast the intra-professional referral patterns among PCPs with referral patterns to DCs, and to identify predictors of PCP referral to DCs.

METHODS: We mailed a survey instrument to all practicing PCPs in the state of Iowa. Descriptive statistics were used to summarize their responses. Multivariable logistic regression analyses were conducted to identify demographic factors associated with inter-professional referral behaviors.

RESULTS: A total of 517 PCPs (33%) participated in the study. PCPs enjoyed strong intra-professional referral relationships with other PCPs. Although patients exhibited a great deal of interest in chiropractic care, PCPs were unlikely themselves to make formal referral relationships with DCs. PCPs in a private practice arrangement were more likely to exhibit positive referral attitudes towards DCs (p = 0.01).

CONCLUSION: PCPs enjoy very good professional relationships with other PCPs. However, the lack of direct formalized referral relationships between PCPs and chiropractors has implications for efficiency, continuity, quality, and patient safety in the health care delivery system. Future research must focus on identifying facilitators and barriers for developing positive relationships between PCPs and chiropractors.

PubMed Full Record         Related Articles        Free Full Text     View all chiropractic posts.

Getting physicians to accept new information technology: insights from case studies

From the May 23 CMAJ:  [free full text]

Zitner D. Physicians will happily adopt information technology [commentary]. CMAJ 2006;174:1583-1584.

Excerpt: In this issue [of CMAJ], Lapointe and Rivard report on their analysis of computer information system (CIS) implementations at 3 hospitals to understand better the dynamics of physicians' resistance to such implementations. The level of resistance varied. In 2 cases it was met with responses from implementers that reinforced the resistance behaviours, and the systems were eventually removed; in the other case the responses to the resistance were supportive, the resistance decreased and the system was ultimately successful.

Lapointe and Rivard suggest that an understanding of power dynamics is critical to learn why modern health information tools and techniques are not readily adopted. The slow adoption of health information systems in Canada is startling because Canada is thought of as a civilized country that often undertakes collective action around shared interests. Although an understanding of power and the relationships between various constituencies is important in a civilized society, theories of power and how power is exercised are not critical to learn why most Canadian physicians fail to adopt modern health information tools.
PubMed Record    Free Full Text 

Lapointe L, Rivard S. Getting physicians to accept new information technology: insights from case studies. CMAJ 2006;174:1573-1578.

BACKGROUND: The success or failure of a computer information system (CIS) depends on whether physicians accept or resist its implementation. Using case studies, we analyzed the implementation of such systems in 3 hospitals to understand better the dynamics of physicians' resistance to CIS implementation.
METHODS: We selected cases to maximize variation while allowing comparison of CIS implementations. Data were collected from observations, documentation and interviews, the last being the main source of data. Interviewees comprised 15 physicians, 14 nurses and 14 system implementers. Transcripts were produced; 45 segments of the transcripts were coded by several judges, with an appropriate level of intercoder reliability. We conducted within-case and cross-case analyses of the data.
RESULTS: Initially, most staff were neutral or enthusiastic about the CIS implementations. During implementation, the level of resistance varied and in 2 instances became great enough to lead to major disruptions and system withdrawal. Implementers' responses to physicians' resistance behaviours played a critical role. In one case, the responses were supportive and addressed the issues related to the real object of resistance; the severity of resistance decreased, and the CIS implementation was ultimately successful. In the other 2 cases, the implementers' responses reinforced the resistance behaviours. Three types of responses had such an effect in these cases: implementers' lack of response to resistance behaviours, antagonistic responses, and supportive responses aimed at the wrong object of resistance.
INTERPRETATION: The 3 cases we analyzed showed the importance of the roles played by implementers and users in determining the outcomes of a CIS implementation.
PubMed Record     Free Full Text 

A randomized trial of teaching clinical skills using virtual and live standardized patients

From the May 2006 issue of the Journal of General Internal Medicine:

Triola M, Feldman H, Kalet AL, Zabar S, Kachur EK, Gillespie C et al. A randomized trial of teaching clinical skills using virtual and live standardized patients. J Gen Intern Med 2006; 21(5):424-429.

BACKGROUND: We developed computer-based virtual patient (VP) cases to complement an interactive continuing medical education (CME) course that emphasizes skills practice using standardized patients (SP). Virtual patient simulations have the significant advantages of requiring fewer personnel and resources, being accessible at any time, and being highly standardized. Little is known about the educational effectiveness of these new resources. We conducted a randomized trial to assess the educational effectiveness of VPs and SPs in teaching clinical skills.
OBJECTIVE: To determine the effectiveness of VP cases when compared with live SP cases in improving clinical skills and knowledge.
DESIGN: Randomized trial.
PARTICIPANTS: Fifty-five health care providers (registered nurses 45%, physicians 15%, other provider types 40%) who attended a CME program.
INTERVENTIONS: Participants were randomized to receive either 4 live cases (n=32) or 2 live and 2 virtual cases (n=23). Other aspects of the course were identical for both groups. RESULTS: Participants in both groups were equivalent with respect to pre-post workshop improvement in comfort level (P=.66) and preparedness to respond (P=.61), to screen (P=.79), and to care (P=.055) for patients using the skills taught. There was no difference in subjective ratings of effectiveness of the VPs and SPs by participants who experienced both (P=.79). Improvement in diagnostic abilities were equivalent in groups who experienced cases either live or virtually.
CONCLUSIONS: Improvements in performance and diagnostic ability were equivalent between the groups and participants rated VP and SP cases equally. Including well-designed VPs has a potentially powerful and efficient place in clinical skills training for practicing health care workers.
PubMed Record       Journal Record        May issue  

Friday Fun: Do you feel like dancing?

 Thanks to Sue Pelletier, we will not miss Friday Fun this week. Sue writes:

If you feel like dancing, check out these two short videos: 

• The Evolution of Dance
• Black-and-white video of swing dancers from way back when, set to hip-hop music. It is amazing!

PLoS Clinical Trials: new Open Access journal launched by Public Library of Science

PLoS Clinical Trials, launched May 2006, provides an unbiased, peer-reviewed venue for clinical trials results in all fields of medicine and public health. The journal aims to increase the breadth, depth, and transparency of clinical trials reporting. [All articles are free online.]

PLoS is the Public Library of Science, a nonprofit organization of scientists and physicians committed to making the world’s scientific and medical literature a freely available public resource.

Some recent articles:
Patient Safety Requires a New Way to Publish Clinical Trials [editorial]
Factors That Can Affect the External Validity of Randomised Controlled Trials
Principles for Strengthening the Integrity of Clinical Research