Medical Meetings’ social media columnist Brian McGowan, PhD (aka, @CMEadvocate on Twitter) has, along with colleagues, done some interesting research into how physicians are using social media in their work—something CME providers need to know if they want to incorporate social technology into their activities.

They will be presenting the results of their study at Medicine 2.0′11 next month, but in the meantime, check out this post on the Medicine 2.0 blog for a sneak peak at the data.

Outcomes fatigue, where healthcare professionals just run their pen down the form rather than provide thoughtful, meaningful answers to post-activity evaluations, is all too common. On August 10, continuing medical education professionals got together at the #CMEChat hashtag on Twitter to hash out some better approaches to measuring learning.

Led by @meducate (aka, Medical Meetings columnist Lawrence Sherman), the group began by agreeing that writing good questions is both art and science—you need to ask the right questions in the right way, and you need to ensure that the structure is in place to evoke high-quality answers. And you have to ask the right number of questions—enough to get meaningful results, but not so many that participants throw down their pens halfway through. One person suggested that providers approach the evaluation from the learners’ perspectives—“why are they asking me this and what do they want to know?” Another suggestion was to “start with the end in mind.”

Another part of the problem is that it can be hard to craft effective outcomes questions if you don’t have an in-depth knowledge of the therapeutic area, as may be the case with a medical writer. But involving an expert can be time-consuming, cumbersome, and expensive, as one person pointed out.

It also doesn’t help that some providers tend have a two-part approach: Part 1 being to ask questions to fulfill the Accreditation Council for CME’s requirements; and Part 2 being to measure what’s been learned.

There also was agreement that more open-ended questions can be valuable.

Standardize or Not?

The CMEChatters talked about the possible benefits of using a set number and/or type of questions. As one tweeter said, “Formulaic writing gives me the creeps,” but it does make it easier to compare data across programs when you can compare apples to apples. One person suggested that evaluation tools should at least be validated, or perhaps even standardized.

Another said a standardized evaluation tool is a must if you want to compare outcomes across programs. But as the conversation moved back and forth between outcomes and evaluations, one CMEChatter warned that everyone should remember that “outcomes is not the same as evaluations”—while they’re not mutually exclusive, they are different. As another said, “Evaluation is a process, outcomes is a science.”

Validation and Value

They also talked about the limits to the value of self-reporting of intent to change without actually validating that change. But validation is another big can of worms. Some docs don’t want to provide answers that will be tracked to them or just be under the microscope by CME providers, the CMEChatters said.

Fatigue or Indifference?

What is it that make so much evaluation data so distasteful for learners? Are they fatigued, or just indifferent? Have we created an environment where learners simply check boxes without thinking through each evaluation question?

As one person said, “Um, yes.”

So where does that indifference come from?

There are some “so-called learners” who aren’t really there to learn. They’re just there for the free meal or credits. If they don’t care about the learning, why should they care about the evaluation? So, as one person quipped, “How do we turn the munch bunch into learners?” While several people admonished that learners deserve more respect than this exchange implied, others said it’s important to be realistic—not everyone is there to learn.

These folks should read some of the data on the clinical impact of CME.

Creating Understanding on Why Answering These Questions Matter

So, what would make a training/education so exciting that they want to be there, and to fill out the form afterward?

The problem is, they view it as jumping through hoops to get their credit. They don’t realize why providers need their information, said one CMEChatter. CME providers need to involve the learners in the educational process, and they need to design the educational intervention around the notion that you want to make people want the education. Find ways to create a positive, creative, safe learning environment, said another. “So much depends on learning environment, [the] expectations set by leaders and teachers in the room.”

“If evaluation is important to you, why not make it important for persons filling it (not by force, [but by] seduction).” The provider community can show learners that there are reasons why they ask these questions, and that the answers have value. The key word, one person said, is environment, and the key concept is trust. “Do learners trust teachers to use data appropriately?”

Short of paying them, as one chatter suggested tongue in cheek, you can encourage them to reflect on what they learned through the evaluation form, so filling out the form becomes another way to cement what they learned. The only problem with reflection is when the educational activity isn’t significant enough (in a CME context) to arouse deep reflection.

One way to help overcome their reluctance to participate in outcomes validation measures would be to provide aggregate outcomes to the learners so learners could see the value of responding to those questions, said one tweeter. “If you’re not making use of the results (formatively for yourself) in evaluations, you’ll get fatigue,” added another. “Feedback to the learners is critical, but often omitted,” a CMEChatter agreed.

Or you could just get rid of the form altogether and follow the lead of one of the CMEChatters, who uses coaching to monitor learning process.

Post-Activity Evaluation Practice Pearls

Here are a few of the pearls of wisdom the CMEChatters imparted at the end of the chat:

• Give the learners a framework to design their own CME.

• Don’t leave the form to the last minute—plan it when you plan the intervention.

• Include open questions so that you’re not eating your own dog food all the time.

• Instead of focusing so much on assessment, maybe the focus should be placed more on the development of the earning environment.

Don’t miss the next #CMEChat, coming to your desktop Wednesday, 8/17, at 11 am Eastern.

And, as always seems to be the case these days, Tom Sullivan beat me to it when it comes to crunching the numbers of the ACCME’s 2010 Data Report, which was released yesterday afternoon. Here’s his analysis —thanks Tom!

It was interesting to read on the CME LinkedIn group one person’s criticism that the report’s emphasis on incomes should be shifted to an annual CME report “that describes how the CME community impacted the healthcare system. Focus on how ‘accredited’ CME was integrated into the healthcare quality efforts. Provide examples of CME offices that evolved and help us understand best practices in this evolution. AND, collect and report data on education and life-long learning as a fundamental element of the healthcare improvement system.”

While I think the income piece is important to track year over year, I agree that an annual report of the nature he describes would be fantastic. I’m just not sure ACCME is the right stakeholder to take that on. I’m not sure who has the resources and capability to do it, but wouldn’t that be an excellent tool to use to help our legislators, regulators, press, and everyone else involved understand what CME is and how it is central to maintaining the health of the U.S. healthcare system? Anyone willing to take it on?

Derek makes some interesting points in this post, The Value (Or Lack Thereof) Of Self-Reported Outcomes In CME. I know he’s right about people just ripping through the checkoffs like (pick a metaphor) and not providing meaningful data other than a baseline from which you can find outliers, but I’m wondering if maybe he’s not asking the right question. He proposes three ways to get better data, including shortening up the form to asking just four questions. Which I’m sure would help, especially when two of them require write-in answers.

As he says, “My hope is, by simplifying and reducing the form to these four basic questions, more participants would be willing to take the time to give thoughtful, articulate answers that would provide meaningful insight into the achievement of the desired outcomes of the activity.”

The question I think we need to be asking is why learners aren’t motivated to give those thoughtful answers to begin with. As always, in my mind, anyway, it comes down to the old “what’s in it for me?” Obviously, they don’t see enough value in filling out the form to make it worth their while to do more than the bare minimum. Other than shortening the form, which just makes it less painful, not more valuable, what can you as a CME provider do to engage learners in the outcomes-gathering process so that they actually want to do a good job with those forms?

Update: I just did a quick archive search and found this article on outcomes from back in 2003 that touches on these issues. Some of the tips people gave me then were:

* Keep it short

* Use open-ended questions

* Ask about each objective (so much for keeping it short!), not just if objectives were met

* Offer incentives (free registration/hotel/airfare to participate in another activity)

• Tell attendees that you’ll be following up with a survey in three or six months (check the article for tips on how to get them to participate in follow-up surveys and questions to ask)

In re-reading this thing, it’s actually holding up pretty well for something written eight years ago. While that make me feel pretty good in one way, it’s kind of sad that we haven’t made much progress over that time in figuring this out.

Here’s another kind of handy sidebar from the article that I think still holds up:

Measurement Tools

Here’s a look at just a few of the many ways to measure CME outcomes:

SELF-REPORT THROUGH EVALUATIONS: Asking attendees what they learned and how they planned to (or how they did) use what they learned through immediate, post-meeting evaluation forms and follow-up mailings, e-mailings, faxes, and telephone interviews

Pros: Easy to implement, relatively inexpensive

Cons: Not very reliable, can be difficult to get a significant number of responses

CASE STUDIES: Presenting attendees with a case study related to a specific practice area, both as a pre-test and as a post-meeting evaluation. Can be done via telephone, fax, e-mail, or mail

Pros: While still a form of self-report, studies have found it to be reliable in terms of predicting physician behavior; can be as cost-effective as evaluations

Cons: Need to have expertise to design an effective case study; can be difficult to get responses

CHARTS/PATIENT CARE RECORDS: Measuring baseline performance and post-meeting behavioral improvement by looking at attendees’ patient care records

Pros: Highly effective form of evaluation, especially when the records are available in database form

Cons: Privacy issues can be an impediment; can be difficult to obtain outside of hospitals and large healthcare systems

STANDARDIZED PATIENTS: Objective, structured clinical exams where physicians visit stations and examine patients presenting a particular disease. Docs have to come up with the right answer before they can move on to the next station.

Pros: Highly effective form of evaluation; allows CME provider to observe physician interacting with actors posing as patients

Cons: Requires a lot of time and resources to develop and implement

In the past I’ve wondered why communications among the key players in the CME community tend to be so vague and open to interpretation. Just today I ran across an article that further helped me better understand why sometimes it’s actually beneficial to keep it vague: Read Wired’s In Praise of Vagueness. This sums up the main point:

“Sometimes, precision is dangerous, a closed door keeping us from imagining new possibilities. Vagueness is that door flung wide open, a reminder that we don’t yet know the answer, that we might still get better.” Which is basically what Dr. Kopelow has said to me when I’ve asked him to get specific on some of the finer points of ACCME regulations. I get it. It’s still frustrating, but I get it.

Your Capsules guest blogger today is CME Hulk, a great green glob of CME goodness whose Twitter feed I have been enjoying mightily (not to mention his strange-yet-compelling video tweaking of the ACCME and a Q&A he did on Confessions of a Medical Educator). He had to think about it for a while, but I am thrilled to say he took me up on the offer to write a guest blog. I don’t quite have the physique for it, but if I work out enough, maybe I can adopt his problem-solving approach too. Anyway, enjoy! As he says, THIS GOOD. YOU POST ON BLOG. MANY PEOPLE READ, MAKE BETTER CME.

THIS CME HULK. CME HULK WRITE BLOG POST. TELL YOU WHAT WRONG CME. YOU LISTEN CME HULK. WHY? CME HULK HAVE CCMEP. THAT WHY. CME HULK MAKE YOU BETTER CME. PLUS CME HULK SMASH IF NOT LISTEN.

HERE 3 THINGS WRONG CME.

1. TOO MANY CME PEOPLE NOT USE BRAIN

WHY CME HULK SAY THIS? THAT EASY. WHAT 2 MOST COMMON THING CME PEOPLE DO NOT BE COMPLIANCE? NOT KNOW? CME HULK KNOW, HAVE CCMEP. CME PEOPLE NOT GET GRANTOR PEOPLE SIGN LOA. THAT 1. CME PEOPLE NOT GET SPEAKER PEOPLE SIGN COI FORM. THAT 2. CME HULK NOT UNDERSTAND. WHY CME PEOPLE NOT USE BRAIN? THIS NOT ROCKET SCIENCE. THIS VERY SIMPLE. ALWAYS HAVE CME PEOPLE AND GRANTOR PEOPLE SIGN LOA. ALWAYS HAVE SPEAKER PEOPLE SIGN COI FORM. THIS MAKE CME HULK ANGRY. YOU NOT LIKE CME HULK WHEN ANGRY. CME HULK MAKE EASY. YOU GET SIGNATURES, CME HULK NO SMASH. YOU NOT GET SIGNATURES, CME HULK SMASH. THIS GOOD DEAL.

2. WHY NOT WANT FREE MONEY?

HERE NOTHER THING CME HULK NOT GET. DOCTOR NEED LEARN SO BETTER TAKE CARE PEOPLE. CME PEOPLE HAVE PROGRAM HELP DOCTOR LEARN. COST MONEY MAKE CME PROGRAM, WHO PAY? DOCTOR? OR GET MONEY FROM PHARM PEOPLE AND MAKE CME PROGRAM CHEAP? SOME PEOPLE SAY, NO TAKE PHARM PEOPLE MONEY. IT BAD. CME HULK THINK THIS CRAZY. PEOPLE WANT GIVE CME HULK MONEY, CME HULK SAY THANK YOU, TAKE MONEY. THEN MAKE GOOD CME. YOU NO WANT PHARM PEOPLE MONEY, THAT FINE. GIVE CME HULK. PEOPLE SAY NO TAKE PHARM PEOPLE MONEY MAKE CME HULK ANGRY. MAKE CME HULK WANT SMASH. MAYBE CME HULK JOIN ACRE INSTEAD.

3. NO GOOD PICME TICIPATION

CME HULK TALK PEOPLE DO PICME. TELL CME HULK GET 5 PEOPLE STAGE C. MAYBE 10 PEOPLE. TELL CME HULK THIS GOOD TICIPATION. THIS NOT GOOD TICIPATION. THIS LOUSY TICIPATION. PEOPLE SAY, BUT CME HULK, HOW GET GOOD TICIPATION IN STAGE C PICME? THIS EASY. CME HULK TELL YOU HOW HE DO. CME HULK HAVE CCMEP. WHEN DO PICME, CME HULK SEND LETTER TO DOCTOR FOR STAGE C. SAY, YOU DO THIS NOW. IF DOCTOR NO DO, CME HULK CALL ON PHONE. SAY, YOU DO THIS NOW. IF STILL NO DO, CME HULK RUN TO DOCTOR OFFICE. FIND DOCTOR. SAY, YOU DO THIS NOW OR CME HULK SMASH. THEN SHOW CCMEP CERTIFICATE. DOCTOR ALWAYS DO STAGE C.

Thanks, Mr. Hulk! You can guest blog here any time.

Want to know how to use social media effectively in CME? Free webinar today at 2 ET led by Brian McGowan and Joseph Kim. Come join us! If you can’t make it today, it will be available for viewing on demand. We’ll also be tweeting live at the #SoMeCME hashtag on Twitter.

Now that patients are tapping into the Internet to get smarter about what ails them, should they have more of a role in the continuing medical education that their docs must undertake to maintain their licensure and certification? There are, as Lawrence Sherman points out, many benefits in including patients in the full continuum of post-graduate healthcare provider education. CME providers can tap into a Facebook page or Twitter hashtag for patients who have a specific disease and find out where they consider the gaps in their care to be and tailor the education to close those gaps. You can put a human face on a dry scientific session by having a patient tell his story in person or hers via video. You can include patients in the post-CME-activity evaluation to get feedback on whether the education actually did improve the care they receive from their HCP.

It all sounds great, except for one thing. As Anne Finger says, “We’re all aware of clinicians who can barely hide their exasperation when a patient offers Web-accrued knowledge to assist in the differential diagnosis. Some doctors feel that their high-level scientific meetings are not the place for reports from patients.”

In other words, patients may have more access to information, but that doesn’t mean the relationship between docs and patients has necessarily changed, at least, not universally. While some docs may now be willing to concede that Google-educated patients have a place on the dais at their medical society’s annual conference, many either aren’t willing to switch teacher/learner chairs with their patients, or have run into too many self-educated, woefully misinformed patients not to discount the worth of what any of them have to say.

While docs may not want to hear a patient’s take on the latest scientific findings at a CME activity, patients have an important story to tell—their own. I believe one of the roles a CME provider can have is to help docs learn how to become better practitioners by listening to those stories. The scientists can teach them the science that will make them effective clinicians, but patients can teach them the art of being a good doctor.

P.S. Writing this reminded me of a story I wrote a while back about a meeting that brought together people with Prader-Willi Syndrome and those who treat them. It includes what I think is a good list of tips at the end for how to incorporate patients into the programming as participants. Our latest Ethical Hypothetical also discusses some precautions you might want to take when including patients on the faculty side of CME.

P.P.S. I asked my favorite CME LinkedIn group for thoughts on including informed patients in CME, and learned some interesting things. For example, did you know that in the U.K. patient feedback is an integral piece of physicians’ and other healthcare professionals’ annual appraisals? I wonder how many U.S. healthcare organizations include include patient feedback in their HCP evaluations? Now that so many of us check out our docs on Angie’s List or some other social site to see what patients say about the care they provide, it’d be almost bizarre if the organizations that employ them don’t consider that criteria as well.

As I may have mentioned before, there’s a group of CME professionals that e-gather every Wednesday at 11 am ET on Twitter to talk about the art and science of CME. The moderator of the 7/12/11 session, Brian S. McGowan, PhD (aka CMEAdvocate on Twitter), posed several questions, which the group then dug into in bites of 140 characters or less. The full archive of the chat is available, but I thought I’d make it easier to wade through by providing a summary of some of what we talked about (and fellow #CMEChatters, if I misinterpreted anything, please let me know in the comments!).

Brian began by pointing out that the CME community was told back in 2006 that online education was effective. What has changed with online CME since then?

While the CME tweeters said that while more and better interactive formats are available now, too much of online CME is an endurance test, with overly long activities that consist of talking headshots and PowerPoint slides, they said. Too often CME providers spend a lot of time and energy trying to reproduce the live experience online; “Is that really the goal?” asked one person. And while there has been progress with Web 2.0 tools that include the interactivity and peer feedback that is vital to learning, CME providers have been slow to embrace the power of these platforms.

Is the lack of Web 2.0 adoption related to the science of learning (and e-learning), or is it just a matter of wanting to stick with the status quo? Exacerbating the issues, said one tweeter, was a disconnect between IT, Web, and education departments that often may not be on the same page. And, while the technology is there for the taking, not everyone wants to commit the time and costs of implementing it (more on this later).

Covering Daily and Archived Coverage

Daily conference coverage is a staple at many live CME conferences. Reporting what happens is necessary if you want to move the content and knowledge outside of the room, but does anyone really learn using this method?

One problem, one person tweeted, is that the learner is at the mercy of what the writer deemed to be important—this can be the biggest pitfall, another noted, since it limits the spectrum of coverage to the writer’s preferences (note from Sue: Hmm, consider the source when reading this writeup!). And does the writer’s status in that field affect the credibility of their coverage choices? The question is how to balance the simple reporting of the CME and the need for engaged learning and being able to put that learning into the context of a healthcare professional’s practice, something that will likely depend on the education needs of individual learners. One suggestion that was greeted with Twittery enthusiasm (several retweets) was that organizations could set up reporting plans that matched their abstract review and approval plans. This might be more efficient, but would it be more effective, questioned another tweeter.

However, aside from providing credit, how is this daily coverage any better than what gets reported in the evening news, queried a #CMEChatter. Actually, said another, the evening news may be more effective in reaching docs—think about the “physician as consumer,” and patients, who may benefit as soon as their next appointment. Another noted that the evening news and the daily coverage often are coming from the same sources set up by the medical societies.

Whatever the shortcomings of daily reporting, whether by news media or the conference organizers, another said, “some coverage is better than no coverage for those who can’t attend.”

Which brought the group to tweeting about the pros and cons of archiving and repurposing live lectures. If it’s archived in a timely and media-appropriate fashion, it can be a great way to reach more learners, said one #CMEChat participant. It’s also cost effective, extending your reach and giving you more bang for your buck, said another. Repurposed archived activities also can ensure the entire body of knowledge gets disseminated, not just what one writer deemed to be important.

A weakness, of course, is that you end up with the long, talking-head online lecture that the chatters said is a disadvantage of online CME. So you basically can’t win, said one tweeter: “Online coverage gets skewed by buzz and what is reported, repurposed video tells a broader story, but is indigestible?”

But what if Web 2.0 features were enabled for all repurposed content and faculty were encouraged to re-engage? “Nirvana” and “Valhalla” were mentioned in response. “If faculty could re-engage asynchronously using Web 2.0 tools, then all learners could reap the benefits of ‘live’ Q&A” for real interactivity,” said one. That would be the model of medical education in the near future, another tweeted. But they were quickly brought back to earth when another person pointed out that there remains the problem of getting the audience to engage. And then there’s the faculty: “Just one more thing for them to do. ‘I’m going to need a higher honorarium for that,” snarked a tweeter.

Clearing New Channels

Brian finished up by asking what the #CMEChatters would build if they could build their own channel for sharing new medical data.

It would reinforce learning by including e-mail and text notification options to alert learners to new comments, and include ratings, voting, and sharing as well as comments, they said. “I would love to see Web 2.0 be part of every online CME activity and I want Wi-Fi in every live session (plus powerstrips),” said one person. While that sounds great, another person questioned how much time participants would be willing to spend on the Web 2.0 piece, since he can barely get them to complete a short evaluation. The best way to get them involved is to have a colleague show how they’re successfully using social media, someone tweeted. Another said it was up to providers to show learners the benefits and create the top-of-mind desire to engage. Then there’s the stick approach: It also can be a required as a prerequisite to the learning, suggested another.

And, voicing the frustrations of the already overloaded, another person said, “Does the world need a new channel? Isn’t our attention scattered enough already?” While a few agreed with that sentiment, most seemed to want to find ways to overcome the barriers to making their vision become reality. While the costs of re-engineering a Web site might seem prohibitive to a small provider, the costs will go down if you “cut out 40 percent to 50 percent of the fluff” and concentrate efforts on just the things that will enhance learning. “The idea that this all costs more is untenable,” he added. “We are adding in efficiency and stripping away proprietary shiny silver objects.” Another person noted that the new learning management system platforms have Web 2.0 built in already.

Stay tuned for the next #CMEChat on Wednesday at 11 am ET. It really is all that and a piece of pie!

I’d like to point you toward a really interesting place to hash out the ins and outs of continuing medical education (not the politics or funding issues, but the science of art of CME): #CMEChat, a weekly Twitter discussion that takes place on Wednesday mornings at 11 am ET. I don’t always get to participate—for some reason that time seems to get clogged with meetings and conference calls—but when I have joined in, I’ve been really impressed with the level of conversation that happens in 140-character snippets. And now there are videos extolling the virtues of #CMEChat by Brian McGowan and Lawrence Sherman, but this one had me literally laughing out loud.