It’s long been said that opinions are like bellybuttons—everyone has one. But until recently, not everyone had a bully pulpit from which to shout those opinions to the world at large. Now that pretty much every site on the Web is asking us for reviews, comments, thumbs-ups, Facebook “likes,” Twitter retweets, and on and on and on, are we better informed decision-makers? Or is this review-heavy culture making us, well, stupid and lazy? That’s what I got to thinking about after reading Chris Colin’s Wired Essay, Rate This Article: What’s Wrong with the Culture of Critique. He says:

“Our ever more sophisticated arsenal of stars and thumbs will eventually serve to curtail serendipity, adventure, and idiotic floundering. But more immediate is the simple problem of contamination. When the voices of hundreds of strangers, or even just three shrill ones, enter our heads, a tiny but vital part of ourselves is diminished…There’s an essential freedom in being alone with one’s thoughts, oblivious to and unpolluted by anyone else’s. Diminish that aloneness and we start to doubt our own perspective.”

To which I say, are we people or sheep?

It’s just one more source of information to add to our decision-making arsenal, IMHO. And one that can be pretty useful. From my own little data point of one, we ate at a restaurant we earlier had dismissed as a possibility after reading reviews on Yelp that persuaded us that what was inside was far better than the outside. And it was fabulous. I regularly depend on reviews of nearly impossible-to-find adequately supportive clothing items (ladies, you know what I mean) to hear if they live up to their advertising when actually worn by human females in a gym. And when I bought my long longed-for outback hat, I scoured the reviews to find one that not only sounded and looked good, but whose reviewers were people who wore the hat doing the types of things I would wear it doing. You don’t get that in a catalog. I’m sure there’s no shortage of opinion spam, which research suggests we may not be so good at detecting, but overall I believe that people want to be helpful when they write reviews, and overall that’s been my experience. I think it enhances, not diminishes, the decision-making experience.

We don’t cede our decision-making to strangers by allowing their opinions to inform it. Reviews are just another factor to consider. And keep in mind that you may not agree with even the most august and respected of official reviewers (I actually read a glowing newspaper review of Cowboys & Aliens, which was one of the most boring movies I’ve seen in a long time). As President Reagan would say, “Trust, but verify.”

What I worry more about is that, if we come to rely more and more on trusted review sources, we will in fact “curtail serendipity, adventure, and idiotic floundering,” which is where I tend to find the most interesting things. But the flip side is that reviews may in fact encourage serendipity by pointing us toward things we otherwise might not have considered, as was the case with our Yelp-inspired restaurant choice.

While you may—and I hope you do!—invite attendees, speakers, and anyone else involved in your event to rate, review, and comment on your offerings, I’d hope everyone will use the opinions as just one more factor to consider, not an excuse to check their brains at the door.

I read this post about the trouble with trying to get meaningful information out of evaluation forms on Confessions of a Medical Educator this morning (and had to post this response to it on our Capsules blog). But while this problem is especially acute with those in continuing medical education who are required in many cases to get some meaningful data on what attendees learned and how they use what they learn, I can’t imagine there’s anyone involved in continuing adult education in any field who doesn’t share this concern (in fact, I dug through our archives and came up with this article from Financial & Insurance Meetings providing some tips).

You may not be familiar with Moore’s 7 Levels for CME Outcomes Measurement if you’re not working with healthcare providers, but I think it could be adapted easily for other niches as well, and is a good starting point for how to think about measuring learning outcomes. May Dr. Moore forgive me if I end up mangling his pyramid, but my translation/added commentary for those who provide continuing adult ed outside of the medical arena would be:

1. Participation–do they show up?
2. Satisfaction–is the room temperature comfortable? The food good? The lecture not too snooze-inducing?
3. Learning
3a. Declarative knowledge–can they tell you want it was that they were supposed to learn?
3b. Procedural knowledge–do they now know how to do what they learned?
4. Competence–if you gave them a test or asked them to demonstrate what they learned while still at the session, could they?
5. Performance–what do they now do differently back at the office as a result of what they learned?
6. Client benefit–what has changed for the better for their clients/customers/employees now that they’ve made these changes in how they work based on what they learned?
7. Community improvement–how has the community at large benefited from improvements in the client population?

Pretty daunting stuff, especially when you start getting into the outer circles of levels 5-7, which is difficult enough to measure in medical settings and may in fact be almost impossible for some other professions. Still, I’ve never been handed an evaluation that shot higher than level 2, but I think most adult education should be shooting for–and trying to measure results–at least levels 3-4.

The always-brilliant Jeffrey Caufade has some ideas on how we can begin to get there, including being sure to ask these two questions, and to design an activity that provides learning worth making a change for. Then there’s this article I wrote a while back that, while aimed specifically for CME providers, has some ideas I think anyone could use to improve the value of the evaluation process. I know there must be a ton more on this topic that I’m just not coming up with in a quick skim of the top of my head (please post links in the comments!).

What do you do to ensure your evaluations are meaningful and not just “smile sheets”? Do you (and do you want to) measure the outcomes of the education you provide to see what people actually walk away with and if/how they change what they do because of it? Or are smile sheets good enough (and is that the reason we call attendees attendees and not learners)?