There are quite a few ways to conduct a formative evaluation:

through pilots, focus groups, committee reviews, or informally when a colleague proofreads a document. (See the sidebar on p. 116 for additional methods.) Whatever route(s) you choose for formative and summative evaluation, asking the right questions will yield rich information, not just data. Here's how to get started.

New Treatment of Information
Ask relevant questions of participants - questions that address how the meeting fits into the larger context of organizational goals and objectives. Revisit learner outcomes, and ask questions that assess whether the meeting content was transferred and, better yet, integrated. As you stay learner-centric, the questions naturally fall into a higher-level "chain of impact" evaluation as follows:

• Level 1/Reaction - How relevant was the speaker?
• Level 2/Learning - How knowledgeable are you now about the strategies presented?
• Level 3/Application - What have you integrated into your job?
• Level 4/Business Results - Has productivity improved because of the meeting?
• Level 5/Return on Investment - What were the total costs of the meeting (taking into account the salaries of participants, development costs, travel expenses, etc.)? How do the business results translate into dollar results to balance against those costs? What are the intangible (not measurable) benefits?

As you can see, the upper-level questions will yield answers that offer a sense of the meeting's impact on the participant's organization. And while the first two questions can be asked immediately following the meeting, the questions at Level 3 and above should be part of a follow-up survey, in order to allow enough time for participants to gauge the meeting's impact on both their day-to-day role and their overall organization.

It's important to use terminology consistently in the evaluations. Use the term "ROI" only when truly measuring return on investment (typically suggested for only five to 10 percent of all meetings), not when you're trying to measure learning. Creating a new acronym (such as ROE, for "Return on Events") does little to help position meeting professionals in mainstream business.

When you demonstrate that you have used learners' feedback from prior evaluations to shape the current meeting, it conveys the message that their feedback is valued - and makes them more inclined to complete the current evaluation.

Recognition That Learners Are More Challenging to Reach
The ease of conducting surveys on the Internet helps meeting planners to conduct fast, effective surveys, and to justify their cost. "In 2003, prior to the introduction of online surveys, 26 percent of attendees completed the evaluations," said Jodie Ann Cady, CMP, with the Michigan Association of Realtors. "In 2008 - by keeping our surveys short, focused, and easy to complete over the Internet - 70 percent of attendees completed the online evaluations. The increased response rate provides us with current data from active members, which we use to develop future programs. We use respondent feedback for program development, which allows us to meet our members' needs. We also gain specific knowledge regarding the value our members place on exhibits and sponsors, which we can use when recruiting new exhibitors and sponsors."

Higher Expectations of Meetings
Meeting planners should not waste valuable "real estate" on surveys with questions that don't serve their learners or stakeholders. It's too late to ask whether the room was comfortable or the chicken was tasty. Those are basic needs that should have been met; your stakeholders are more interested in whether learners learned.

That leaves meeting planners to ask questions that focus on what their learners care about:

• professional or personal development
• valued information
• networking
• group dynamics
• informal learning time
• engagement of the audience
• speakers who facilitate versus preach
• emotional connection
• reflection, integration, or action-planning opportunities.

It's also important to ask the kinds of questions that interest stakeholders:

• Were the right people (participants) in attendance?
• Did the meeting inform the participants (according to the meeting's purpose)?
• Did the meeting help motivate participants to perform at their best?
• Were participants equipped to communicate the message of the meeting?
• Can participants effectively apply the knowledge and skills learned at this meeting?
• What new activities do participants plan to start? What activities do participants plan to stop?
• Did this meeting give participants ideas to help them positively influence:
• customer/member satisfaction
• employee/member engagement
• productivity
• sales/retention
• quality
• cost
• efficiency
• time savings?

Make an effort to find out what your trade-show exhibitors are interested in learning about your attendees, and give them the tools to validate their participation in your event. "Since exhibitors pay a large sum of money to exhibit, justifying their booth cost or any other convention cost helps show management define booth pricing," said Michele S. Brouse, CMP, CASE, director of client relations at CompuSystems Inc. "Offering services such as lead retrieval makes it easier for exhibitors to determine their return on objectives. The more leads they receive, the more sales they can expect. More sales justifies returning to the show as exhibitors in the following years."

Meetings Seen as an Experience
Yes, learners expect meetings to provide them with an "experience," but the sizzle has to support the substance. u The multisensory environment must promote learning at the conscious and subconscious levels. Evaluating the experiential aspects allows a meeting planner to judge what worked and what needs to be improved in the future, and involves asking the following:

• Did we meet participants' needs, whether felt or expressed, current versus anticipated?
• Did the approach (task-centered, topic-centered, or problem-centered) provide both knowledge and "an experience" for participants?
• Was information sequenced properly?
• Did social and informal exchanges enhance the experience of learning?
• Did emotional components help achieve the goals and outcomes?
• How did the rational, educational, and serious elements combine to create an experience? u Did the meeting balance content, the delivery environment, and reinforcement strategies?
• Will the meeting extend beyond the close of the official event through technology, informal conversation, or new communities?

The evaluation process is not based on one single set of questions or formulas, but rather depends on the individual objectives and outcomes desired from a meeting. While there is no silver bullet for evaluations, if you lack insightful learner feedback, how can you be sure your next meeting is on target?

Funnels for Feedback
Sources for formative and summative evaluation include:

• interviews
• incident reports, or reports of performance discrepancies
• telemarketing
• focus groups
• sampling or pilot groups
• questionnaires
• blogs, listservs, or discussion boards
• Web site questions/polls
• kiosk questions
• comments from stakeholders (meeting planner, supplier, sponsor, participant, facilitator, speaker).

Takeaways From This Year-Long Series
In many organizations, meetings and education are separate departments, each managing its own responsibilities. Educators think one way; meeting planners approach meetings another way. Throughout the How Adults Learn, Now series this year, we've offered meeting planners ways to incorporate adult learning in the design of their events. Here are some key takeaways from the four-part framework we've used throughout the series.

New Treatment of Information
Content must be integrated into the environment in a way that encourages participation and engagement. Planners must prepare room sets that promote participation. This directly affects lighting (no darkened rooms), technology (utilizing interactive technology tools, not simply PowerPoint), and space requirements. Planners must seek venues with flexible space for formal and informal learning.

Recognition That Learners Are More Challenging to Reach
Because participants are starved for time and overwhelmed by information, planners should create highly focused agendas that offer shortened attendance options. All marketing materials should answer the "WIIFM" (what's in it for me?) question. Unusual, 3D, or highly visual marketing materials may help cut through the clutter in participants' in-boxes. Younger generations seek programming that is more interactive and peer-to-peer, and less reliant on traditional subject-matter experts.

Higher Expectations of Meetings
Meeting planners are expected to produce high-impact meetings consistently and with reliability. This requires that meetings be designed from a holistic perspective that takes into account how participants will experience the event. Meeting agendas should be structured to promote interaction between and among participants, presenters, and content. Logistics play an important role - both as the link between strategy and education and in efforts to create the optimal learning environment.

Meetings Seen as an Experience
The meeting experience starts even before participants set foot in the destination or venue. Meeting planners can audit every step of the meeting with an objective eye toward whether the experience is positive, negative, or neutral. Close coordination with all supplier partners helps build a positive experience. When all suppliers understand the objectives and desired outcomes, they can work in concert to create a cohesive experience.

Sue Tinnish is principal, SEAL Inc., which focuses on improving the content of meetings for associations and corporations. She is director of the Accepted Practice Exchange (APEX) program.

Glen C. Ramsborg, Ph.D., CMP, is senior director of education for PCMA.

The How Adults Learn, Now series is sponsored by the Hiltons of Chicago, www.hiltonfamilychicago.com.