FCN Presentation - Measuring Communication - Nov 2012

MITRE Principal Consultant Jeff Brooke shared a presentation on how to think about measurement in everyday communication planning.

1. © 2012 The MITRE Corporation. All rights reserved.
Presentation to the
Federal Communicators Network

Jeffrey Brooke, ABC
Principal Consultant
Organizational Communication & Change Management

[email protected]

Presentation approved for Public Release:12-4611. Distribution Unlimited
* A B C – A c c r e d i t e d i n B u s i n e s s C o m m u n i c a t i o n b y t h e I n t e r n a t i o n a l A s s o c i a t i o n o f B u s i n e s s C o m m u n i c a t o r s

Measuring Communication -
Simple Approaches That Can
Change Your Image

2. Agenda
Logistics
–30 minute presentation
–Q&A – via text box
–Presentation will be forwarded

Illustration - a typical communicator dilemma
What to measure
–Conceptual framework for measurement
How to measure
–Other examples and tools


| 2 |

3. Communication measurement
is a way of thinking
Measurement is about:
–Gathering data
–To tell a story
–That allows people to make decisions

Not about:
–Complicated math
| 3 |

4. Start with internal communication
Focusing on internal communication today
An organization’s brand/reputation, comes from the inside out.
Employees treat customers as management treats employees.
–Effective communication improves employee engagement
–Engaged employees deliver better for the customer
Here’s an example.

| 4 |

5. Scenario - Call center at “the Bureau”
| 5 |

6. Call center agent, Jill, answers question and
completes service applications for citizens
| 6 |

7. Goal: communication with Jill will align her
performance with the Bureau’s goals
Communication
Desired
interaction with
citizens
| 7 |

8. Head of call centers needs quality improved
(response accuracy)
| 8 |

9. His e-mails, speeches, newsletters have stressed
quality – but haven’t produced change
| 9 |

10. He wants your help
getting his quality message through
“What other communication should I do?

I thought maybe a special newsletter each month focused
just on quality, like industry best practices.
Then maybe a contest between units and then spotlight
the quality superstars in the newsletter each month.

What do you think?”
| 10 |

11. Stepping back for a moment…
The communication flow – classic model
Communication Flow
Sender Message Channel Receiver Results
Noise
Feedback
| 11 |

12. How does the basic model work in our
large and complex organizations?
| 12 |

13. Communicator
Perspective
Stakeholder Perspective
(e.g. employees)
Client Program Owner
Perspective
Sender Message Channel Receiver Results
Organizational roles that tie
to steps in the model
Communication Flow
Noise
Feedback
| 13 |

14. Planned communication in an organization
Steps that operationalize the model
Communicator
Perspective
Plan Deploy
Communication
Activities
Stakeholder Perspective
(e.g. employees)
Frame of
Reference
Communication
Outcomes
Client Program Owner
Perspective
Stakeholder
Behaviors
Results
Communication Flow
| 14 |

15. Planning & measures are intertwined
Planning starts at the end, works backward
Communicator
Perspective
Plan Deploy
Communication
Activities
Stakeholder Perspective
(e.g. employees)
Stakeholder
Assessment

(Frame of
Reference)
Communication
Outcomes

(Knowledge &
Attitude)
Client Program Owner
Perspective
Stakeholder
Behaviors
Results
Planning Flow
| 15 |

16. Communication planning
Explore measures for all steps, not just the end
Communicator
Perspective
Plan
Measures
Deploy
Measures
Stakeholder Perspective
(e.g. employees)
Stakeholder
Assessment

(Frame of
Reference)
Measures
Communication
Outcomes

(Knowledge &
Attitude)
Measures
Client Program Owner
Perspective
Stakeholder
Behaviors
Measures
Results
Measures
Planning Flow
| 16 |

17. How to Measure Communication
Overview of Key Tools
Qualitative
–Interviews
–Focus Groups (and Structured Group Interviews)
–Polls
Quantitative
–Surveys
–Counting
| 17 |

18. Qualitative Data Gathering
Top tools Best for
understanding

Examples
•Focus groups
•Interviews
•Polls

•Depth—by probing

•To understand the
range of issues on a
topic

•Story-telling nature of
dialogue can uncover
forces around
change—enablers
and barriers to
change
•What people know

•How people feel
about a topic

•Frame of reference
on a topic


| 18 |

19. Quantitative Data Gathering
Top tools Best for

Examples
Surveys

Prediction
•Breadth—how widely
shared is an opinion

•Precise, for tracking
changes over time.
•Comparison over
time, and between
organizations

•Correlation analysis
for prediction

Counting

Outputs

•Number of articles
•Event attendance
•Percentage of
articles dedicated to
each strategy
| 19 |

20. Communication planning
1. Results
Communicator Perspective
Stakeholder Perspective
(e.g. employees)
Client Program Owner
Perspective
Results
Plan What to measure How to measures
Questions to ask:

Outcomes:
•What needs to change and why?
•How will you know if you’ve
accomplished your goal?
Examples:

•Customer satisfaction
•Budget requirements
•Congressional
requirement
Examples:

•Survey
•Financial statements
•GAO/committee
feedback
Outputs:
•What changes in outputs are needed
to achieve the goals?
•How will you measure progress?
•Number
•Quality
•Cost

•Counting
•Sampling
•Cost accounting
| 20 |

21. Results
Outcomes and Outputs
Will be measured by the program owner

Outcome: Higher satisfaction from
–GAO/Congress
–Citizens

Output: Higher quality calls

| 21 |

22. Communication planning
2. Targeted stakeholder behavior
Communicator Perspective
Stakeholder Perspective
(e.g. employees)
Client Program Owner
Perspective
Stakeholder
Behaviors
Plan What to measure How to measure
Questions to ask

•Who needs to do what differently to
produce the output changes?
•How will you measure the behaviors?
•What will change to enable the new
behaviors?

Examples

Actual behaviors
•Applying new training
•Using new procedures

Behavioral indicators

Examples

Business process
measures


Survey questions
| 22 |

23. Behaviors
Not clear

Ron assumes the agents have leeway.
Sees his communication as motivation.

| 23 |

24. What to tell Ron?
One approach
“Yes! I can help you with a quality-focused newsletter and other
ways to get out your message.”

“AND—
–how do you feel about first letting me do some feedback sessions
to get a sense of what will and won’t work. I might find better
options for you to consider.”
| 24 |

25. Communication planning
3. Stakeholder assessment – change readiness
Communicator Perspective
Stakeholder Perspective
(e.g. employees)
Stakeholder
Assessment
1. Frame of
Reference for
the change
Client Program Owner
Perspective
Plan What to measure How to measure
Questions to ask the program owner

Client hypothesis about reaction to the
change:

•What changes will enable the new
behaviors or remove barriers?
•What is the stakeholder’s frame of
reference on what you are asking
them to do differently?
•What might they see as enabling or
blocking them?
Examples

Validate with the
stakeholder:

•Will you be able to do
this, in light of the
planned support?

Examples

Focus group
Interview
| 25 |

26. Conduct stakeholder assessment
Focus groups at call centers
| 26 |

27. Finding:
Quality measured quarterly, at the office level
| 27 |

28. Quantity Metric–
Every 30 minutes, Group Level
Finding:
Quantity measured every 30 minutes, at the group level
| 28 |

29. Frame of reference determines
what Jill will notice, how she will interpret it
Communication

Frame of
Reference

(Enablers & Barriers)

Resulting
interaction
with citizens
| 29 |

30. Stakeholder assessment:
Focus groups determine Jill’s frame of reference
Frame of
Reference

(Enablers &
Barriers)

Elements of frame of
reference

•Leadership
•Mission & Vision
•Culture
•Mgt. practices
•Policies &
procedures
•Work unit climate
•Individual needs
•Task & individual
skills

(See Burke-Litwin model)
Resulting
interaction
with citizens
| 30 |

31. What would you tell Ron?
What would you recommend
to him as a next step?
Might this change how Ron
sees you?

What to tell Ron?
| 31 |

32. Communication planning
3. Stakeholder assessment – communication fitness
Communicator Perspective
Stakeholder Perspective
(e.g. employees)
Stakeholder
Assessment
2. Frame of
Reference on
communication
Client Program Owner
Perspective
Plan What to measure How to measure
During deployment:
How to assess the effectiveness of each
communication activity during
deployment.

Past experience:
Usefulness of communication channels
and approaches with this stakeholder.

“Fitness” of communication

Examples:
- Clarity
- Right channels
- Amount
- Timing
- Exposure


During deployment

•Survey
•Poll
•Focus group

Past experience
•Compilation of
previous data
•Communication audit

“Fitness” measures.
What do they think of
our stuff? Does it “fit”
their needs?”
| 32 |

33. Example:
Ongoing assessment of "fitness" of messages

Weekly survey of employees and managers
Hottest topic of the week
Statistically reliable sample
| 33 |

34. Example "Fitness" measures
Message penetration and usefulness
| 34 |

35. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Q7. How would you rate the
amount of communication you've
received on this topic? (Running
average for "About right" is 68%)
Q6. How important is the topic of
"new system procedures" to you?
(Running average for "Important" is
65%)
Q5. Overall, how satisfied were
you with this communication?
(Running average for "Satisfied" is
86%)
Q4. To what extent do you feel
this communication was timely?
(Running average for "Timely" is
82%)
Timely
Satisfied
Important
About right Too much Too little
Unimportant
Dissatisfied
Not timely
Example "Fitness" measures
Timeliness, satisfaction, importance, amount
| 35 |

36. Communication planning
4. Communication outcomes
Communicator Perspective
Stakeholder Perspective
(e.g. employees)
Communication
Outcomes

Knowledge &
Attitude
Client Program Owner
Perspective
Plan What to measure How to measure
What knowledge and attitude will
produce the targeted behaviors, as seen
through the stakeholder’s frame of
reference?
Examples

•Knowledge and attitudes
(perceived)
•Actual knowledge

Examples

•Survey
•Poll
•Focus group
| 36 |

37. Example knowledge measure
| 37 |

38. Example measure of feelings
9%
23%
17%
20%
31%
13%
38%
25%
0%
25%
0%
5%
10%
15%
20%
25%
30%
35%
40%
A great deal Somewhat A little None I’m not sure yet.
To what extent do you feel the new Performance Plan
system will help the agency reach its goals?
Employees
Sups
| 38 |

39. Example: Measuring knowledge and feelings
for course correction
Situation:
–Executive Committee interactive video town hall
Topic:
–Transformation update – two years into effort
Update on each of five change strategies
Scripts weighted heavily on “why this strategy is important”
Measurement Plan
–Pre-event random survey – two questions for each strategy
Know: “How important do you feel this is to the agency transformation?”
Feel: “How effectively do you think this strategy is being implemented?”
–Recommend possible script changes
–Post-event repeat survey
| 39 |

40. | 40 |
Measuring knowledge and feelings
Pre-event survey (two days prior)
Several strategies showed:
•good scores for importance
(feeling)
•lower scores for
implementation (knowledge)

Comments indicated lack of
knowledge about early
implementation efforts and
results.

Updated script for these
strategies to weight more heavily
on pilots and early
implementation efforts.

41. Measuring knowledge and feelings
Post-event survey (one hour after event)
Scripts adjustments were
successful.
| 41 |

42. Communication planning
5. Deploy communication activities
Communicator Perspective
Deploy
communication
activities
Stakeholder Perspective
(e.g. employees)
Client Program Owner
Perspective
Plan What to measure How to measure
What types of communication will best
convey the targeted knowledge and
attitudes?

Communication Outputs

Examples:
•# of articles and events
•% of content supporting
each business strategy
•% of stakeholders who
provide input on a change
•# of awards for a desired
behavior
•# informal leader
conversations
Examples

•Counting

| 42 |

43. Targeting value change with
messages
Primary communication activities
Content, such as intranet
pages, publications and formal
presentations.

Interaction, such as input to
decisions, dialogue with
managers, and collaboration
with colleagues.

Leadership example, such as
how leaders signal what they
value through promotions,
recognition, budget, their time
and informal comments.
| 43 |

44. Output measures
Content analysis – weekly data collection
| 44 |
Themes Totals
% of total
editorial
space E-Bulletin Managers Bulletin Magazine Webbie
Total Length 66586 37702 15582 7289 6013
Vision 222 0% 0 0% 222 1% 0 0% 0
Mission 239 0% 239 1% 0 0% 0 0% 0
Values 12026 18% 2249 6% 498 3% 6165 85% 336
Value: Integrity 208 0% 0 0% 0 0% 0 0% 0
Value: Teamwork 10712 16% 2053 5% 498 3% 6165 85% 336
Value: Commitment 1423 2% 196 1% 0 0% 0 0% 0
Value: Dependability 389 1% 0 0% 0 0% 0 0% 0
Goals 39169 59% 16872 45% 11046 71% 2330 32% 1911
Goal 1: Develop a flexible
digital… 3421 5% 1092 3% 0 0% 1284 18% 0
Goal 2: Prepare and
equip… 6601 10% 3694 10% 1076 7% 1046 14% 0
Goal 3: Continue to
transition the… 784 1% 567 2% 0 0% 0 0% 0
Goal 4: To serve the
public's needs by… 892 1% 342 1% 352 2% 0 0% 23
Goal 5: Develop an
integrated enterprise
approach to… 32496 49% 14603 39% 10694 69% 0 0% 1888
Quotes 16629 25% 5112 14% 3973 25% 7090 97% 315
Photos/Images 497 55 9 24 25
188 63 10 60
Total articles or services 786

45. Example output measure: Percentage of total editorial space,
by category, for all six channels 1%
0%
6%
28%
25%
27%
14%
17%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Vision Mission Values Goals Strategic
Priorities
AdministrativeRecognition Quotes

46. Example output measure: Percentage of total editorial space,
by category, for E-Bulletin 1% 1%
3%
25%
19%
23%
8%
13%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Vision Mission Values Goals Strategic
Priorities
AdministrativeRecognition Quotes

47. Example output measure: Percentage of total editorial
space, deep dive in “goals” coverage 28%
3%
5%
2% 2%
25%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Goals Total Goal 1: Develop
a…
Goal 2: Prepare
and equip…
Goal 3: Continue
to transition the
…
Goal 4: To serve
the public's
needs by…
Goal 5: Develop
an enterprise
approach to…

48. Communication Measures
Integrated with Communication Planning
Communicator
Perspective
Communi-
cation
plan
elements
Measures
for each
element
5.
Tactics to
Deploy

Communication
Outputs

Stakeholder
Perspective
3.
Stakeholder
Assessment
(Frame of
Reference)

Change
Readiness




Communication
Fitness

4.
Communication
Outcomes

Knowledge
and
Feelings
Client Program Owner
Perspective
2.
Stakeholder
Behaviors

Specific
behaviors
needed

Changes that
will enable the
behaviors


1.
Results

Outcomes
and outputs
| 48 |

49. Does anyone really care?
Measures can move your “brand” from overhead, toward operations.

Communications Executive Council survey: 2012 vs. 2007 data

Responsibilities of communication function
–#1 increase: Social Media 80% (from zero)
–#2 increase: Measurement 82% (from 64%)

Top communication strategic priorities for 2013
–#1: Employee engagement
–#2: Improving leadership communication
–#3: Building company presence in social media

Strategies to improve communication function effectiveness
–#1 Improved consultative partnership with business leaders
–#2 Deeper business acumen on the team

CEC 2012 report: Taking Your Communications Budgeting and Planning to the Next Level
| 49 |

50. What next?
Deeper topics
–Design and use tools for valid and reliable data
–Communication audits
Broader topics
– Looking at measures across organizations and time
–What communication function can do to add value

Feedback on today’s presentation
Reach out to me and FCN community
Q&A

| 50 |