Data collection: Designing an Observational System

Data collection

1. Copyright © 2014 by The University of Kansas
Data Collection: Designing an
Observational System

2. Copyright © 2014 by The University of Kansas
What is an observational system?
•An observational system is the system you use to
collect the data that you need to analyze in order to
evaluate your program. It details the way you’ll look at
the process, progress, and outcomes of your work, and
how you’ll examine the behavior, conditions, or events
that you’re concerned with.

3. Copyright © 2014 by The University of Kansas
Why design an observational system?
•It can help you get reliable information.
•It can help you find out exactly what you need to
know, eliminating or reducing wasted effort.
•It can ensure that observations are made.
•It can make it easier to analyze your data.
•It can help you avoid haphazard evaluation.

4. Copyright © 2014 by The University of Kansas
•It will make it easier to justify your findings.
•It can help you gain credibility with funders and
policy makers.
•It can let you pass on your practices with
confidence.
•Most important, it can give you the best
information possible about what’s working in
your program, and what you need to adjust.
Why design an observational system?

5. Copyright © 2014 by The University of Kansas
When should you design an
observational system?
•If you can, you should design the system before
your program begins, so that you can watch it
and its effects from the very beginning.
•If that’s not possible, you should design your
system before you start your evaluation, ideally
at the start of a program cycle.

6. Copyright © 2014 by The University of Kansas
Who should design an
observational system?
•Observational systems are usually best
designed by a participatory group that includes
both researchers or evaluators and people who
will do the actual observation.

7. Copyright © 2014 by The University of Kansas
How do you design an
observational system?
1.Review your evaluation questions.
2.Decide what you need to observe to answer your
questions.
•Participants’ behavior.
•Someone else’s behavior.
•Conditions.
•Observations of results of behavior.
•Participants’ knowledge or attitudes.
•Someone else’s knowledge or attitudes.
•Goal attainment.
•Interactions.
•Program process or implementation (e.g., number
of participants).

8. Copyright © 2014 by The University of Kansas
3.Decide how the observations will be conducted.
•Direct observation.
•Participant observation.
•Self-reports, including individual and group
interviews, focus groups, journals, surveys, etc.
•Second-hand reports, including interviews, journals,
surveys, etc.
•Electronic or mechanical observation.
•Tests or other similar observation tools.
•Public records and the like for community-level
indicators.
How do you design an
observational system?

9. Copyright © 2014 by The University of Kansas
4.Decide when you need to observe.
•Pre- and post- observation.
•At regular intervals during the evaluation period.
•At irregular intervals during the evaluation period.
•At specific times during the evaluation period.
•Continuously.
How do you design an
observational system?

10. Copyright © 2014 by The University of Kansas
5.Define and describe the behaviors, conditions,
and/or events that observers should be
concerned with.
6.Train observers in:
•What it’s important to record, and why.
•The definitions and descriptions of the behaviors,
conditions, events, or situations to be observed.
•Effects of observation.
•Observer bias.
•Observer drift.
How do you design an
observational system?

11. Copyright © 2014 by The University of Kansas
7.Devise checks for reliability and accuracy.
•Use an external standard.
•Check for inter-rater reliability.
•Use random third-party checks.
8.Adjust the system for the next evaluation.
How do you design an
observational system?