DevelopingHighQualityBaseline.ppt wildlife

wildlife baseline data

1. Developing a High
Quality Baseline
Salimah Samji & Mona Sur
World Bank, New Delhi
June 21, 2006

2. Overview
What is a baseline?
Why you should care
The phases of conducting a baseline
The errors to avoid in each phase
How to manage the common errors
Elements of a baseline survey (TOR)

3. What is a baseline?
Fixing the time at the base – a benchmark
from which you measure progress
Snapshot of indicators at a time
Instrument used to:
Test hypotheses of project (assess results)
Planning (refine targeting, indicators to
monitor)

4. Why you should care …
To identify whether there were any benefits for
the investments made
Were objectives met?
What factors explain the result?
How can the program be improved?
Compare alternative models to get the biggest
bang for your buck
To inform next generation projects
Evidence-based policy making – demonstration
effect for government

5. The phases of conducting a baseline
Design
Implementation – actual survey
Data entry and analysis
Report writing

6. Phase 1: Design Phase

7. P1: Check-list
Clear objectives (what is the problem?)
Clear idea of how you will achieve the
objectives (causal chain or hypotheses)
Clear and measurable indicators
Clear (precise and unambiguous)
Relevant (to objectives)
Monitorable

8. Example of a causal chain
Impact
Outcomes
Outputs
Activities
Inputs Facilitators, Revolving fund (credit)
Forming, federating and organizing SHGs
Number of SHGs, decreased input prices
Increased use of credit for income generation, Loan
repayment rates
Higher income levels
Example: APDPIP

9. P1: Check-list (cont’d)
Design survey instrument – keep it simple and
related to the objectives and hypotheses you
want to test
Link surveys to GIS – use consistent units
Select controls/counterfactuals to attribute
change (causality)
Timing of baseline
Before project (what if project never materializes)?
2 years into the project (intervention has begun)?
Other factors (seasonality)

10. P1: Check-list (cont’d)
Sampling strategy
Random allows inferences about a
population
Random
Stratified random (include groups
which could be excluded)
Non-random – use a group
smaller than the population

Introduces selection bias. Note: every
stratification introduces a level of
bias.
Population
Size
Sample
Size
10 10
50 44
100 80
500 217
1,000 278
3,000 341
50,000 381
1,00,000 385

11. P1: Check-list (cont’d)
Over sample for attrition
Design the database system for data entry
Translate the questionnaire
Back translate to verify
Provide adequate training on
The objectives and importance of the study
How the sections are linked and what the questions
mean
Attend the training if possible (stay involved)

12. P1: Check-list (cont’d)
Test the questionnaire
Ask yourself, can you answer these
questions?
Are they relevant to the outcomes?
Will they be understood?
Field test: To test both how the surveyor
administers the instrument AND how the
respondent understands the question.

13. P1: Why it is important to field test (i)
When asking a question about the level of
awareness, the surveyor used a word that could
mean awareness or knowledge – the
respondent understood it to mean education. :
The question was: “Ram/ Gita knows about everything that happens
(vikas) in the village. For instance, they know [the name of the
sarpanch, when and where the Gram Panchayat meets, nature and
type of development work in village, etc.]”

14. P1: Why it is important to field test (ii)
CASE STUDY: Domestic Violence Study in India
“In studying domestic violence, a question in the survey instrument asked if
female respondents had ever been beaten by their husbands in the
course of their marriage. Only 22 per cent of the women responded
positively to this question – a domestic violence rate much lower than
studies in Britain and the US had shown. In probing the issue with in-
depth interviews we discovered that the women had interpreted the word
‘beating’ to mean extremely severe beating – when they had lost
consciousness or were bleeding profusely and needed to be taken to the
hospital. Hair pulling, ear twisting, etc, which were thought to be more
everyday occurrences, did not qualify as beating. Reponses to a broader
version of the abuse question, comparable to the questions asked in the
US and UK surveys, elicited a 70 per cent positive response.”
Source: Vijayendra Rao (1998) – “Wife-Abuse, Its Causes and Its Impact on
Intra-Household Resource Allocation in Rural Karnataka”

15. Phase 2: Implementation

16. What would you do if you …
Were a city person who didn’t speak the local
language very well
Had to travel to several villages and spend
hours asking people questions that have no
relevance to you.
Were paid a small sum per questionnaire
Not monitored by supervisors
This is not your full time job

17. The answer is simple
Sit at home or in a bar and fill
out the questionnaires!!

18. P2: Providing incentives and motivation
Sub-contracting surveyors from the state
who speak the language
Include women surveyors
Include a supervisor who conducts data
scrutiny
If possible pay reasonable wages
Randomly verify questionnaires to reduce
the likelihood of false responses (inform
them beforehand - during the training)

19. Phase 3: Data Entry and Analysis

20. P3: Check-list
Data Entry
Make the data entry system as fool proof as
possible - has unique identifiers to link both
household, village and GIS data
Ensure database allows for merging of data
Do not change/erase data on questionnaires
Raw data should always be input as is, changes
can then be made in the database software
(programatically) with documentation

21. P3: Check-list (cont’d)
Often data entry is contracted out.
Name variables corresponding to the question and section in the
questionnaire – include a dictionary
Code descriptive answers (to facilitate analysis)
All fields should be filled (NA or NR)
Units should be uniform by district
Totals calculated by formula not from summary column
Consistency checks – check for missing entries, wrong
entries, sample statistics, patterns (queries should be
inbuilt)
Validity checks – similar questions in different places on
the questionnaire (RCH example)

22. P3: Check-list (cont’d)
Data analysis
Common mistakes in interpreting data
No analysis!
No correlations, crosstabs, statistical significance levels or
regressions
Over generalizing the results
Mis-reporting statistics
Using % when the numbers are small
Attributing causality when it is not demonstrated

23. Phase 4: Report Writing

24. P4: What the report should be …
Simple, Clear and Relevant
State limitations (attribution, causality)
Major findings should be upfront
Focus on quality rather than quantity
Technical details in an appendix
Should always
include the questionnaire in the appendix
ask for electronic copy of data
Request copies of filled out surveys
Essential if you change consultants at midterm
or want to conduct internal analysis to compare
modes of delivery (data lost example).

25. How to manage the common errors
Phase 1: Design
Clear objectives and hypotheses – know what you
want to test
Identify a person in your unit who will manage this
process
Write a good TOR, remember the baseline determines
the quality of your panel
You can add questions as project evolves but cannot
change questionnaire – threat to internal validity
Identify consultants

Procurement – focus on quality not the cheapest bid “if you
throw peanuts you’ll attract monkeys”

Ideally you should have a black-list of organizations

26. How to manage the common errors
Phase 2: Implementation
Organize an impact evaluation workshop if
necessary
Randomly verify questionnaires to reduce the
likelihood of false responses (no filling it in a
bar)
Pay reasonable wages to surveyors (if
possible)
Show the client and firm that you care

27. How to manage the common errors
Phase 3: Data entry and analysis
Double-data entry (2 separate organizations and
verify. Payment based on quality of data entry)
Select 15 questionnaires at random and check data
entry – person in your unit managing
Check data quality (consistency and validity checks)
Hold an IE workshop to build data analysis capacity (if
necessary)

28. How to manage the common errors
Phase 4: Report writing
Agree on an outline beforehand
Dedicate a chapter on indicators you are
tracking
Focus on quality not quantity
Think “Big Picture”

29. Elements of a Baseline Survey
Terms of References
1.Background: Project objectives and components
2.Survey design: Consult a sampling expert!!!
3.Survey instruments
4.Guidance on survey implementation
5.Data processing and analysis
6.Staffing
7.Duration and time schedule
8.Submission of reports and datasets
9.Support to the firm
10.Budget & Payment Schedule
11.Annexes: Draft questionnaires, Results Framework

30. Baseline Survey Design:
Typical Tasks for Consultants
Recommend the methodology for sampling
Calculate the optimal sample size
Develop the sample frame and select the sample
The final sample and details of the statistical
methodology used to select the sample need to
be cleared by the project
Construct the sample weights and provide
documentation on the methodology used to
construct the weights

31. Survey Instruments:
Questionnaires
Design or refinement and adaptation of the data
collection instruments
Specify levels of data collection
Length of questionnaires
Prepare all support documentation including coding
guides, interviewer and supervisor manuals and the
data entry manual
Translation and back-translation
Skip patterns, coding open ended questions

32. Guidance on Survey
Implementation
Implementation plan
Selection and training of field workers: specify
minimum duration of training
Pilot testing should be explicitly specified in ToR
Responsibility for all field operations, including
logistical arrangements for data collection and
obtaining household consent lies with Consultants.
Ask for field-work progress reports
(bi-weekly/monthly)

33. Staffing
Sampling expert/statistician
Technical specialists as relevant
Economist
Sociologist.
Core survey staff: the survey manager, the
field manager, the data manager
Enumerators, supervisors and data entry staff

34. Baseline Report & Data
Explicitly request final electronic datasets
—with complete documentation.
Agree on outline of baseline report up-
front.

35. Managing a Baseline Survey
Consult the experts—survey specialist and
sampling specialist and develop the ToR in
consultation.
Selection committee should include a survey expert
and social scientists in addition to technical experts.
You can never over-supervise!!! Hire third-party
supervision consultant if needed.
Question the data and the findings.

36. Lets recap what you have learned
The devil lies in the detail
Be watchful
No pain, No gain