HSME_1109 Notes 1.pptx monitoring and ealuation notes

M n E Social

1. DATA COLLECTION METHODS HSME_1109

2. What is monitoring and evaluation? A guide to the basics The development sector loves to measure — and maximize — its impact. Every penny spent is meant to reach beneficiaries, directly or indirectly. That means one of the biggest challenges is using limited resources to reach as many people as possible. How can development organizations stretch their resources? The best way is to make sure that every cent, person, and chunk of time is used optimally. This is where monitoring and evaluation comes into play. More recently, as issues of accountability and transparency become more critical in the private and public sector, monitoring and evaluation have also become a central focus in these sectors. Monitoring and Evaluation (M&E) is a continuous management function to assess if progress is made in achiev­ing expected results, to spot bottlenecks in implementation and to highlight whether there are any unintended effects (positive or negative) from an investment plan, programme or project (“project/plan”) and its activities.

3. What is monitoring and evaluation? What Is Monitoring ? Simply put, monitoring is an ongoing process of collecting and analyzing data to check a program’s efficiency. This data is used to plan, monitor and improve programs . For example, a program providing nutritious meals to school children to reduce prevalence of anemia may do monthly monitoring checks on several factors : Number of children fed (to make sure that targets are met) If children are falling sick (to check for quality of food) Number of people in the field (to know if there is enough manpower to run the program efficiently) Number of parents providing nutritious food to their children (to see whether the program has led to positive outcome, such as behavior change) Tests for aneamia (to check for effectiveness of the program)

4. What is monitoring and evaluation? Typically, monitoring seeks to answer three (3) questions:

5. What is monitoring and evaluation? There are three key elements of monitoring : Continuous process : monitoring is also called process evaluation because it is a continuous process that runs through the entire program. Usually, there is a dedicated M&E officer who handles this process. Regular data collection : data is collected at regular intervals (monthly, bi-monthly, or quarterly, for example) using a pre-set questionnaire, which has metrics that are decided at the beginning of the program. Data is also collected at the beginning of the program, which serves as a baseline. Identify gaps in implementation : monitoring data is extremely important because it helps the program make adjustments during the implementation phase. Monitoring helps identify gaps that keep the program from making maximum impact. Changes during implementation help development organizations get better results and use all their resources better.

6. What is monitoring and evaluation? What Is Evaluation? Evaluation is the process to check whether a program has met its objectives. There are several key features of evaluation:

7. What is monitoring and evaluation? For example, say that a non profit wants to increase coverage of hepatitis C vaccination in a village. They start a program providing free vaccinations at health centers and through door-to-door services. Evaluation would measure how many people are vaccinated before and after the program to see if coverage has increased. What if the government is simultaneously running an awareness campaign to encourage Hepatitis C vaccinations? How can the non profit know how much of the change in coverage is caused by their program, and how much is caused by the government’s program? This is where the control group (which isn’t covered by the non profit’s program) comes in. Say the control group’s coverage increases by 20% and the target group’s coverage increases by 50%. Then the non profit would know that the government’s program led to a 20% increase in vaccination, and their program led to a 30% increase in vaccination.

8. What is monitoring and evaluation? Monitoring and evaluation are synergistic. Monitoring information is a necessary but not sufficient input to the conduct of rigorous evaluations. While monitoring information can be collected and used for ongoing management purposes, reliance on such information on its own can introduce distortions because it typically covers only certain dimensions of a project's or program's activities, and careful use of this information is needed to avoid unintended behavioral incentives. In contrast, evaluation has the potential to provide a more balanced interpretation of performance. But evaluation is a more detailed and time-consuming activity, and because of its greater cost it needs to be conducted more sparingly. One approach is to rely on monitoring information to identify potential problem issues requiring more detailed investigation via an evaluation.

9. What is monitoring and evaluation? Monitoring is conducted after a programme has begun and continues throughout the programme implementation period. Monitoring is sometimes referred to as process, performance or formative evaluation. (Adapted from Gage and Dunn 2009, Frankel and Gage 2007, and PATH Monitoring and Evaluation Initiative ) An evaluation should provide evidence-based information that is credible, reliable and useful. The findings, recommendations and lessons of an evaluation should be used to inform the future decision-making processes regarding the programme.

10. Importance of data in m & e M&E involves collecting data, monitoring key indicators of a program, and evaluating whether it has met its objectives. M&E isn’t possible without a monitoring and evaluation plan. This is a document that includes the objectives of the program and the activities designed to achieve them. An M&E plan outlines the procedure that will be used to evaluate whether or not the objectives have been met. It should include the data that will be collected, the method of collection and analysis, how the data will be used, and the resources that will be required to implement this plan. The whole M&E plan must be aligned to the overall goals of the project. For example, if the program goal is to increase the number of school-going children in a district of Uttar Pradesh, India, every element of the M&E plan will be designed with this objective in mind. A M&E plan is most effective when it is desig

11. Importance of data in m & e A M&E plan is most effective when it is designed at the beginning of the program. This helps scope and allocate the required resources right at the start. However, the plan should be an ever-evolving process, and it should be revised if there are changes to the program during evaluation. A M&E plan is critical because it: Helps an organization make informed decisions about their program based on evidence. Identifies ways to use resources more effectively and efficiently. Helps identify the impact of the program, gaps in implementation, and things that worked successfully. Provides data that can help convince donors to invest more or help program officers devise alternative approaches to address their problems.

12. Importance of data in m & e A M&E plan has the following components :

13. Importance of data in m & e It is very important to update the M&E plan as and when there are changes in the program that affect the original plans . While developing a M&E plan, keep these guiding principles in mind:

14. Importance of data in m & e Data is the heart and soul of monitoring and evaluation (M&E). Valid , reliable and accurate data can reveal and improve the performance and impact of your intervention and support decision making and learning, while enhancing your credibility and accountability. Data is divided into different categories based on how you source them and the techniques you employ to gather and analyse them.

15. What is data? In the pursuit of knowledge , is a collection of discrete values that convey information, describing quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted. Data, as a general concept, refers to the fact that some existing information or knowledge is represented or coded in some form suitable for better usage or processing. Data is the smallest units of factual information that can be used as a basis for calculation, reasoning, or discussion . There is a subtle difference between data and information. Data are the facts or details from which information is derived. Individual pieces of data are rarely useful alone. For data to become information, data needs to be put into context.

16. What is data? DATA INFORMATION MEANING Data is raw, unorganized facts that need to be processed. Data can be something simple and seemingly random and useless until it is organized. Example : Each student's test score is one piece of data. When data is processed, organized, structured or presented in a given context so as to make it useful, it is called information. Example : The average score of a class or of the entire school is information that can be derived from the given data.

17. Data collection Organizations collect data to make better decisions. Without data, it would be difficult for organizations to make appropriate decisions, and so data is collected at various points in time from different audiences . Data collection is a systematic process of gathering observations or measurements. Whether you are performing research for business, governmental or academic purposes, data collection allows you to gain first-hand knowledge and original insights into a particular problem. While methods and aims may differ between fields, the overall process of data collection remains largely the same. There are a number of ways to collect data but there is no one single best way.

18. Data collection Before one may commence data collection there are several considerations: What you need to know W here the data resides R esources and time available C omplexity of the data to be collected F requency of data collection . To collect high-quality data that is relevant for evaluation purposes, there are critical steps that need to followed.

19. Data collection Step 1: Define the aim Before you start the process of data collection, you need to identify exactly what you want to achieve, that is, what is the practical or scientific issue that you want to address and why does it matter? Next , formulate one or more questions that precisely define what you want to find out . The starting point for any evaluation must be the use and users of information. If the evaluator is not given a question or set of questions to answer from the major stakeholders - the organization itself or a funding agency then these must be developed . This is done ideally in consultation with stakeholders and agreed upon before the evaluation design is developed.

20. Data collection The development of evaluation questions consists of four steps (not linear in process ): a . Clarifying t he G oals and O bjectives of the Programme ; Evaluation starts with the evaluator clarifying SMART objectives from which outcomes can be formulated. The evaluator will then be involved in designing or obtaining instruments that are able to collect the required data The closer the objectives are to outcomes that can be directly and reliably measured, the more likely it is that a useful evaluation will result. This applies to large, complex and smaller, poorly resourced programmes .

21. Data collection Evaluation instruments provide opportunities for detailing specific questions which elicit meaningful responses. Designing of these instruments may help promptly indicate that a given objective is too ambitious or wide-ranging to be achievable in the short term and that the outcomes set have to be modified accordingly. Measurable outcomes are the signals of success, which will be used to indicate whether the objectives have been achieved. They indicate the extent of progress and should be capable of collection over a period of time on a constant basis. Measurable outcomes are most meaningful when they are derived from specific targets or objectives.

22. Data collection An evaluation instrument is a tool which specifies what information has to be collected including, where appropriate, lists of questions to be asked; what the source of information is; the format for recording the information etc. Objectives, measurable outcomes and evaluation instruments are thus closely interrelated. b . Identifying Key Stakeholders of The Programme Stakeholder involvement has become an important part of evaluation design, as it has been recognized that an evaluation must address much more than simply the needs of the funding agency .

23. Data collection In fact, participatory or collaborative evaluation is one way of ensuring that the evaluation is responsive to and meets the needs of stakeholders . The purpose of the evaluation will dictate the extent to which stakeholders participate in the evaluation and in what phases of the evaluation they participate. c . Prioritising Evaluation Q uestions Several criteria should be considered when developing and listing evaluation questions such as:

24. Data collection Who wants to know ? Will the information be new or confirmatory ? How important is the information to various stakeholders ? Are there sufficient resources to collect and analyse the information needed to answer the questions ? Can the question be addressed in the time available for the evaluation? d . Determining Which Q uestions C an B e A ddressed Once the set of evaluation questions is determined, the next step is selecting how each will be addressed and developing an overall evaluation design.

25. Data collection It is at this point that decisions regarding the types and mixture of data collection methodologies, sampling, scheduling of data collection and data analysis need to be made . These decisions are quite interdependent, and the data collection methods selected will have important implications for both scheduling and analysis plans. Step 2: Matching Questions W ith A ppropriate Data Collection Methods Data for evaluations can be collected through a wide range of data sources . Data sources can be classified into various categories. The most common ones are shown in the following table:

26. RESEARCH DESIGNS The problem under investigation, evaluation design, evaluation theory/tradition, the evaluation questions, and the literature reviews help to steer the researcher toward either the quantitative or qualitative track . These, in turn, inform the specific research design to be used and the procedures involved in them, such as sampling, data collection instruments or protocols, the procedures, the data analysis, and the final interpretation of results. Research/Evaluation designs are the specific procedures involved in the research process: data collection, data analysis, and report writing.

27. RESEARCH DESIGNS Survey Designs In some forms of quantitative research/evaluation, you may not want to test an activity or materials or may not be interested in the association among variables . Instead, you seek to describe trends in a large population of individuals. In this case, a survey is a good procedure to use . Survey designs are procedures in quantitative research in which you administer a survey or questionnaire to a small group of people (called the sample) to identify trends in attitudes, opinions, behaviors , or characteristics of a large group of people (called the population).

28. RESEARCH DESIGNS Correlational Designs Correlational designs are procedures in quantitative research in which investigators measure the degree of association (correlation) between two or more variables using the statistical procedure of correlational analysis . This degree of association, expressed as a number, indicates whether the two variables are related or whether one can predict another . To accomplish this, you study a single group of individuals rather than two or more groups as in an experiment. In some forms of quantitative research/evaluation, you may not want to test an activity or materials or may not be interested in the association among variables.

29. RESEARCH DESIGNS Instead, you seek to describe trends in a large population of individuals. In this case, a survey is a good procedure to use. Survey designs are procedures in quantitative research in which you administer a survey or questionnaire to a small group of people (called the sample) to identify trends in attitudes, opinions, behaviors , or characteristics of a large group of people (called the population).

30. RESEARCH DESIGNS Experimental Designs Some quantitative researchers/evaluators seek to test whether an intervention makes a difference for individuals . Experimental research procedures are ideally suited for this. Experimental designs (also called intervention studies or group comparison studies) are procedures in quantitative research in which the investigator determines whether an activity or materials make a difference in results for participants . You assess this impact by giving one group one set of activities (called an intervention) and with holding the set from another group.

31. RESEARCH DESIGNS Mixed Methods Designs You decide to collect both quantitative data (i.e., quantifiable data) and qualitative data (i.e., text or images ). The core argument for a mixed methods design is that the combination of both forms of data provides a better understanding of a research problem than either quantitative or qualitative data by itself. Mixed methods designs are procedures for collecting, analyzing , and mixing both quantitative and qualitative data in a single study or in a multi-phase series of studies . In this process, you need to decide on the emphasis you will give to each form of data (priority), which form of data you will collect first (concurrent or sequential), how you will “mix” the data (integrating or connecting), and whether you will use theory to guide the study (e.g., advocacy or social science theory).

32. RESEARCH DESIGNS Action Research Designs Like mixed methods research, action research designs often utilize both quantitative and qualitative data, but they focus more on procedures useful in addressing practical problems in projects, interventions, schools, hospitals, etc . Action research designs are systematic procedures used to gather quantitative and qualitative data to address improvements in their program like the educational setting, their teaching, and the learning of their students . In some action research designs, you seek to address and solve local, practical problems. In other studies, your objective might be to empower, transform, and emancipate individuals in the settings.

33. RESEARCH DESIGNS Grounded Theory Designs Instead of studying a single group, you might examine a number of individuals who have all experienced an action, interaction, or process . Grounded theory designs are systematic, qualitative procedures that researchers/evaluators use to generate a general explanation (grounded in the views of participants, called a grounded theory) that explains a process, action, or interaction among people . The procedures for developing this theory include primarily collecting interview data, developing and relating categories (or themes) of information, and composing a figure or visual model that portrays the general explanation . In this way, the explanation is “grounded” in the data from participants. From this explanation, you construct predictive statements about the experiences of individuals

34. RESEARCH DESIGNS Narrative Research Designs You may not be interested in describing and interpreting group behaviour or ideas, or in developing an explanation grounded in the experiences of many individuals . Instead, you wish to tell the stories of one or two individuals. Narrative research designs are qualitative procedures in which researchers describe the lives of individuals, collect and tell stories about these individuals’ lives, and write narratives about their experiences . In education, these stories often relate to school classroom experiences or activities in schools. Instead of studying a single group, you might examine a number of individuals who have all experienced an action, interaction, or process.

35. RESEARCH DESIGNS Ethnographic Designs You may be interested in study in gone group of individuals, in examining the min the setting where they live and work, and in developing a portrait of how they interact . An ethnographic study is well suited for this purpose . Ethnographic designs are qualitative procedures for describing, analyzing , and interpreting a cultural group’s shared patterns of behavior , beliefs, and language that develop overtime. In ethnography, the researcher provides a detailed picture of the culture – sharing group, drawing on various sources of information . The ethnographer also describes the group within its setting, explores themes or issues that develop over time as the group interacts, and details a portrait of the group.

36. DATA COLLECTION AND EVALUATION THEORIES What data is/are required depends on a number of variables The needs of the client ; The timing of the monitoring/evaluation in the project cycle ; The nature of the project ; The purpose of monitoring or evaluation For data collection, the general rule is use available data if you can. (Its faster, less expensive, and easier than generating new data .). However , find out how earlier evaluators: collected the data defined the variables ensured accuracy of the data.

37. DATA COLLECTION AND EVALUATION THEORIES If you must collect original data THEN: establish procedures and follow them (protocol) maintain accurate records of definitions and coding pre-test , pre-test, pre-test verify accuracy of coding, data input . The evaluator has to consider the specific questions being addressed by the evaluation and the audience for the answers. This will influence the selection of evaluation design, data sources and methods of data collection

38. DATA COLLECTION AND EVALUATION THEORIES Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations . They specify such things as evaluation purposes, users, and uses, who participates in the evaluation process and to what extent, general activities or strategies, method choices, and roles and responsibilities of the evaluator, among others (Fournier,1995; Smith,1993). Largely, such theories are normative in origin and have been derived from practice rather than theories that are put into practice (Chelimsky,1998). Additionally, specifying program theory recently has been put forth as an essential competency for program evaluators ( Stevahn , King, Ghere , & Minnema , 2005 ). Stufflebeam and Shinkfield (2007) note, ‘‘...if evaluators do not apply evaluation theory... then it is important to ask why they do not . Perhaps the approaches are not sufficiently articulated for general use, or the practitioners are not competent to carry them out, or the approaches lack convincing evidence that their use produces the needed evaluation results’’ ( p.62).

39. DATA COLLECTION METHODS

40. DATA COLLECTION TOOLS

41. DATA COLLECTION METHODS Participatory Data Collection Methods Participatory data are data that are collected when interacting with people. Examples of participatory data are: Transect walks – walks that researchers take around a community observing the people, surroundings, and resources, and which can help identify issues that might need further investigation. Social mapping – can be used to present information on village layout, infrastructure, demography, health patterns, wealth and other community issues Community meetings – information gathered during meetings of people in the community, such as comments, questions asked, etc.

42. DATA COLLECTION METHODS Mapping One approach that can be used when working in communities is mapping – drawing” a conceptual picture of the various elements that make up a community, including resources and assets, and how they interact with one another . This approach brings together members of the community in order to better understand the community and how the intervention fits (or does not) within the community . It can be used as part of any approach if appropriate to the evaluation questions . Mapping is a method for collecting and plotting information on the distribution, access, and use of resources within a community . Mapping is a useful tool in participatory evaluation or any approach involving stakeholders because it provides them with a way to work together.

43. DATA COLLECTION METHODS Mapping At the same time, it increases everyone’s understanding of the community. It is possible that people have different understandings of the community based on their status and experience . The process of mapping can be used to generate discussions about local development priorities. It can be used to verify secondary sources of information. Mapping can also capture changes or perceived changes. There are many kinds of mapping. They include :

44. DATA COLLECTION METHODS Transect mapping R esource mapping H istorical mapping S ocial mapping H ealth mapping W ealth mapping L and use mapping D emographic mapping . While the process of mapping probably applies more to planning interventions and in engaging citizens in a process that allows them to ultimately create a vision of what they wish to happen and a strategy for change, it can also be used in evaluations .

45. DATA COLLECTION METHODS It may prove interesting to see if people in the community know about the project you are evaluating, for example: if the community doesn’t identify the project as an asset, it will raise the question of why that is the case It also may help to understand whether the project being evaluate dislocated in the are as of greatest need or how they are co-located with other resources in the community . If they are co-located with other resources, do they work collaboratively ? If not, what are the barriers ?

46. DATA COLLECTION METHODS Two approaches are resource mapping and asset mapping . These are very similar. Both provide a way for people to understand their own community . Resources are often more narrowly defined in terms of institutional resources while asset mapping usually includes the assets of individuals within the community as well . Both assume that those closest to the community understand how it works and why it works . Asset mapping has a more explicit intention to use this process to bring about change .

47. DATA COLLECTION METHODS Resources and assets can include : Individual assets: skills, talent, networks, money, property, etc . Civic assets: faith associations, clubs, social groups, etc . Institutional assets: businesses, schools, health services, economic development and planning offices, social services, agricultural services, public transportation, cultural facilities, charities and foundations, other government offices, etc . Environmental assets: parks, clean air and water, roads, farmland, housing, etc.

48. DATA COLLECTION METHODS The mapping process may require participants to observe specific aspects of the community or interview people in the community, especially to identify individual assets . Alternatively, mapping can be done at group meetings . One process is to ask people to draw their community in terms of how they spend their day or in terms of places they are likely to visit during the year . Each person is given a big sheet of paper and magic markers or colored pencils . Each map is hung on a wall and then the whole group discusses common elements as well as differences . Together, they draw the larger map. It is not unusual for the maps that individuals in a community make to look different . In some places, the map that the women draw will be different from the map that the men draw because their lives and day-to-day experiences are different. In other places, people with different educational and occupational background will draw different maps.

49. DATA COLLECTION METHODS Tools for Mapping - GPS The global positioning system, usually called GPS, is a navigation system that uses satellites to identify locations on Earth . More than two dozen GPS satellites broadcasts precise timing signals by radio signals to GPS receivers, allowing them to accurately determine their location . They give the longitude, latitude, and altitude of locations, in any weather, day or night, anywhere on Earth . GPS has become a vital global utility, indispensable for modern navigation on land, sea, and air in all location of the world . It can also be a valuable tool to assist with mapping. You can use a GPS devise to pin-point a location and learn its latitude and longitude components . These can then be recorded to find the exact location later or to communicate the exact location to another person.

50. DATA COLLECTION METHODS Tools for Mapping-Google Earth Another tool that can assist evaluators is Google Earth. Google Earth is a free-of-charge, downloadable program from the Internet . It maps the entire earth by pasting images obtained from satellites and aerial photography and geographical information systems (GIS) on to an image of a three dimensional globe . Many large cities are available in a resolution high enough to see individual buildings, houses, and even cars. The degree of resolution available is based somewhat on the points of interest, but all land is covered in at least 15 meters of resolution . You can enter coordinates, or simply use the mouse to browse to a location on the globe. For example, Google Earth can be helpful for collecting baseline data . You can locate an area with Google Earth, save the image and print it. The pictures show locations of buildings, forests, rivers, lakes, etc . You can collect date at later dates and compare to the image you captured as part of the baseline data to the current situation.

51. DATA COLLECTION METHODS Available Data Sometimes data have already been collected that can be used to answer our questions. When you used at a gathered by others, you need to find out how they carried out data collection, how they measured each variable, the decision rules they used to code and clean the data, and how they treated missing data and non-responses . Examples of typical sources of available data : F iles/records Computer databases Industry reports Government reports Other reports or prior evaluations Census data Documents (budgets, policies and procedures, organizational charts, maps ).

52. DATA COLLECTION METHODS Using Agency Records Agency records are a common source of evaluation information . Agencies may have already collected the data you need from clients, community, and internal information systems . The agencies may also have summarized and reported the information. Agency reports may include: I nternal management reports B udget documents R eports to the public or funding agencies S ome evaluation or monitoring information. Key Issues to Consider in using agency records: Are the available data valid? Are the available data reliable? Are the available data accurate?

53. DATA COLLECTION METHODS Collecting Data from Paper Files, Records, or Documents : Sometimes the data are available but not in a form that is easy to analyze . You may have to collect information that is in files or documents . In this situation, you want to develop a data collection instrument (DCI) that specifies exactly what data you want to collect and how you want to code it . This data collection instrument is like a close-ended questionnaire with specific items that have fixed responses . Develop an instrument that is easy, simple, and clear. You can use a form that specifies exactly what data you want. Setup your procedures and train everyone who will be collecting the data . You want to make sure everyone codes data in exactly the same way. Establish decision rules for coders and test for inter- rater reliability. Once you have developed an instrument, pre-test it . When working with official documents that describe current activities or practices, try to verify that the documents accurately reflect what is actually practiced . Observations and interviews may help in verifying actual practices. For example, observe a training program: do they really hold classes every week and are the participants diverse?

54. DATA COLLECTION METHODS Collecting Computer Data: Obtain the database structure, data dictionary, coding schemes . Find out what is needed to transfer the data to your computer . Verify the accuracy of the data in the computer. Transfer the data with a minimum of effort (no re-typing)to avoid introducing new errors from data entry procedures . Advantages and Challenges of Using Available Data Advantages: Often less expensive and faster than collecting the original data yourself. Challenges: There may be coding errors or other problems . Data may not be exactly what is needed . You may have difficulty getting access . You have to verify validity and reliability of data.

55. DATA COLLECTION METHODS Observation Observation enables you to see what is happening. You can see a lot by just looking around . You can observe: traffic patterns, land use patterns, the layout of city and rural environments, the quality of the housing, the condition of roads, or who comes to a rural health clinic for medical services . When you use observation techniques, you can be an observer in one of three ways: unobtrusive, participant, or obtrusive . Unobtrusive Observer - No one knows you are observing. For example, if you visit a local market that has been given resources for development, you can observe the activity within the shops, the traffic in the area, and may even enter into casual conversations with shoppers. Of course, to be unobtrusive you must look like someone who would be likely to be seen in that market place

56. DATA COLLECTION METHODS Participant Observer – You actually participate in the activity, typically without anyone knowing you are observing. For example, you may make some purchases in the local market as if you were just a regular shopper but you really are evaluating the merchant – shopper interactions. Participant data are often used to check the quality of customer service, such as tax advice . Obtrusive Observer – The people being observed know you are there to observe them. For example, if you come into the market place with a clipboard and video camera and are introduced as an observer, then everyone knows you are there and for what reason .

57. DATA COLLECTION METHODS The following guidelines suggest ways to plan for an observation: Develop a checklist to rate your observations. Develop a rating scheme. Have more than one observer, if feasible. Train observers so they observe the same things. Be sure to pilot test the observation data collection instrument. To do this, two observers go to the same area and complete their rating sheets. After they complete their sheets, compare them . If there are big differences, give more training and clarification . If there is little differences proceed with the larger study . For a less formal approach to the observation, you can have a few key questions in mind when you arrive for your observation

58. DATA COLLECTION METHODS Advantages and Challenges of Observational Data Collection : Advantages: Collects data on actual behaviour rather than self-reported behaviour or perceptions. It is real time rather than retrospective. Challenges : Observer bias; Potentially unreliable ( two observers may see things very differently unless data collection is highly structured ); Interpretation and coding challenges; Sampling can be a problem; C an be labor intensive; Low response rates.

59. DATA COLLECTION METHODS Surveys Surveys are great for collecting data about people’s perceptions, opinions, and ideas. They are less accurate in measuring behaviour because what people say they do may or may not reflect what they actually do . A key component of a survey is the sample; ideally, the sample is representative of the population as a whole Surveys can be either structured or semi-structured . Structured surveys are precisely worded with a range of predetermined responses that the respondent can select. Everyone is asked exactly the same questions in exactly the same way, and is given exactly the same choices to answer the questions . The number of response options you offer should generally be an odd number, ( i.e ., 3, 5, or 7) so that the neutral response is readily apparent to the respondent . The time to use two options is when, for example, you want only a “yes’ or “no” answer . Semi-structured surveys ask the same general set of questions, but leave many, if not all, of the answers open-ended .

60. DATA COLLECTION METHODS Advantages of structured and semi-structured surveys. Structured surveys are : Harder to develop: you have to be absolutely certain you have covered all possible pieces of information, since there are no “catch-all” open-ended questions that could fill in the gaps Easier to complete: checking a box takes less time than writing a narrative response Easier to analyse More efficient when working with large numbers of people. Semi-structured surveys are : A little easier to develop: you can include fairly broad open-ended questions that will capture anything else missed in the structured sections, so there is less danger of leaving something else Labour intensive to conduct Harder to analyze but provide A rich source of data Subject to bias in the interpretation of the open-ended responses Burdensome for people to complete as A self administrated questionnaire.

61. DATA COLLECTION METHODS Three of the methods of collecting surveys are: In-person interviews: group or individual Self-administered questionnaire Mail, phone, or e-mail surveys. In-person interviews are useful when you want an in-depth understanding of experiences, opinions or individual personal descriptions of a process. It is also useful when other approaches will not work; for instance, self-administered questionnaires only work when the population can read the language of the survey and are motivated enough to respond. Self-administered questionnaires can also be either structured, semi-structured, or a combination. These are written surveys that the respondent completes. Self-administered questionnaires should be short and take no more than 30 minutes to complete (shorter is better). Another choice for collecting data with a survey is to use the postal system or a technology, such as telephone or the Internet.

62. DATA COLLECTION METHODS Advantages and Challenges of Surveys Advantages: Best when you want to know what people think , believe, or perceive; O nly they can tell you that. Challenges : People may not accurately recall their behaviour or may be reluctant to reveal their behaviour if it is illegal or stigmatized. What people think they do or say they do is not always the same as what they actually do.

63. DATA COLLECTION METHODS Response Rates One of the major issues in survey research is the response rate . Response rate is the percentage of people who actually participate out of the total number asked . A good evaluator always gives the number of people (or units, such as organizations) surveyed, the number who responded, the response rate, and a description of efforts that were made to increase the response rate (i.e. follow-up telephone calls .) What is desirable as a response rate varies, depending on the circumstances and uses of the survey data ? The problem with low response rates is that it becomes a volunteer or self selected sample . The problem with a volunteer sample is that people who choose to participate might be different from those who choose not to . Maybe only people who are really angry at management will choose to answer a survey; this will result in a more negative assessment of management than would have been the case if everyone, or at least a more representative group, had participated .

64. DATA COLLECTION METHODS Cover Letters for Questionnaires When you are sending a questionnaire by mail or email, you will want to include a cover letter. The cover letter will introduce you and give the participants more information about the purpose of the questionnaire. The cover letter should include the following: be personally addressed identify who is asking for the survey state the purpose of the interview and an overview of questions state how the information will be used assure confidentiality and/or anonymity provide a name and phone number (or email address) of contact person provide instructions for returning the survey.

65. DATA COLLECTION METHODS Focus Groups A focus group is a type of qualitative research methodology in which small homogenous groups of people are brought together to informally discuss specific topics under the guidance of a moderator. But the structure of the focus group is anything but informal. There is a script, a set of open-ended questions that are prepared ahead of time. The moderator can improvise , though, with probes or additional questions as warranted by the situation. The group process tends to elicit more information than individual interviews because people express different views and engage in a dialogue with each other . The moderator is able to facilitate the dialogue as well as explore their reasons and feelings behind those differences. The conversation is often not linear; participants may bring up information or different perspectives at any time.

66. DATA COLLECTION METHODS Purpose of Focus Groups T he purpose of focus groups is to elicit reliable data, not just interesting information. Focus groups can: help develop a survey questionnaire clarify sample selection contextualize survey data be used in tandem with surveys be used as a separate data collection tool.

67. DATA COLLECTION METHODS Advantages and Challenges of Focus Groups Advantages : Relatively quick and easy; May take less staff time than in-depth in-person interviews; Provides flexibility to make changes in process and questions; ability to explore different perspectives . Challenges: Analysis is time consuming; Potential challenges include participants might be different from rest of population, risk of bias in interpreting the data; and the risk of the group being influenced by moderator or dominant members of the group .

68. DATA COLLECTION METHODS Techniques for Moderating Focus Groups The facilitator will direct the meeting and manage the time. Facilitators have to : Be familiar with the script, rather than reading it, so the session appears conversational Make sure everyone is heard, by: asking “what do other people think?”, stating “We have heard from a few people; do others have the same or different views ?” Manage time, closing off discussion, and moving to the next topic when appropriate Set ground rules, such as: there is no such thing as a wrong comment and no criticism of others is permitted . Say as little as possible, letting conversation flow across the table with minimal direction Keep personal views outside the room Use active listening Accept all views while managing differences of opinion : “ So , we have different perspectives .” Probe for elaboration − “ Tell us more .” The moderator will ask many questions which will be determined by the purpose of the focus group

69. DATA COLLECTION METHODS Steps in conducting an FGD Introduce the focus group meeting . – highlight the purpose of focus group, introduce the sponsor, describe how participants were selected, describe how the information will be used, state the ground rules and give and overview of the process Have the participants introduce themselves. Present the first question, it should be easy, an ice-breaker. Ask the main questions. Ask the last questions. They should be summary questions ( “What was the most important thing that was said here that we should take with us?” ) Summarize your understanding and ask for confirmation. Ask if there are other comments or questions. Write-up impressions, major issues, and major points of discussion. Include anything unusual that happened.