Non experiemental research Basics for research beginners.

Non experiemental research INCLUDES QUALITATIVE, QUANTITATIVE AND CORRELATIONAL DESIGNS.

1. RESEARCH METHODOLOGY NON-EXPERIEMENTAL RESEARCH DESIGN PRADEEP BALAKRISHNAN 1

2. What is research? A systematic means of problem solving ( Tuckman 1978) 2

3. OBJECTIVES Define the terms  research design  and  non-experimental research designs . Identify the four major types of non-experimental designs and differentiate them from one another. Describe the characteristics of descriptive studies and identify the criteria to evaluate them. Describe the characteristics of correlational studies. Differentiate comparative and correlational designs. Identify the criteria for evaluating comparative designs as well as the criteria for evaluating correlational designs. Differentiate between correlational and prediction studies. Explain the differences between simple predictive studies and multiple regression studies. Identify the issues of concern for predictive and issues of concern for regression studies. Identify the problems in interpreting correlation coefficients. Describe the characteristics of causal-comparative designs. Differentiate between ex-post-facto and correlational causal-comparative designs. Identify the criteria for evaluating these designs. Describe the characteristics of surveys and identify three reasons for their popularity. Differentiate cross sectional and longitudinal surveys. Identify the issues of concern for cross sectional and longitudinal surveys. Describe the steps necessary to design a survey. Identify the concerns related to low response rates and  i dentify seven ways a researcher could increase response rates. Identify the advantages and disadvantages of Internet based surveys. Use the evaluation criteria for each type of non-experimental study to evaluate an actual research article. 3

4. Research design - the plan and structure of research to provide a credible answer to a research question 4

5. Purpose of non-experimental designs describes current existing characteristics such as achievement, attitudes, relationships, etc. 5

6. Non-experimental designs Four types of designs Descriptive Relationships Comparative Correlational Causal-comparative Survey 6

7. Descriptive designs Studies that describe a phenomena Statistical nature of the description (e.g., frequency, percentages, averages, graphs, etc.) Importance of these designs in the early stages of the investigation of an area Criteria for evaluating descriptive studies Conclusions about relationships should not be drawn Subjects and instruments should be described completely 7

8. Relationship designs Two types Comparative designs Correlational designs Designs describing the relationship between two or more variables Comparative designs These studies investigate the relationship of one variable to another by examining differences on the dependent variable between two groups of subjects If math scores for males are significantly higher than those for females, a relationship exists between gender and math achievement If the academic self-concept scores for ninth graders are significantly different than those for twelfth graders, a relationship exists between grade level and academic self-concept If the third grade achievement scores for whites are not significantly different that those for non-whites, no relationship exists between ethnicity and achievement 8

9. Criteria for evaluating comparative designs Subjects and instruments are described completely Criteria for identifying the different groups is clearly stated No inferences are made about causation Graphs and images depict the results accurately 9

10. Correlational designs 1. Simple correlation  - studies examine the relationship between two variables Examples Math achievement and math attitudes Teacher effectiveness and teacher efficacy Cautions in interpreting correlations A relationship between two variables (e.g., achievement and attitude) does not mean one causes the other (i.e., positive attitudes do not cause high levels of achievement) Attenuation - the possibility of low reliability of the instruments makes it difficult to identify relationships Restriction in range - a lack of variability in scores makes it difficult to identify relationships Everyone scoring very, very low Everyone scoring very, very high Large sample sizes and/or using many variables can identify significant relationships for statistical reasons and not because the relationships really exist 10

11. 2. Prediction  - studies examine the predictive nature of the relationships between variables Simple predictive studies Multiple regression Simple predictive studies  - performance on one variable (i.e., the predictor) is used to predict performance on a second variable (i.e., the outcome or criterion) Examples Scholastic Aptitude Test (SAT) scores are used to predict freshmen grade point averages Scores from a mathematical attitude scale are used to predict math achievement scores Importance of the time interval between collecting the predictor and criterion variable data Factors influencing correlations Attenuation - the possibility of low reliability of the instruments measuring the predictor and criterion variables makes it difficult to identify relationships Length of time between the predictor and criterion variable data collection Existence of many factors, not only the one being examined, that influence the criterion variable 11

12. Multiple regression - studies that examine performance on several variables (i.e., predictor variables) to predict performance on a single variable (i.e., criterion) Examples Scholastic Aptitude Test (SAT) scores, high school grade point average, and high school rank in class are used to predict freshmen grade point average Math attitude scale scores, academic self-esteem scale scores, and prior math grades are used to predict math achievement scores Issues of concern Sample size of at least 10 subjects for each predictor variable Relationships among the predictor variables (i.e., colinearity ) 12

13. 3. Significance of correlation coefficients Statistical significance Does a statistical relationship exist? Is the observed correlation significantly different from zero? Practical significance Does a relationship of practical importance exist? Coefficient of determination (r 2 ) - the percentage of the criterion variable variation that can be explained by the variation in the predictor variable 13

14. 4. Guidelines for interpreting the size of correlation coefficients a. Much larger correlations are needed for predictions with individuals than with groups Crude group predictions can be made with correlations as low as .40 to .60 Predictions for individuals require correlations above .75 b. Exploratory studies Correlations of .25 to .40 indicate the need for further research Much higher correlations are needed to confirm or test hypotheses c. Multiple correlation coefficients (i.e., those resulting from multiple regression analyses) of .20 - .40 are common and usually indicate practical significance 14

15. 5. Criteria for evaluating correlational studies Causation should not be inferred from correlational studies The reported correlation should not be higher or lower than the actual correlation Practical significance should not be confused with statistical significance The size of the correlation should be sufficient for the use of the results Prediction studies should report the accuracy of predictions for new subjects Procedures for collecting data should be clearly indicated 15

16. Comparing comparative and correlational designs Comparative - one variable and two or more groups Correlational - one group and two or more variables 16

17. Causal-comparative designs Ex-post-facto designs Correlational causal-comparative studies Criteria for evaluating causal-comparative studies 17

18. Ex-post-facto designs Ex-post-facto designs investigate the relationships between independent and dependent variables in situations where it is impossible or unethical to manipulate the independent variable Example - what is the effect of pre-kindergarten (Pre-K) attendance on first grade achievement Cannot mandate Pre-K attendance for children Characteristics and resources of families who do or do not send their children to Pre-K may influence first grade achievement Similarities with correlational  and  experimental research designs Issues of concern Selecting subjects who are as similar as possible on all characteristics except the independent variable Generalizing beyond the subjects studied 18

19. Correlational causal-comparative studies Use of correlational models to investigate possible cause and effect relationships Sophisticated statistical models Path analysis Structural equation modeling Fundamental limitations of all correlational research designs apply 19

20. Criteria for evaluating causal-comparative studies Primary purpose is to investigate causal relationships when experimental designs are not possible Presumed causal condition has already occurred Potential extraneous variables are considered Existing differences between groups being compared are controlled Causal conclusions are made with caution 20

21. Using surveys in non-experimental designs Surveys represent a data collection method that is very useful in descriptive and correlational studies because it is versatile, efficient, and generalizable Types of surveys Cross sectional designs - information is collected from one or more groups at the same time Examples Student's, teacher's, administrator's, and parent's opinions regarding an extended school year Elementary, middle, and secondary teachers' feelings toward a new school board policy Issue of concern - comparisons across groups can be the result of differences between subjects within the groups Fifth and seventh graders opinions can be affected by a change in the attendance zones of a school Longitudinal designs - information is collected from the same subjects over time Example - changes in the academic self-concept of students from the sixth to the twelfth grade Issues of concern Loss of subjects over time Difficulty tracking subjects over time 21

22. Steps in designing a survey 1. Define a purpose and objectives 2. Identify the resources needed and the target population Costs Preparations Printing Mailing costs - sending and returning Analyzing Length of the survey Time needed to administer the survey Sample size 3. Choose the method Paper Electronic Telephone Interview 22

23. 4. Develop the items - guidelines Use clear, unbiased, non-ambiguous language Keep it short and simple Use grammatically correct language Do not write leading items Use the same response scale for all items Be consistent with wording 5. Design the format White space Font size 6. Develop directions Make them clear with no ambiguity Indicate clearly how subjects are to respond Indicate where responses are recorded Indicate what subjects should do when finished 23

24. 7. Develop a letter of transmittal Keep it brief Include a statement of the purpose of the research Include a statement of the benefits of the research 8. Pilot test 15-20 representative subjects Identify concerns Clarity Format Responding Directions Time to complete 24

25. Response rates Low response rates are the major limitation of survey use Suggestions for increasing response rates Design the survey well Contact the subjects several times especially following-up on non-respondents Include a self-addressed return envelope Use a good transmittal letter Use a telephone for follow-up Use incentives for completing the survey 25

26. Using electronic surveys Internet based surveys E-mail attachments Web pages Advantages Reduced time and cost Easy access Quick responses Ease of creating data sets Disadvantages Limited to those with access to the technology Confidentiality and privacy issues Online survey research centers 26

27. Dependent variable The changes to the dependent variable are what the researcher is trying to measure with all their techniques. The  dependent  variable can not change or grow unless something else is happening or influencing it. What might be influencing the dependent variable is the independent variable. The independent variable can stand alone all by itself and does not require anything else to happen in order for change or growth to occur. 27

28. Dependent Variable = this variable is the ‘effect’ Dependent Variable = should only vary in response to the IV Dependent Variable = also known as the criterion variable 28

29. Independent variable A variable believed to affect the dependent variable. This is the variable that you, the researcher, will manipulate to see if it makes the dependent variable change. Independent Variable = this variable is the ‘cause’ Independent Variable = can be manipulated or allowed to vary Independent Variable = also known as the predictor variable 29

30. Quantitative Research Strategy Investigation aims to assess a pre-stated theory (Deductive Reasoning) Often involves hypothesis testing Attempts to minimise the influence of the researcher on the outcome Quantitative data infers statistics Data collection therefore requires ‘closed’ responses 30