Definition of Evaluation

Passmore

1. Evaluation of Investment In Workforce Education & Development

2. What we will do… Define evaluation as a process. Make decision-making a focus of evaluation: Decision-making processes Decision settings Decision types Describe use of criteria in evaluation. Delineate criteria for evaluation. Suggest a meta-methodology for evaluation. Specify four types of evaluation. Present a general evaluation model.

3. Decision types Actual Intended Ends Means PLANNING RECYCLING STRUCTURING IMPLEMENTING To determine objectives To judge and react to attainments To design procedures To utilize, control, and refine procedures

4. What we will do… Define evaluation as a process. Make decision-making a focus of evaluation: Decision-making processes Decision settings Decision types Describe use of criteria in evaluation. Delineate criteria for evaluation. Suggest a meta-methodology for evaluation. Specify four types of evaluation. Present a general evaluation model.

5. Describe use of criteria in evaluation

6. Criteria A decision requires judging among two or more alternatives. Criteria are rules or tests for judging. Criteria are yardsticks for values. Values = estimated or assessed worth. Values result from systems of beliefs, perceptions, interpretations of facts, judgments of needs. Values are shared and/or idiosyncratic. Decision-makers apply criteria. They must “ own ” criteria. Often need assistance in identifying and finding criteria.

7. Four kinds of values that impinge on establishment of criteria Instiutional values E.g., “ What does the Board of Trustees expect our College to do? ” Values external to the institution E.g., “ What does the State Legislature believe the College should produce? ” Subsystem maintenance values E.g., “ What do my College colleagues believe we should do? ” Private values E.g, “ What do I think our College should produce? ”

8. What we will do… Define evaluation as a process. Make decision-making a focus of evaluation: Decision-making processes Decision settings Decision types Describe use of criteria in evaluation. Delineate criteria for evaluation. Suggest a meta-methodology for evaluation. Specify four types of evaluation. Present a general evaluation model.

9. Delineate criteria for evaluation

10. Criteria for evaluation Scientific Internal and external validity, reliability, objectivity Practical Relevance, importance, scope, credibility, timeliness, pervasiveness Prudential Efficiency

11. Criteria for evaluation: Scientific Internal validity No alternative explanations are possible for results obtained. External validity Findings are generalizable across people, places, and time. Reliability Accuracy of findings; reproducibility. Objectivity Intersubjectivity; lack of bias.

12. Criteria for evaluation: Practical Relevance Information serves decisions. Importance “ Best ” information culled to serve decisions. Scope All necessary information is provided. Credibility Clients/users believe information. Timeliness Information provided when needed. Pervasiveness Information provided to all clients/users.

13. Criteria for evaluation: Efficiency Conducted with the “ best ” use of resources.

14. What we will do… Define evaluation as a process. Make decision-making a focus of evaluation: Decision-making processes Decision settings Decision types Describe use of criteria in evaluation. Delineate criteria for evaluation. Suggest a meta-methodology for evaluation. Specify four types of evaluation. Present a general evaluation model.

15. Suggest a meta-methodology for evaluation

16. Work breakdown of evaluation design

17. Delineation of information needs

18. Delineation of information needs

19. Delineation of information needs

20. Delineation of information needs

21. Work breakdown of evaluation design

22. Plan for obtaining information

23. Plan for obtaining information

24. Plan for obtaining information

25. Work breakdown of evaluation design

26. Plan for providing information

27. Plan for providing information

28. What we will do… Define evaluation as a process. Make decision-making a focus of evaluation: Decision-making processes Decision settings Decision types Describe use of criteria in evaluation. Delineate criteria for evaluation. Suggest a meta-methodology for evaluation. Specify four types of evaluation. Present a general evaluation model.

29. Specify four types of evaluation

30. Evaluation & decision-making

31. Evaluation types Actual Intended Ends Means CONTEXT PRODUCT INPUT PROCESS For planning decisions For recycling decisions For structuring decisions For implementing decisions

32. Evaluation types: Context Defines relevant environment. Identifies unmet needs and unused opportunities. Diagnoses problems that prevent needs from being met and opportunities from being used. Is macro-analytic and systematic. Involves contingency and congruence studies to set direction. Contingency - searches for opportunities and pressures outside system to promote improvement within. Congruence - compares actual and intended system performance to affirm objectives or consider new objectives.

33. Evaluation types: Input Determines how to use resources to achieve aims. Identifies relevant capabilities. Specifies strategies for achieving goals. Fabricates specific designs to accomplish aims. End product of input evaluation is one or more procedural designs, with a lists of costs and benefits. Assesses alternative designs in light of resource requirements/limits. Identifies procedural barriers, and risk/consequences of failure to overcome. Relevance of various designs to aims. Build or make? Project proposals embody the results of input evaluation.

34. Evaluation types: Input (cont.) Although context evaluation is systematic and macro-analytic, input evaluation is ad hoc and micro-analytic. Some questions asked: Is need documented? What strategies already exist? What are costs and benefits of competing strategies? What is track record for competing strategies? Are designs logical? Based on sound theory? Assumptions met? Legal? Moral? Side effects? Likely to be adopted?

35. Evaluation types: Process Aims: Detects or predicts defects in procedural design or in implementation in early stages. Provides information for programmed decisions. Records implementation as it occurs. Depends on context and input evaluation. More adequate context and input evaluation, the less critical the need for process evaluation. Less adequate context and input evaluation, the more critical process evaluation becomes. More important than product evaluation in early development stages, but less important as more structure and knowledge is developed.

36. Evaluation types: Product Interprets attainments at end of project cycle or as often as necessary during cycle for accountability or improvement. Involves: Devising operational definitions of objectives. Obtaining criterion measures. Comparing measures with some absolute or relative standards. Making rational interpretations of outcomes using recorded context, input, and process information. Criteria are instrumental or consequential: Instrumental -- related to intermediate accomplishments prerequisite to some end. Consequential -- pertain to fundamental, penultimate ends sought.

37. What we will do… Define evaluation as a process. Make decision-making a focus of evaluation: Decision-making processes Decision settings Decision types Describe use of criteria in evaluation. Delineate criteria for evaluation. Suggest a meta-methodology for evaluation. Specify four types of evaluation. Present a general evaluation model.

38. Present a general evaluation model

39. Evaluation of Investment In Workforce Education & Development