An overview of Test Methodology & Practicum in.pptx

Test validity and Reliability

1. An overview of Test Methodology & Practicum in Oral Assessment Jarrod Anderson

2. Introduction Why is testing important? Teachers are able to evaluate their students A student is able to see how effective they have been in learning and applying course content. Teachers can perhaps see how successful they have been in teaching and the course content.

3. Why do you think we need tests? For students to be given a grade. To make meaningful comparisons between students To determine a students level of learned knowledge in reference to course content/objectives. To evaluate language proficiency i.e. to get into university courses. For students who need a certain level of proficiency for their career. For teacher assessment.

4. Test Methodology Overview

5. Main Points Validity Reliability Backwash

6. Validity Defined: Is the extent to which a test accurately measures what it intends to measure. For a test to be valid, the test should be developed and prepared in a way so that the results are an indication of a students actual ability.

7. Validity Content Face Response

8. Content Validity

9. Content Validity Main Points To achieve content validity the test needs to be made up of content and tasks that are included in classroom lessons. Use Direct Testing Make the student perform the skill which needs to be assessed.

10. In testing speaking ability in this Direct testing?

11. Content Validity Main points You need a representative enough sample to ensure validity is achieved. The sample needs to be practical. The teacher needs to decide what should be included and what can be omitted in order to get a comprehensive enough sample.

12. Face Validity

13. Face Validity Main points The test should be seen by both teacher and student as fair and relevant. It looks like is measuring what it intends to measure.

14. Face Validity Main points Will be high if the test is… Well constructed, expected format with familiar tasks Clearly doable within the time frame Items are clear and uncomplicated Directions are crystal clear Tasks relate to the course work (content Validity) The difficulty is reasonable

15. It could be said that Face Validity is also concerned with optimizing student performance.

16. Response Validity

17. Response Validity Main Points Describes the extent to which examinees respond in a manner expected by the test developer Closely related to Face Validity If a test format is unfamiliar, instructions are unclear then students responses may not reflect their true ability

18. Activity For each situation below state whether it would be a threat to validity. A questionnaire that takes more than an hour to complete The test has some questions to trick the into giving a wrong answer The test does not include everything that has been taught The candidate is unable to understand the examination directions A speaking test which consists of multiple choice grammatical items The test format is unfamiliar and contains content not covered in the syllabus

19. Reliability Reliability is concerned with making sure a test is consistent in obtaining a students score. If the student were to take the test or a similar test on 2 different occasions, there score should be more or less similar, regardless of when they took it.

20. Reliability A students performance (or lack of it) may not only be due to their knowledge of the subject matter, but rather other factors may be involved that influence their score.

21. Threats to Reliability Temporary or Psychological changes. Scoring: Intra-rater and Inter rater reliability Test Administration

22. Threats to Reliability Temporary or Psychological changes. This is the most common form of unreliability. Can be caused by tired, sickness, emotional disturbances (nervousness) and can all effect how a student may perform.

23. Threats to Reliability Scoring: Intra-rater reliability A teacher or scorer is likely to become less accurate as he/she gets tired. Scorers can also be influenced by appearance and personality traits. Consistency here is most important

24. Threats to Reliability Scoring: Inter -rater reliability There may be great differences between scorers and how they allocate marks to their students. In tests that are “high stakes” tests, marking should be standardized.

25. Threats to Reliability Test administration Problems can occur when the time is inconsistent. How is cheating dealt with? Inconsistencies in the test environment.

26. How can we reduce these threats to Reliability? What are some solutions to ensure greater reliability in Scoring and Test Administration?

27. Backwash

28. Backwash Defined Backwash is the effect that tests have on learning and teaching. These effects can be both positive and negative. How can these effects be negative? Can be true when a teacher teaches to the test and rather than focusing on learning a subject matter, focuses on how to pass the test.

29. Achieving Positive Backwash Use Direct Testing If a test requires a student to speak, then it is most likely that the teacher will use activities and tasks that correlate with what the test requires the students to do. Students will realize the need to practice these activities because it will help prepare them for the test.

30. Achieving Positive Backwash Criterion referenced testing Are tests where specific objectives have been stated and levels of achievement have been determined prior to the test being taken. For example..

31. Achieving Positive Backwash Criterion referenced testing Students should know what they are going to be tested on and how they are going to be assessed. Students should also understand that regardless of how other students perform, they will get a grade according to their level of performance in the test.

32. Achieving Positive Backwash Negative Backwash Norm referenced testing. Where student achievement is based upon how other students do. Can result in students passing or failing regardless how well or badly they do. In such cases, motivation and classroom participation could be low.

33. Achieving Positive Backwash Feedback It is important when giving a student their score to remember how affected by it they could be

34. Difference between this...

35. o r this.

36. Achieving Positive Backwash Feedback The way the information is presented c an make a difference between a learning experience and one where the student becomes unmotivated and starts to not care.

37. Achieving Positive Backwash Feedback Comment on test performance, don’t just give your student a score Student should have access to their teacher and feel comfortable in coming to see them.

38. Achieving Positive Backwash Practicality To achieve positive Backwash for the teacher the test should be easy to administer, score and interpret. We don’t like complicated scoring procedures that are going to take a lot of time

39. Achieving Positive Backwash Practicality Checklist Are the administrative details clearly established before the test Can the test be administered smoothly Are all the materials and equipment ready Is scoring/evaluation “time friendly”

40. Activity How does direct testing achieve positive backwash? How does criterion referenced testing motivate students? Can giving feedback to a student result in negative backwash? How?

41. The KET Speaking Test

42. KET Speaking Test Is a Cambridge speaking test Is the easiest test for beginner students in EFL/ESL. The speaking part goes for 8-10mins. 2 parts to the test

43. KET Speaking Test