THE POTENTIAL OF SEQUENCE ANALYSIS FOR ANALYSING DATA FROM AN ONLINE MATHEMATICS ENVIRONMENT

Presentation for ICME-15

1. THE POTENTIAL OF SEQUENCE ANALYSIS FOR ANALYSING DATA FROM AN ONLINE MATHEMATICS ENVIRONMENT 15th International Congress on Mathematical Education 8 July 2024, Sydney, Australia Southampton Education School, Dr Christian Bokhove Professor in Mathematics Education

2. Intelligent Tutoring Systems (ITS) 70s and 80s - CAI tutors (computer-assisted instruction) guided learners through each step of a problem solution by giving hints and feedback More modern variations are usually called Intelligent Tutoring Systems, or ITSs ( VanLehn , 2011) Positive reviews in support of the effectiveness of ITSs (Kulik & Fletcher, 2016; Ma et al., 2014; Steenbergen -Hu & Cooper, 2013, 2014; VanLehn , 2011). Appropriate help-seeking behaviour in relation to feedback use, can positively influence student learning gain (Tai et al., 2016).

3. Methodology - platform Maths -Whizz is an intelligent online tutor for 5 to 13-year-olds. 1200 learning objectives in 22 topics. 100s of schools, 8 areas, 150,000 students.

4. Sample - logfiles All records of school users in the UK in Years 3-5 who had at least 100 lesson records in academic year ’18-’19 (N=1,799). For this paper, we selected those students that had at least 365 days between first and last usage (range>365), which ensures some usage across two school years (N=1,113) between December 2010 and June 2020. The dataset included 856,205 records, with each row corresponding to a single attempt at a lesson.

5. Variables ID, date of birth, gender. Start and end date of Whizz usage. Average difficulty all exercises by student. Total time, total no of exercises, total no of lessons. Total score, total help. Derived: Range (end – start date) Precision (score/ no_questions ) Intensity ( nlessons /range) Process/sequences: Run modes

6. Data analysis Sequence analysis – discrepancy analysis Rstudio - The TraMineR package, an R-package for mining, describing and visualizing sequences of states or events ( Gabadinho et al., 2011).

7. Optimal number of clusters (elbow, silhouette, gap), indicated k=3. Sequence length is a confounder for the cluster solution. In Type 3, the ‘replay’ (yellow) mode, is much more prevalent, possibly indicating help-seeking behaviour .

8. Totalhelp and precision interact with each other. More help extends the sequence length and extends precision, but only up to a point. For too difficult questions help does not increase the sequence.

9. Substantive interpretation Well-tailored help helps the student along without compromising difficulty. However, this is subject to a Goldilocks ‘sweet spot’: help-seeking does not contribute to precision if the learning content is too difficult or the student too proficient. Help is not needed if the learning content is too easy Help-seeking might not be productive if a student is not proficient enough yet.

10. Usefulness of sequence analysis We used large-scale data from log files. These often consist of sequences of events, and hence sequence analysis is an appropriate data analysis approach. We showed the application of sequence analysis, in particular discrepancy analysis and the use of a regression tree, to show how help-seeking and precision interact.

11. Thank you Dr Christian Bokhove [email protected]