Why Students Cannot Easily Integrate Component Skills: An Investigation of the Composition Effect in Programming

Slides for ACM ToCE journal paper presentation at SIGCSE 2025

1. 1 Why Students Cannot Easily Integrate Component Skills: An Investigation of the Composition Effect in Programming Yun Huang 1 , Christian Schunn 2 , Julio Guerra 1 , Peter Brusilovsky 2 1 Austral University of Chile, 2 University of Pittsburgh Feb, 2025 https:// dl.acm.org / doi /pdf/10.1145/3673239 April, 2021, LAK ‘21

2. Outline Introduction Methods Results General discussion Conclusions 2

3. Introduction Introductory programming courses are challenging to students [Guzdial, 2015; McCracken et al., 2001] A central challenge: integrating component skills Called the composition effect in this work Is it a challenge of doing multiple things at once related to cognitive load? requiring new conceptual understanding? or both? Better practical instructional designs and theoretical understanding 3

4. Prior work: Skill Integration and the Composition Effect The nature of skill integration : two views from cognitive and learning sciences research: It requires procedural knowledge , with additional working memory demands [Ambrose et al., 2010; LeBlanc and Weber-Russell, 1996] It requires conceptual knowledge of how components integrate [Koedinger et al., 2012] Composition effect : the phenomenon of the whole being more difficult than the sum of the parts, related to the study of skill integration Most empirical work were in mathematics, and lacked error analysis for errors in integration [Alibali et al., 2014; Heffernan and Koedinger, 1997] 4

5. Prior work: Challenges of Learning to Program While the composition effect has not been directly studied in programming, there has been abundant empirical research into: Difficulties in acquiring component skills and in programming generally One source: misconceptions of basic constructs [du Boulay, 1986; Guzdial , 2015; …] Another source: working memory/cognitive load [Anderson and Jeffries, 1985; …] Difficulties in integration in code writing One source: arrangement bugs rather than misconceptions of basic constructs [du Boulay, 1986; Perkins and Martin, 1986; Spohrer and Soloway, 1985] E.g., having the first item of data read outside the loop rather than inside the loop Another source: plan-like understanding [Rist, 1986, 1989] Very limited both in scope and depth (e.g., what's the nature of these bugs?) May not generalize to code tracing [Lopez et al., 2008; Robins et al., 2003] 5

6. Current study Original motivation: Learning modeling and personalization in adaptive practice systems Goal: Systematically investigate the composition effect (and thus skill integration) in code tracing regarding its prevalence, strength, sources of errors, and stability Extend prior work: On composition effect: Systematic error analysis for errors in skill integration On programming integration difficulties: Greater depth, code tracing New method : Difficulty Factors Assessment (DFA) VS. Contrasting answers between experts and novices [Lister et al., 2004; … ] RQs: RQ1: Is there a robust composition effect across problems and students? RQ2: What are the sources of errors for the composition effect? RQ3: What is the individual-level stability of the composition effect across problems, error types, and topics? 6

7. Methods: Difficulty Factors Assessment (DFA) The DFA approach was used to study the composition effect in algebra [Heffernan and Koedinger, 1997] Comparing the performance on a problem with a difficulty factor (e.g., involving two maths operators) and another without (involving one maths operator) ⇒ Uncover skills or factors that are most difficult for students The current work applies DFA in programming research (for the first time) 7 Key idea: Compare performance on decomposed vs. composed problems

8. Methods: Participants and Materiales An introductory Python programming course in the School of Informatics at the Austral University of Chile in 2017 Each session: 1 hr in-class paper quizzes, 2 weeks after the lecture on the topic 86 students in total (82% male, M age=19); part of course grades 17 problem sets in total (5+6+6) Students randomly assigned to one of the three forms 8

9. RQ1: Is there a robust composition effect across problems and students? Numeric value of the composition effect: Difference in the proportion correct between decomposed problems and the matched composed problem Descriptive stats of overall proportion correct: Integration problems: 0.60 Sequential problems: 0.75 Multiplication of Basic problems: 0.76 Significant composition effects overall and per topic (all p < 0.001) Overall: 0.16 (95% CI = [0.11, 0.20], N = 17) Per topic: for: 0.17, while: 0.15, lists: 0.15 9

10. RQ1: Is there a robust composition effect across problems and students? Most of the problem sets (88%), and the majority of students (60%) showed at least small composition effect sizes (details in our paper) Effect size of the composition effect: 10 The answer is yes!

11. RQ2: What are the sources of errors for the composition effect? We first present a check of whether our composition effect construct is independent of component skill levels (RQ2.1) Next, we present the qualitative coding of the errors Finally, we report our systematic investigation of the errors in terms of their relative frequencies and relative contributions to the strength of the composition effect (RQ2.2) 11

12. RQ2.1 Is the Composition Effect Independent of Component Skill Levels? There was no relationship between the composition effect a student showed and their component skill levels measured via either Sequential problems (Kendall’s Tau-b = –0.06, p = 0.52, 95% CI = [–0.27, 0.15], N = 60) or Basic problems (Tau-b = –0.08, p = 0.44, 95% CI = [–0.28, 0.13], N = 60) 12 The answer is yes!

13. Qualitative Coding of the Integration Errors Integration Errors : errors committed on an Integration problem when the same student succeeded in the matched Sequential problem Two coders: paper authors, experts in programming, inter-rater reliability Kappa = 0.71 Three categories: misunderstanding errors (MEs) , process errors (PEs) , and unclassified errors (UEs). We first defined the correct understanding of how components integrate, called Correct Understanding of Integration (CUI) E.g., Maintaining values across iterations (one kind of CUI): When conducting repeated operations using loops, the value of a variable defined outside a loop and updated in the loop body in an iteration is maintained when moving to the next iteration. If an answer was incompatible with CUI, then it was labeled as ME. If an answer was compatible with the CUI and contained the minimally implied answer properties from the CUI, then it was labeled as PE If neither ME nor PE categories could be applied , it was labeled as UE 13

14. 14

15. Validity of our coding All three types of identified MEs were closely related to specific misconceptions identified in prior literature in other contexts "Maintaining values across iterations" related to "confusing function and structure" [ Vainio and Sajaniemi , 2007] "Swapping variables" and "nested loops" related to a "state and execution" [ Kurvinen et al., 2016] The accumulated evidence across analyses (shown later) also support validity, e.g., Students exhibited stability within each error type but not across error types , supporting that our coding captured different types of errors With the increase of performance levels, there was a significant decrease in the number of MEs but a stable number of PEs, supporting that MEs are related to domain knowledge whereas PEs are related to WMC. 15

16. RQ2.2 How common and dominant is each error type underlying the composition effect? Students with different integrative skill levels had a similar number of PEs but varying MEs 16

17. RQ2.2 How common and dominant is each error type underlying the composition effect? The majority of problem sets (59%) involved both types of Integration Errors Most common source was MEs: 42% of all Integration Errors Another common source was PEs: 23% of all Integration Errors Most (65%) problem sets were dominated by MEs 35% of problem sets were dominated by PEs MEs were associated with larger effect sizes of the composition effect: Largest effect sizes all came from MEs dominant problem sets Regression predicting overall effect sizes, the standardized coefficient of the number of MEs (beta = 0.12, p < 0.001) was around two times as large as that of the number of PEs (beta = 0.05, p < 0.001) 17

18. RQ3: What is the individual-level stability of the composition effect across problems, error types, and topics? Analysis: 17 problem sets divided into halves, where each half covered a similar number of problem sets for each error type by topic (2 × 3) configurations [Pronk et al., 2022] Calculated correlation of the Mean Composition Effect (MCE; mean % correct Sequential problems - % correct Integration problems per student) between halves Results: Overall split-half stability of the composition effect across problem sets was moderate (Kendall’s Tau-b = 0.35, p < 0.001, 95% CI = [0.17, 0.54], N = 60) Students exhibited stability within each error type but not across error types, and the stability was stronger within ME problem sets than within PE problem sets: 18 (Kendall’s Tau-b with 95% CI, N = 60)

19. RQ3: What is the individual-level stability of the composition effect across problems, error types, and topics? Students also exhibited stability of the composition effect within each topic but not across topics 19 However, students exhibited stability across topics of their component skill levels

20. Conclusions We found a robust composition effect across problems and students in novice code tracing, mostly with small or medium effect sizes We identified two potential sources of errors for this composition effect: conceptual misunderstandings of how to integrate component skills and process errors resulting from slips related to cognitive load challenges. Additional conceptual knowledge often appears to be needed when component skills are integrated in specific ways The primary source appears to be misunderstandings of how skills integrate Composition effects may not generalize across topics, which suggests that domain topics may serve as the conceptual basis for the composition effect We found stable individual differences in susceptibility to the composition effect in both overall and finer-grained levels 20

21. Implications Theoretically: It contributes to building theories that generally explain challenges in complex skill learning: skill integration appears to be best explained by a combination of both the conceptual view and the procedural view The mental existence of latent integrative skills can be inferred from well-designed empirical investigations Practically: Importance of building curricula and instructions with focus on skill integration Personalized learning technologies that recognize students’ integration errors, dynamically track students’ integrative skills, and provide corresponding support 21

22. Limitations and future work Some of the coded MEs might still involve PEs Future work: design conceptual questions targeting hypothesized integrative skills, collect think-aloud or interview data We have not investigated in depth whether the underlying knowledge or skill of integration transfers to new contexts Replication in other student populations, programming skills, and programming paradigms Code writing Imperative programming here, what about functional programming? [Fisler et al., 2016] 22