A Study of Variable-Role-based Feature Enrichment in Neural Models of Code

Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.

- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia

1. A Study of Variable-Role-based
Feature Enrichment in Neural Models of Code
Aftab Hussain -- Md Rafiqul Islam Rabin -- Bowen Xu -- David Lo -- Mohammad Amin Alipour
University of Houston
Singapore Management University
InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering,
co-located with the 45th International Conference on Software Engineering, ICSE 2023
Melbourne, Australia

2. Deep Neural models are
widely being used in
Software Development
Visual Studio
Visual
Studio Code
Neovim

6. Need to investigate approaches that can improve
the performance of deep neural models, under
more sustainable settings.

7. Feature
Enrichment

8. Add extra information to the training data.

9. Add extra information to the training data.
E.g., indicate data dependence between
variables.

10. So what information exactly
can we add?

11. Related Works
Feature Enrichment

Allamanis et al. [26] showed adding features that capture global context can
increase the performance of a model.
Each feature is a binary function based on the context tokens.
Interesting features in
code that could help
performance of CI
Models.

12. Related Works
Feature Enrichment

Rabin et al. [27] found that code complexity features can improve the
classification performance

Recent studies have shown that state-of-the-art models heavily rely on
variables [13, 28], specific tokens [29], and even structures [30].

Interesting features in
code that could help
performance of CI
Models.

13. Related Works
Changing Code Representation (Code Modelling)

NLP models to capture textual patterns in code [20, 21].

Augment code with dataflow graph info GraphCodeBERT. [24].

Dynamic embeddings, improved performance in RNN for code completion and
bug fixing. [25]

14. We try an idea from the CS Education
Doman, avoiding need to change code
representation

15. -Sajaniemi et al., classified variables
in programs into role categories
based on how they are used.

Variable Roles

16. -Sajaniemi et al., classified variables
in programs into role categories
based on how they are used.

-They applied the role concepts in
teaching programming learning
tasks.
Variable Roles

17. -Sajaniemi et al., classified variables
in programs into role categories
based on how they are used.

-They applied the role concepts in
teaching programming learning
tasks.

-Roles helped students in
understanding programming
concepts and applying programming
steps in writing programs.
Variable Roles

18. Roles Examples

19. for (int i=0 ; i<5; i++){
...
}
Stepper
A numeric for-loop variable that
iterates through a range of values
via arithmetic-operation based
updates
Roles Examples

20. ArrayList<String> list = new
ArrayList<String>();
list.add("A");
list.add("B");
...
// Iterator to traverse the list
Iterator iter = list.iterator();
while iter.hasNext() {...}
for (String elem: Elements){
...
}





Walker Type I
An enhanced
for-loop variable
Walker Type II
A container
iterator object
Roles Examples

21. Can adding variable roles in training
neural models of code improve their
performance and robustness, and thus reduce
the effort needed on training?

22. We present an unsupervised feature enrichment approach
based on variable roles.

We evaluate its impact on
the performance and robustness of Code2Seq.

23. for (int i=0 ; i<5; i++){
...
}
Adding Roles Information

24. for (int i=0 ; i<5; i++){
...
}
Adding Roles Information

25. for (int stepper_i=0 ; stepper_i<5; stepper_i++){
...
}
Adding Roles Information

26. Input
Program
Our Feature Enrichment Approach

27. Role Detector
Input
Program
Our Feature Enrichment Approach

28. Role Detector
List of Stepper
and Walker
Variables
Input
Program
Our Feature Enrichment Approach

29. Role Detector
List of Stepper
and Walker
Variables
Role Augmenter
Input
Program
Our Feature Enrichment Approach

30. Role Detector
List of Stepper
and Walker
Variables
Role Augmenter
Input
Program
Output
Program
Our Feature Enrichment Approach

31. Evaluating our Approach

32. Evaluating our Approach

33. Evaluating our Approach

34. Evaluating our Approach

35. Evaluating our Approach

36. Evaluating our Approach

37. The Datasets
Java-Large
We augment the Java-Large dataset,
for the method name prediction task:
I/P-method body, O/P-method name

38. The Datasets
Java-Large
We augment the Java-Large dataset,
for the method name prediction task:
I/P-method body, O/P-method name

39. The Datasets
Java-Large
We augment the Java-Large dataset,
for the method name prediction task:
I/P-method body, O/P-method name

40. Evaluating our Approach

41. Evaluating our Approach
Code2Seq-R - best performing Code2Seq after 50 epochs of training
with Augmented Dataset

Code2Seq-O - Pretrained Released Version of Code2Seq that had
been obtained after 52 epochs of training with the Original Dataset.

42. Evaluating our Approach

43. Original
(JL Test)
Generating Test Sets for Performance Evaluation

44. Original
Original,
Roles-added
+ Roles
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

45. Original
Original,
Roles-added
Filter out un-augmented
methods
Original,
Filtered
Original,
Roles-added,
Filtered
+ Roles
(JLR Test (F))(JL Test (F))
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

46. Original
Original,
Roles-added
Filter out un-augmented
methods
Original,
Filtered
Original,
Roles-added,
Filtered
+ Roles
(JLR Test (F))(JL Test (F))
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

1. All inputs without roles info.
2. All inputs with roles info.
3. Inputs with steppers and variables
only, but without roles info.
4. Inputs with steppers and walkers only,
with roles info.

47. Original
Original,
Roles-added
Filter out un-augmented
methods
Original,
Filtered
Original,
Roles-added,
Filtered
+ Roles
(JLR Test (F))(JL Test (F))
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

1. All inputs without roles info.
2. All inputs with roles info.
3. Inputs with steppers and variables
only, but without roles info.
4. Inputs with steppers and walkers only,
with roles info.

48. Original
Original,
Roles-added
Filter out un-augmented
methods
Original,
Filtered
Original,
Roles-added,
Filtered
+ Roles
(JLR Test (F))(JL Test (F))
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

1. All inputs without roles info.
2. All inputs with roles info.
3. Inputs with steppers and variables
only, but without roles info.
4. Inputs with steppers and walkers only,
with roles info.

49. Original
Original,
Roles-added
Filter out un-augmented
methods
Original,
Filtered
Original,
Roles-added,
Filtered
+ Roles
(JLR Test (F))(JL Test (F))
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

1. All inputs without roles info.
2. All inputs with roles info.
3. Inputs with steppers and variables
only, but without roles info.
4. Inputs with steppers and walkers only,
with roles info.

50. Original
Original,
Roles-added
Filter out un-augmented
methods
Original,
Filtered
Original,
Roles-added,
Filtered
+ Roles
(JLR Test (F))(JL Test (F))
(JL Test) (JLR Test)
Generating Test Sets for Performance Evaluation

1. All inputs without roles info.
2. All inputs with roles info.
3. Inputs with steppers and walkers
only, but without roles info.
4. Inputs with steppers and walkers only,
with roles info.
We assess overall performance of the models using these test sets.

51. Original
(JL Test)
Generating Test Sets for Robustness Evaluation

We similarly get four
transformed
(noise-induced) test sets.

52. Original Transformed
Transform Variables (Noise induction)
(JLT Test)(JL Test)
Generating Test Sets for Robustness Evaluation


We similarly get four
transformed
(noise-induced) test sets.

53. Original Transformed
Transformed,
Roles-added
+ Roles
Transform Variables (Noise induction)
(JLT Test) (JLTR Test)(JL Test)
Generating Test Sets for Robustness Evaluation


We similarly get four
transformed
(noise-induced) test sets.

54. Original Transformed
Transformed,
Roles-added
Filter out un-augmented
methods
Transformed,
Filtered
Transformed,
Roles-added,
Filtered
+ Roles
Transform Variables (Noise induction)
(JLT Test) (JLTR Test)
(JLTR Test (F))(JLT Test (F))
(JL Test)
Generating Test Sets for Robustness Evaluation


We similarly get four
transformed
(noise-induced) test sets.

55. Original Transformed
Transformed,
Roles-added
Filter out un-augmented
methods
Transformed,
Filtered
Transformed,
Roles-added,
Filtered
+ Roles
Transform Variables (Noise induction)
(JLT Test) (JLTR Test)
(JLTR Test (F))(JLT Test (F))
(JL Test)
Generating Test Sets for Robustness Evaluation


We similarly get four
transformed
(noise-induced) test sets.
We assess overall robustness of the models using these test sets.

56. Results

57. Results – Overall Performance

58. Results – Performance
The impact of role augmentation is indistinguishable with respect to Code2Seq’s
performance in making method name predictions.

59. Results – Robustness
The impact of role augmentation is indistinguishable with respect to Code2Seq’s
robustness in making method name predictions.

60. Discussion

61. Discussion
- Code2Seq may already be capable of capturing the
surrounding structural context of a certain variable.

62. Discussion
- Code2Seq may already be capable of capturing the
surrounding structural context of a certain variable.

- Need to compare between variables more reliant on code
structure, and those that can be more flexibly used, e.g.,
fixed-value variables.

63. Discussion
- Code2Seq may already be capable of capturing the
surrounding structural context of a certain variable.

- Need to compare between variables more reliant on code
structure, and those that can be more flexibly used, e.g.,
fixed-value variables.

- Only two variable roles were considered. Although common,
they only covered 8% of the samples of the whole dataset
(1,117,159 of ~14m methods).

64. Conclusion
- We investigated the impact of explicitly adding variable role
information in code datasets on the performance of Code2Seq.

65. Conclusion
- We investigated the impact of explicitly adding variable role
information in code datasets on the performance of Code2Seq.

- To the best of our knowledge, this is the first work to evaluate
the impact of Sajaniemi et al.’s notion of variable roles, a
concept that was found to help students learn programming, on
neural models of code.

66. Conclusion
- We investigated the impact of explicitly adding variable role
information in code datasets on the performance of Code2Seq.

- To the best of our knowledge, this is the first work to evaluate
the impact of Sajaniemi et al.’s notion of variable roles, a
concept that was found to help students learn programming, to
neural models of code.

- This work motivates the need of a systematic framework on
how to provide models meaningful information to enable them
to learn faster and perform better.

67. Thank you