Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis
Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D'Antoni, and Bjoern Hartmann
In large classes, teacher feedback on individual student coding mistakes is not typically feasible. Feedback is often limited to autograders that only report the results of running student code against a test suite. Recent research in program synthesis has improved the utility of autograders by generating potential fixes to student code and hinting at them to students. However, these fully automatic approaches lack a teacher’s deep domain knowledge and can generate technically correct but subjectively bad fixes that no teacher would recommend.
We contribute a new mixed-initiative approach which combines the teacher’s judgement and expertise with data-driven program synthesis techniques that cluster buggy student solutions and learn code transformations from examples. We describe two implementations of this idea. In the MISTAKEBROWSER system, transformations are learned from examples of student-written bug fixes. In contrast, the FIXPROPAGATOR system interactively learns transformations as the teacher fixes student bugs. Regardless of how transformations are learned, the teacher then adds domain knowledge in the form of feedback and hints for each cluster. Feedback can be returned to the original students, but can also be used for future students whose buggy solutions fall into the same cluster. Two studies with teaching assistants suggest that this mixed-initiative approach helps instructors better understand student difficulties for an assignment and helps the teacher provide automatically reusable feedback to a larger number of students.