Asking Novices to Evaluate Code Quality: Understandability and Efficiency

The recent introduction of LLM tools is already having an impact on the way students acquire programming skills. When novices use such tools to develop programs, their coding effort is diminished, yet they need to apply high-level evaluation skills in order to validate the code generated by automatic means, as well as to decide if and to what extent it is appropriate for the task at hand. Consequently, CS1 courses should adapt their practice by shifting the emphasis from code-writing tasks to code-reading and evaluation tasks. To figure out ways to approach this change of instructional perspective, we conducted an exploratory investigation of how a cohort of novices evaluate code. More specifically, after CS1 students had engaged in a simple iterative task, we provided five different solutions for that task and asked them to identify (1) the most/least understandable, (2) the most/least efficient, and (3) the best/worst options overall. A qualitative analysis of their justifications provides insights into their code comprehension and the criteria behind their choices. The main criterion was Understandability (81%) which comprises readability (23%), comprehensibility (78%) and code structure (42%). Efficiency was also an important criterion (75%), nonetheless 40% incorrectly assessed the efficiency of at least one option.