The main approach to solving problems in programming courses often consists of writing and testing individual parts of an algorithm written in a particular language. Students make several attempts to submit the problem to the testing system, each of these attempts reflecting an individual solution state. Usually, to determine the student performance, the average number of submissions to pass the solution or focusing on time taken to complete the problem correctly is calculated. Such metrics are usually not robust, because the time to correct individual errors significantly affects the total time to solve the problem. Also, these metrics do not reflect what the student does not understand in the theoretical aspect. This article proposes a metric based on the probabilistic distance between an observed student solution and a correct solution. As part of the experiment, a group of students solved problems in an online programming environment. Their submissions were evaluated against a model of the algorithmic component necessary for a correct solution. A Markov Model was used to generate problem state graph, connecting program states. The proposed metric of the probabilistic distance to solution was applied to the graph to determine the distances from each solution to the nearest correct ones. The results showed that this metric is useful in determining the distance if the path to the correct solution was typical and consistent with the studied theoretical material. The article provides details of the implementation of the metric of probabilistic distances to the solution and a plan for further research based on current observations.
1. Barnes T., Stamper J. Automatic Hint Generation for Logic Proof Tutoring Using Historical Data. Educational Technology & Society. 2010;13(1):3-12. Available at: https://www.j-ets.net/collection/published-issues/13_1 (accessed 21.01.2021).
2. Valenti S., Neri F. An Overview of Current Research on Automated Essay Grading. Journal of Information Technology Education. 2013;2:319–330. Available at: https://www.informingscience.org/Publications/331 DOI: 10.28945/331 (accessed 21.01.2021).
3. Ihantola P., Ahoniemi T., Karavirta V. Review of Recent Systems for Automatic Assessment of Programming Assignments. 10th Koli Calling International Conference on Computing Education Research. 2010:86–93. Available at: https://dl.acm.org/doi/10.1145/1930464.1930480 DOI: 10.1145/1930464.1930480 (accessed 21.01.2021).
4. Esin T.E., Glukhikh I.N. Automation of Personalized Feedback in the Programming Studies Courses. Modeling, Optimization and Information Technology. 2019;7(1). Available at: http://moitvivt.ru/journal/pdf?id=589. DOI: 10.26102/2310-6018/20 (In Russ) DOI: 10.26102/2310-6018/2019.24.1.043 (accessed 21.01.2021).
5. Jadud M. A First Look at Novice Compilation Behaviour Using BlueJ. Computer Science Education. 2005;15(1):25–40. Available at: https://www.tandfonline.com/doi/abs/10.1080/ DOI: 10.1080/08993400500056530 (accessed 21.01.2021).
6. Feng M., Heffernan N. Predicting State Test Scores Better with Intelligent Tutoring Systems»: Developing Metrics to Measure Assistance Required. 8th International Conference on Intelligent Tutoring Systems. 2006:31–40. Available at: https://dl.acm.org/doi/10.1007/11774303_4 DOI: 10.1007/11774303_4 (accessed 21.01.2021).
7. Lane H., Vanlehn K. Intention-Based Scoring: An Approach to Measuring Success at Solving the Composition Problem. 36th SIGCSE technical symposium on Computer science education. 2005:373–377. Available at: https://dl.acm.org/doi/10.1145/1047124.1047471 DOI: 10.1145/1047124.1047471 (accessed 21.01.2021).
8. Mitrovic A. An Intelligent SQL Tutor on the Web. International Journal of Artificial Intelligence in Education. 2003;13(2-4):173–197. Available at: https://iaied.org/journal/963 (accessed 21.01.2021).
9. Le N., Menzel W. Using Constraint-Based Modelling to Describe the Solution Space of Ill-defined Problems in Logic Programming. Advances in Web Based Learning (ICSL 2007). 2007:367–379. Available at: https://link.springer.com/chapter/10.1007/978-3-540-78139-4_33 DOI: 10.1007/978-3-540-78139-4_33 (accessed 21.01.2021).
