Decision support based on error report clustering in complex open ended assignments quality control

There are processes among the processes of different organizations related to carrying out tasks, implementation of which is controlled manually. This is because of a lack of result-template for the tasks. There is only the system of requirements, which implemented task must satisfy. These tasks are known as complex open ended assignments in online learning. However, the tasks exist in other fields, for example, in the publication process, in the equipment and device production process, etc. Complex open ended assignment quality control stage is ineffective due to time-consuming work of an inspector, who checks the conformity of the tasks against the requirements and prepares feedback for a performer. Intellectual support is beginning to be used for a series of tasks. Intellectual support is based upon automatic task implementation classification with the use of machine learning. However, automatic classification can bring to incorrect task implementation quality assessment. Also classifier does not generate a detailed feedback, which fit for a revision of implemented task. A decision support method based on error report clustering, which allows to create a detailed feedback on implemented complex open ended assignments, is suggested in the paper. Special software, which in conjunction with existing clustering system Carrot2 executes suggested method, is developed. The software is introduced in online pre-defense of graduation qualification thesis process. This led to time reduction in feedback preparing by an inspector.

1. Latypova V.A. A concept of online automated training process management in implementing complex open ended assignments based on the use of error bank. Modeling, Optimization and Information Technology. 2019;7(3). Available from: moit.vivt.ru/wpcontent/uploads/2019/09/Latypova_3_19_1.pdf (In Russ) (accessed 28.09.2020). doi: 10.26102/2310-6018/2019.26.3.015

2. Balfour S. Assessing writing in MOOCs: automated essay scoring and calibrated peer review. Research & Practice in Assessment. 2013;8:40-48.

3. Heaven D. AI peer reviewers unleashed to ease publishing grind. Nature. 2018;563(7733):609-610. doi: 10.1038/d41586-018-07245-9.

4. Kirichenko K.M., Gerasimov M.B. Obzor metodov klasterizatsii tekstovoi informatsii. Proceedings Dialog. 2001. Available from: www.dialog21.ru/digest/2001/articles/kirichenko (In Russ) (accessed 28.09.2020).

5. Nagwani N.K., Verma S. Software Bug Classification using Suffix Tree Clustering (STC) Algorithm. IJCST. 2011;2(1): 36-41.

6. Hammad M., Alzyoudi R., Otoom A.F. Automatic clustering of bug reports. IJACR. 2018; 8(39):313-323. doi: 10.19101/IJACR.2018.839013.

7. Osinski S., Stefanowski J., Weiss D. Lingo: search results clustering algorithm based on singular value decomposition. Intelligent Information Processing and Web Mining. Advances in Soft Computing. Springer, Berlin, Heidelberg. 2004;25:359-368. doi: 10.1007/978-3-540-39985-8_37.

8. Osinski, S. An Algorithm for Clustering of Web Search Results. Master’s thesis. Poznan University of Technology, Poland. 2003. Available from: citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.5832&rep=rep1&type=pdf (accessed 28.09.2020).

9. Popov M.P. Effektivnye priemy nabora i redaktirovaniya teksta. SPb.:BKhV-Peterburg. 2006. (In Russ)

10. Carrot2 official site. Available from: project.carrot2.org (accessed 28.09.2020).

11. Latypova V.A. Programma: bank oshibok. Svidetel'stvo o registratsii programmy dlya EVM RU 2016611178, 27.01.2016. Zayavka № 2015619438 ot 06.10.2015. (In Russ)

12. Latypova V.A. Programma sbora informatsii pri upravlenii protsessom obucheniya pri reshenii slozhnykh otkrytykh zadach. Svidetel'stvo o registratsii programmy dlya EVM RU 2019663469, 17.10.2019. Zayavka № 2019662317 ot 07.10.2019. (In Russ)