A mathematical model of the process of automated construction of review texts based on the use of quotation-aware summarization

This paper examines the automated generation of scientific review texts based on the analysis of a corpus of scientific publications. A mathematical model has been developed that describes an approach based on the use of citation-aware summarization with preliminary selection of citation fragments and formalizes the full data processing cycle, from publication selection and extraction of citing fragments to the generation of individual summaries and the final review text. The model defines a unified formal description of the sequence of data transformation operators and includes a system of quality criteria that ensure control of the plausibility, coverage, and factual consistency of results at all stages of processing. A software system in the form of a modular pipeline is implemented based on the developed model. Experimental studies using the developed model were conducted on the SurGE dataset, which includes 114 topics, over 7,000 cited and over 73,000 citing publications. It is shown that the use of a citation-aware approach with preliminary fragment selection improves the quality of summary generation compared to alternative methods. The following quality criteria were achieved for the resulting review texts: credibility – 0.8744, coverage – 0.9356, factual reliability – 0.9713, and LLM score – 0.9232, which outperforms the results of generation based on the full text of sources (by 7.67 % for credibility, 4.63 % for coverage, 9.22 % for factual consistency, and 4.42 % for LLM score). The obtained results confirm the effectiveness of the proposed approach and the developed model and their applicability for the automated generation of reliable scientific reviews.

1. Syed A.A., Gaol F.L., Boediman A., et al. A survey of abstractive text summarization utilising pretrained language models. In: Intelligent Information and Database Systems: 14th Asian Conference (ACIIDS 2022): Proceedings: Part I, 28–30 November 2022, Ho Chi Minh City, Vietnam. Cham: Springer; 2022. P. 532–544. https://doi.org/10.1007/978-3-031-21743-2_42

2. Afsharizadeh M., Ebrahimpour-Komleh H., Bagheri A., et al. A survey on multi-document summarization and domain-oriented approaches. Journal of Information Systems and Telecommunication. 2022;10(37):68–78. https://doi.org/10.52547/jist.16245.10.37.68

3. Keya F., Jaradeh M.Y., Auer S. Leveraging LLMs for scientific abstract summarization: unearthing the essence of research in a single sentence. In: Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries, 16–20 December 2024, Hong Kong, China. New York: ACM; 2024. https://doi.org/10.1145/3677389.3702588

4. Scherbakov D., Hubig N., Jansari V., et al. The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review. Journal of the American Medical Informatics Association. 2025;32(6):1071–1086. https://doi.org/10.1093/jamia/ocaf063

5. Wu Sh., Ma X., Luo D., et al. Automated literature research and review-generation method based on large language models. National Science Review. 2025;12(6):nwaf169. https://doi.org/10.1093/nsr/nwaf169

6. Guo B., Wen Zh., Yang Y., et al. SGSimEval: a comprehensive multifaceted and similarity-enhanced benchmark for automatic survey generation systems. In: Advanced Data Mining and Applications: 21st International Conference (ADMA 2025): Proceedings: Part II, 22–24 October 2025, Kyoto, Japan. Singapore: Springer; 2025. P. 393–407. https://doi.org/10.1007/978-981-95-3456-2_27

7. Wu S., Liang Ch., Bi Z., et al. AutoSurvey2: empowering researchers with next level automated literature surveys. arXiv. URL: https://doi.org/10.48550/arXiv.2510.26012 [Accessed 7th April 2026].

8. Li Y., Datta S., Rastegar-Mojarad M., et al. Enhancing systematic literature reviews with generative artificial intelligence: development, applications, and performance evaluation. Journal of the American Medical Informatics Association. 2025;32(4):616–625. https://doi.org/10.1093/jamia/ocaf030

9. Su W., Xie A., Ai Q., et al. SurGE: a benchmark and evaluation framework for scientific survey generation. arXiv. URL: https://doi.org/10.48550/arXiv.2508.15658 [Accessed 19th April 2026].

10. Zhao J., Zhang Sh., Xu N., et al. SurveyEval: towards comprehensive evaluation of LLM-generated academic surveys. arXiv. URL: https://doi.org/10.48550/arXiv.2512.02763 [Accessed 14th April 2026].

11. Ji Z., Lee N., Frieske R., et al. Survey of hallucination in natural language generation. ACM Computing Surveys. 2023;55(12):248. https://doi.org/10.1145/3571730

12. Maynez J., Narayan Sh., Bohnet B., et al. On faithfulness and factuality in abstractive summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 05–10 July 2020, Virtual Event. Association for Computational Linguistics; 2020. P. 1906–1919. https://doi.org/10.18653/v1/2020.acl-main.173

13. Lamba N., Tiwari S., Gaur M. Hallucinations in scholarly LLMs: a conceptual overview and practical implications. In: The Second Bridge on Artificial Intelligence for Scholarly Communication (AAAI-26), 20 January 2026, Singapore. 2026. https://doi.org/10.52825/ocp.v8i.3175

14. Lin S., Hilton J., Evans O. TruthfulQA: measuring how models mimic human falsehoods. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 22–27 May 2022, Dublin, Ireland. Association for Computational Linguistics; 2022. P. 3214–3252. https://doi.org/10.18653/v1/2022.acl-long.229

15. Alkaissi H., McFarlane S.I. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. 2023;15(2):e35179. https://doi.org/10.7759/cureus.35179

16. Izacard G., Lewis P., Lomeli M., et al. Atlas: few-shot learning with retrieval augmented language models. Journal of Machine Learning Research. 2023;24:251.

17. Kung T.H., Cheatham M., Medenilla A., et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023;2(2):e0000198. https://doi.org/10.1371/journal.pdig.0000198

18. Anh-Hoang D., Tran V., Nguyen L.-M., et al. Survey and analysis of hallucinations in large language models: attribution to prompting strategies or model behavior. Frontiers in Artificial Intelligence. 2025;8:1622292. https://doi.org/10.3389/frai.2025.1622292

19. Guu K., Lee K., Tung Z., et al. Retrieval augmented language model pre-training. In: Proceedings of the 37th International Conference on Machine Learning, 13–18 July 2020, Virtual Event. PMLR; 2020. P. 3929–3938.

20. Han B., Susnjak T., Mathrani A. Automating systematic literature reviews with retrieval-augmented generation: a comprehensive overview. Applied Sciences. 2024;14(19):9103. https://doi.org/10.3390/app14199103

21. Lewis P., Perez E., Piktus A., et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 06–12 December 2020, Virtual Event. 2020.

22. Li Y., Du X., Wang Y., et al. AI-assisted literature screening: a hybrid approach using large language models and retrieval-augmented generation. International Journal of Medical Informatics. 2025;207:106205. https://doi.org/10.1016/j.ijmedinf.2025.106205

23. Asai A., He J., Shao R., et al. Synthesizing scientific literature with retrieval-augmented language models. Nature. 2026;650(8103):857–863. https://doi.org/10.1038/s41586-025-10072-4

24. Bao T., Nayeem M.T., Rafiei D., et al. SurveyGen: quality-aware scientific survey generation with large language models. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 04–09 November 2025, Suzhou, China. Association for Computational Linguistics; 2025. P. 2712–2736. https://doi.org/10.18653/v1/2025.emnlp-main.136

25. Shi X., Kou Q., Li Y., et al. SciSage: a multi-agent framework for high-quality scientific survey generation. arXiv. URL: https://doi.org/10.48550/arXiv.2506.12689 [Accessed 12th April 2026].

26. Sun Zh., Zhu X., Zhou X., et al. SurveyBench: can LLM(-agents) write academic surveys that align with reader needs? arXiv. URL: https://doi.org/10.48550/arXiv.2510.03120 [Accessed 11th April 2026].

27. Syed Sh., Hakimi A.D., Khatib Kh.A., et al. Citance-contextualized summarization of scientific papers. In: Findings of the Association for Computational Linguistics (EMNLP 2023), 06–10 December 2023, Singapore. Association for Computational Linguistics; 2023. P. 8551–8568. https://doi.org/10.18653/v1/2023.findings-emnlp.573

28. Saini N., Kumar S., Saha S., et al. Scientific document summarization using citation context and multi-objective optimization. In: 2020 25th International Conference on Pattern Recognition, 10–15 January 2021, Milan, Italy. IEEE; 2021. P. 4290–4295. https://doi.org/10.1109/icpr48806.2021.9412201

29. Mao Y., Zhong M., Han J. CiteSum: Citation text-guided scientific extreme summarization and domain adaptation with limited supervision. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 07–11 December 2022, Abu Dhabi, UAE. Association for Computational Linguistics; 2022. P. 10922–10935. https://doi.org/10.18653/v1/2022.emnlp-main.750

30. Gu N., Hahnloser R.H.R. SciLit: A platform for joint scientific literature discovery, summarization and citation generation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 09–14 July 2023, Toronto, Canada. Association for Computational Linguistics; 2023. P. 235–246. https://doi.org/10.18653/v1/2023.acl-demo.22

31. Zhu K., Feng X., Feng X., et al. Hierarchical catalogue generation for literature review: a benchmark. In: Findings of the Association for Computational Linguistics (EMNLP 2023), 06–10 December 2023, Singapore. Association for Computational Linguistics; 2023. P. 6790–6804. https://doi.org/10.18653/v1/2023.findings-emnlp.453

32. Groppe S., Hartung L. ReViz: a tool for automatically generating citation graphs and variants. In: Digital Libraries at Times of Massive Societal Transition: 22nd International Conference on Asia-Pacific Digital Libraries (ICADL 2020), 30 November – 01 December 2020, Kyoto, Japan. Cham: Springer; 2020. P. 107–121. https://doi.org/10.1007/978-3-030-64452-9_10

33. Hu Y., Lei Zh., Dai Zh., et al. CG-RAG: research question answering by citation graph retrieval-augmented LLMs. In: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, 13–18 July 2025, Padua, Italy. New York: ACM; 2025. P. 678–687. https://doi.org/10.1145/3726302.3729920

34. Ding H., Zhao Y., Hu T., et al. SciRAG: adaptive, citation-aware, and outline-guided retrieval and synthesis for scientific literature. In: Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, 24–29 March 2026, Rabat, Morocco. Association for Computational Linguistics; 2026. P. 6440–6460. https://doi.org/10.18653/v1/2026.eacl-long.303

35. Menick J., Trebacz M., Mikulik V., et al. Teaching language models to support answers with verified quotes. arXiv. URL: https://doi.org/10.48550/arXiv.2203.11147 [Accessed 7th May 2026].

36. Li W., Xiao X., Liu J., et al. Leveraging graph to improve abstractive multi-document summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 05–10 July 2020, Virtual Event. Association for Computational Linguistics; 2020. P. 6232–6243. https://doi.org/10.18653/v1/2020.acl-main.555

37. An C., Zhong M., Chen Y., et al. Enhancing scientific papers summarization with citation graph. Proceedings of the AAAI Conference on Artificial Intelligence. 2021;35(14):12498–12506. https://doi.org/10.1609/aaai.v35i14.17482

38. Chen J., Cai Ch., Jiang X., et al. Comparative graph-based summarization of scientific papers guided by comparative citations. In: Proceedings of the 29th International Conference on Computational Linguistics, 12–17 October 2022, Gyeongju, Republic of Korea. International Committee on Computational Linguistics; 2022. P. 5978–5988.

39. Wadden D., Lo K., Kuehl B., et al. SciFact-open: Towards open-domain scientific claim verification. In: Findings of the Association for Computational Linguistics (EMNLP 2022), 07–11 December 2022, Abu Dhabi, UAE. Association for Computational Linguistics; 2022. P. 4719–4734. https://doi.org/10.18653/v1/2022.findings-emnlp.347

40. Zheng T., Deng Zh., Tsang H.T., et al. From automation to autonomy: a survey on large language models in scientific discovery. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 04–09 November 2025, Suzhou, China. Association for Computational Linguistics; 2025. P. 17733–17750. https://doi.org/10.18653/v1/2025.emnlp-main.895

41. Beltagy I., Lo K., Cohan A. SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 03–07 November 2019, Hong Kong, China. Association for Computational Linguistics; 2019. P. 3615–3620. https://doi.org/10.18653/v1/d19-1371

42. Lin Ch.-Y. ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, 25–26 July 2004, Barcelona, Spain. Association for Computational Linguistics; 2004. P. 74–81.

43. Zhang T., Kishore V., Wu F., et al. BERTScore: evaluating text generation with BERT. In: 8th International Conference on Learning Representations, 26–30 April 2020, Addis Ababa, Ethiopia. 2020. URL: https://openreview.net/pdf?id=SkeHuCVFDr

44. Deutsch D., Bedrax-Weiss T., Roth D. Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. Transactions of the Association for Computational Linguistics. 2021;9:774–789. https://doi.org/10.1162/tacl_a_00397

45. Chen Y., Eger S. MENLI: robust evaluation metrics from natural language inference. Transactions of the Association for Computational Linguistics. 2023;11:804–825. https://doi.org/10.1162/tacl_a_00576

46. Zhang Q., Wang Y., Jiang Y., et al. Crowd comparative reasoning: unlocking comprehensive evaluations for LLM-as-a-judge. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 27 July – 01 August 2025, Vienna, Austria. Association for Computational Linguistics; 2025. P. 5059–5074. https://doi.org/10.18653/v1/2025.acl-long.252