Application of neural network methods for identifying speech flow characteristics in building digital profiles of financial services users

The article examines the application of neural network methods for identifying speech flow characteristics in order to build enhanced digital profiles of financial services users. The relevance of the study is driven by the development of remote forms of banking and microfinance services, in which speech communication becomes one of the significant sources of information about the client. The main prosodic characteristics of speech are described, including speech rate, loudness, pitch, pauses, and rhythm, as well as their relationship with psychological personality types according to V.V. Ponomarenko's "7 radicals" methodology. Special attention is paid to the neural network approach to speech signal analysis: preliminary digital processing of the audio recording, extraction of acoustic features, formation of a prosodic parameter vector, and subsequent classification of psychological radicals using modern architectures such as CNN, LSTM, and attention mechanisms. An information technology for decision-making in the banking and microfinance sectors is presented, which involves incorporating a speech information analysis module into the client's digital profile building system. It is shown that the use of a probabilistic approach and multicriteria analysis makes it possible to improve the objectivity of personality assessment, reduce subjective uncertainty in interaction with the borrower, and increase the accuracy of credit scoring. A conclusion is made about the prospects of using neural network analysis of speech characteristics for financial risk management and personalization of financial services.

1. Safuanov S.F., Nazarova E.A. Different methods of psychological portrait preparation for an alleged offender: a comparative analysis. Psychology and Law. 2011;(3):33–43. (In Russ.).

2. Petraevskiy V.A., Kravets A.G. The textual information intellectual analysis method for psychiatric diagnosis. Vestnik of Astrakhan State Technical University. Series: Management, Computer Science and Informatics. 2024;(2):95–104. (In Russ.).

3. Demina R.Y., Ashimov M.M., Lepekhin D.A., Esaulenko V.N. Formation of a training set based on debtors' data stored by a collection agency. Caspian Journal: Management and High Technologies. 2025;(3):38–45. (In Russ.).

4. Kuznetsova V.Yu. Information technology for decision making in a microfinance organization. Systems Engineering and Information Technologies. 2023;5(3):27–41. (In Russ.).

5. Ponomarenko V.V. Practical characterology with elements of forecasting and behavior management: the "7 radicals" methodology. Moscow: Print Pro; 2017. 206 p. (In Russ.).

6. Yang Y., Qin Y., Fan Y., Zhang Zh. Unlocking the Power of Voice for Financial Risk Prediction: A Theory-Driven Deep Learning Design Approach. MIS Quarterly. 2023;47(1):63–96. https://doi.org/10.25300/MISQ/2022/17062

7. Akay O.M. Computational Modeling of Prosody-Syntax Correlations in Russian Spoken Speech: Machine Learning Methods and Linguistic Analysis. In: Russian Grammar: polyparadigmality as a methodological principle of modern scientific research: Proceedings of the IX International Scientific Symposium, 23–27 September 2025, Irkutsk, Russia. Irkutsk: Izdatel'stvo IGU; 2025. P. 416–422. (In Russ.).

8. Chen X., Yu X., Chang L., et al. The Sound of Risk: A Multimodal Physics-Informed Acoustic Model for Forecasting Market Volatility and Enhancing Market Interpretability. arXiv. URL: https://arxiv.org/abs/2508.18653 [Accessed 19th August 2026].

9. Kuzovlev K.D., Kuznetsova V.Yu. A method for identifying psycho-emotional impact in an audio stream based on neural network analysis. Engineering Journal of Don. 2026;(6). (In Russ.). URL: https://ivdon.ru/en/magazine/archive/n6y2026/11269

10. Trigeorgis G., Ringeval F., Brueckner R., et al. Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network. In: Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, 20–25 March 2016, Shanghai, China. IEEE; 2016. P. 5200–5204. https://doi.org/10.1109/ICASSP.2016.7472669

11. Schuller B.W., Steidl S., Batliner A., et al. The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. In: Proceedings of the 14th Annual Conference of the International Speech Communication Association, 25–29 August 2013, Lyon, France. ISCA; 2013. P. 148–152. https://doi.org/10.21437/Interspeech.2013-56

12. Zhao J., Mao X., Chen L. Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control. 2019;47(4):312–323. https://doi.org/10.1016/j.bspc.2018.08.035

13. Mirsamadi S., Barsoum E., Zhang Ch. Automatic speech emotion recognition using recurrent neural networks with local attention. In: Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, 05–09 March 2017, New Orleans, LA, USA. IEEE; 2017. P. 2227–2231. https://doi.org/10.1109/ICASSP.2017.7952552

14. Marienkov A.N., Krivenko A.I. Collection and processing of textual data in the context of social sentiment assessment: methodological aspects. Engineering Journal of Don. 2022;(7). (In Russ.). URL: https://ivdon.ru/en/magazine/archive/n7y2022/7797