DEVELOPMENT OF ARCHITECTURE OF REPOSITORY FOR STORING MULTIMODAL DATA

The article discusses the main features of the design and development of a platform solution for the organization of research work based on multimodal data. In the context of a multiple increase in research data collected in experiments from various sources, as well as an increase in interest in conducting interdisciplinary research in recent years, the development of tools for collecting, storing and processing multimodal data has become urgent. At the same time, the communicative process, which includes audio, video, semantic modality, as well as modalities received from various sensors, is considered as the initial data source. Based on the study, the article formulated the requirements for the repository, and also proposed a model of its architecture. The features of the digital presentation of multimodal data in the form of a linear combination of modalities are considered, an approach to the implementation and development of the repository based on the principles of microservice architecture is presented. The implemented platform is designed to manage and jointly process data in multidisciplinary collective research, including using remote workstations. The architecture was built using the paradigm of reactive interaction of system components, as well as the principles of asynchronous access.

1. Basov O.O. Models and method for the synthesis of multimodal infocommunication systems:phD thesis. Orel. 2016:292.

2. Liu H. et al. Deep learning-based multimodal control interface for human-robot collaboration. Procedia CIRP. 2018;72:3-8.

3. Casanovas A.L. et al. Blind audiovisual source separation based on sparse redundant representations. IEEE Transactions on Multimedia. 2010;12(5):358-371.

4. Shoumy N.J. et al. Multimodal big data affective analytics:A comprehensive survey using text, audio, visual and physiological signals. Journal of Network and Computer Applications. 2019:102447.

5. Kendon A. Gesture:Visible action as utterance. Cambridge University Press, 2004.

6. Christine C.A. et al. A cyberplatform for sharing scientific research data at DataCenterHub. Computing in Science & Engineering. 2018;20(3):49.

7. Parashar M. et al. The Virtual Data Collaboratory. Computing in Science & Engineering. 2019.

8. Lahat D., Adali T., Jutten C. Multimodal data fusion:an overview of methods, challenges, and prospects. Proceedings of the IEEE. 2015;103(9):1449-1477.

9. Kamaruddin N., Wahab A. Interlaboratory data fusion repository system (InDFuRS) for tocotrienols-based treatment. Indones. J. Electr. Eng. Comput. Sci. 2019;13(3):1130-1135.

10. Zheng W.L. et al. Emotionmeter:A multimodal framework for recognizing human emotions. IEEE transactions on cybernetics. 2018;49(3):1110-1122.

11. Jansen M.P. Communicative Signals and Social Contextual Factors in Multimodal Affect Recognition. 2019 International Conference on Multimodal Interaction. ACM, 2019:468- 472.

12. Mannheimer S. et al. Qualitative data sharing:Data repositories and academic libraries as key partners in addressing challenges. American Behavioral Scientist. 2019; 63(5):643-664.

13. Antonio M.G. et al. Qualitative Data Management and Analysis within a Data Repository. Western journal of nursing research. 2019:0193945919881706.

14. DuBois J.M., Strait M., Walsh H. Is it time to share qualitative research data? Qualitative Psychology. 2018;5(3). С. 380.

15. Wilkinson M.D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data. 2016;3.

16. Perazzo J. et al. Creation of Data Repositories to Advance Nursing Science. Western journal of nursing research. 2019;41(1):78-95.