Situation-oriented databases: processing heterogeneous documents of microservices in a document-based storage

The research is focused on a situation-oriented approach to the processing of heterogeneous data obtained from microservices that are widespread due to the implementation of the microservice architecture underlying many information systems. Such information systems are sources of heterogeneous data provided to the user upon request via the Internet. Data in the form of documents is provided by services included in the information system. The volume of such data can be large, and its processing requires specialized technologies available in document-oriented big data storages (SODB). As part of a situationally oriented database, a microservice is implemented that provides data in JSON format through its programming interface. There is a problem of loading and processing large amounts of data in the storage where specialized statistical functions of Map-Reduce are implemented. The manual method of loading and obtaining results for SODB is laborious because it requires the implementation of routine operations for loading data, applying functions to the loaded data, creating functions inside the storage and obtaining results. This task was not considered within the scope of the project on creating situation-oriented databases, and the possibilities for developing specialized elements and methods for processing large-scale heterogeneous data in a hierarchical situational model with the required equipment were not studied. The developed models for processing documents make processing heterogeneous data less laborious and help to create data-driven applications by means of situation-oriented databases in the framework of the introduced data processing model as part of a hierarchical situational model with the involvement of big data processing technologies of specialized document-oriented storages. The proposed tools are examined by the example of the SODB application for solving the problems of course design in the educational process using the developed microservice saturated with heterogeneous data collected while designing a course remotely.

1. Wilde E., Pautasso C. REST: From Research to Practice. Springer Science & Business Media. 2011:528. DOI:10.1007/978-1-4419-8303-9.

2. Mironov V.V., Gusarenko A.S., Yusupova N.I. Situation-oriented databases: polyglot persistence based on REST microservices. Prikladnaya informatika = Applied Informatics. 2019;5(83):87–97. DOI:10.24411/1993-8314-2019-10038. (In Russ.).

3. Mironov V.V., Gusarenko A.S., Tuguzbaev G.A. Extracting semantic information from graphic schemes. Informatika i avtomatizatsiya = Informatics and Automation. 2021;20(4):940–970. DOI:10.15622/IA.20.4.7. (In Russ.).

4. Gusarenko A.S., Mironov V.V., Yusupova N.I. Stream processing of large documents in situationally oriented databases. ITIDS’2018: Trudy 6-oy mezhdunarodnoy konferentsii po Informatsionnym tekhnologiyami intellektual'noy podderzhki prinyatiya resheniy = ITIDS’2018: Proceedings of the 6-th International Conference Information Technologies for Intelligent Decision Making Support. Ufa, Russia: USATU; 2018:7–12. (In Russ.)

5. Seera N.K., Taruna S. Analyzing Cost Parameters Affecting Map Reduce Application Performance. International Journal of Information Technology and Computer Science. 2016;8(8):50–58. DOI:10.5815/ijitcs.2016.08.06.

6. Papadopoulos A., Manolopoulos Y. Parallel bulk-loading of spatial data. Parallel Computing. 2003;29(10):1419–1444. DOI:10.1016/j.parco.2003.05.003.

7. Mironov V.V., Gusarenko A.S., Tuguzbaev G.A. Situation-oriented databases: the formation of personalized graphic documents for educational design support. Modelirovaniye, optimizatsiya i informatsionnyye tekhnologii = Modeling, Optimization and Information Technology. 2020;8(2):1–18. DOI:10.26102/2310-6018/2020.29.2.013. (In Russ.).

8. Gusarenko A.S. Improvement of Situation-Oriented Database Model for Interaction with Mysql. Izvestiya vysshikh uchebnykh zavedeniy Priborostroyeniye =Journal of Instrument Egineering. 2016;59(5):355–63. DOI:10.17586/0021-3454-2016-59-5-355-363. (In Russ.).

9. Mironov V.V., Gusarenko A.S., Yusupova N.I., Smetanin Y.G. JSON documents processing using situation-oriented databases. Acta Polytechnica Hungarica. 2020;17(8):29–40. DOI:10.12700/APH.17.8.2020.8.3.

10. Mironov V.V., Gusarenko A.S., Yusupova N.I. Monitoring YouTube Video Views in the Educational Environment Based on Situation-Oriented Database and RESTful Web Services. Sistemnaya inzheneriya i informatsionnyye tekhnologii = Systems Engineering and Information Technologies. 2021;3(1(5)):39–49.

11. Mironov V.V., Gusarenko A.S., Yusupova N.I. Building of Virtual Multidocuments Mapping to Real Sources of Data in Situation-Oriented Databases. Communications in Computer and Information Science. 1204 CCIS. 2021:167–178. DOI:10.1007/978-3-030-78273-3_17.

12. Kolonko M., Mullenbach S., Polyglot Persistence in conceptual modeling for information analysis. ACIT’2020: Proc. 10th Int. Conf. on Advanced Computer Information Technologies. 2020:590–594. DOI:10.1109/ACIT49673.2020.9208928.

13. Mironov V.V., Gusarenko A.S., Yusupova N.I. Situation-oriented databases: polyglot persistence based on REST microservices. Applied Informatics. 2019;14(5(83)):87–97. DOI: 10.24411/1993-8314-2019-10038. (In Russ.).

14. Mironov V.V., Gusarenko A.S., Yusupova N.I. Stream handling large volume documents in situationally-oriented databases. International Scientific Journal INDUSTRY 4.0 Scientific Technical Union of Mechanical Engineering “INDUSTRY 4.0.” 2018;3(5):240–4.

15. Mironov V., Gusarenko A., Yusupova N. Stream Documents Processing Invariance in Situation-Oriented Databases. 7th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS’2019). Atlantis Press; 2019:309–15. DOI:10.2991/itids-19.2019.55.

16. Sakr S., Zomaya A.Y. Handbook of Big Data Technologies [Internet]. Springer International Publishing; 2017. Available from: https://link.springer.com/book/10.1007/978-3-319-49340-4 (accessed on 12.10.2022).

17. Sakr S., Zomaya A.Y. Encyclopedia of big data technologies. Springer International Publishing; 2019. DOI: 10.1007/978-3-319-77525-8.

18. Bakshi K. Technologies for Big Data. IGI Global; 2014:1–22. DOI:10.4018/978-1-4666-4699-5.ch001.

19. Singh S., Singh P., Garg R., Mishra P.K. Big Data: Technologies, Trends and Applications. International Journal of Computer Science and Information Technologies. 2015;6(5):4633–4639.

20. Mamedova G.A., Zeynalova L.A., Melikova R.T. Big data technologies in e-learning. Open Education. 2018:16;0(6):41–8. DOI: 10.21686/1818-4243-2017-6-41-48.

21. Jiang D., Ooi B.C., Shi L., Wu S. The performance of MapReduce: an in-depth study. Proceeding Proc VLDB Endow. 2010;3(1–2):472–483. DOI:10.14778/1920841.1920903.

22. Shim K. MapReduce Algorithms for Big Data Analysis. In: Madaan A, Kikuchi S, Bhalla S, editors. Lecture Notes in Computer Science. Databases in Networked Information Systems. Berlin, Heidelberg: Springer; 2013: 44–8. DOI:10.1007/978-3-642-37134-9_3.

23. Tao Y., Lin W., Xiao X. Minimal MapReduce algorithms. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. New York, NY, USA: Association for Computing Machinery; 2013:529–40. DOI: 10.1145/2463676.2463719.

24. Győrödi C.A., Dumşe-Burescu D.V., Zmaranda D.R., Győrödi R.Ş., Gabor G.A., Pecherle G.D. Performance Analysis of NoSQL and Relational Databases with CouchDB and MySQL for Application’s Data Storage. Applied Sciences. 2020;10(23):8524. DOI: 10.3390/app10238524.

25. Anderson J.C., Lehnardt J., Slater N. CouchDB: The Definitive Guide: Time to Relax. O’Reilly Media, Inc.; 2010:274.

26. Lennon J. Beginning CouchDB. Apress; 2010:299.

27. Gusarenko A.S. Certificate of state registration of the computer program No. 2022617552. Microservice of a situationally oriented database for loading data and Map-Reduce functions into a document-oriented storage. 2022. (In Russ.).

28. Gusarenko A.S. Certificate of state registration of the computer program No. 2022617505. Situational database modules for extracting large documents and archives from RESTful services of heterogeneous data stores. 2022. (In Russ.).

29. Apache CouchDB. Available from: https://couchdb.apache.org/ (accessed on 17.10.2022).

30. Mironov V.V., Gusarenko A.S., Yusupova N.I. Situation-Oriented Databases: Processing Office Documents. Modelirovaniye, optimizatsiya i informatsionnyye tekhnologii = Modeling, optimization and information technology. 2022:28;10(2):1–16. DOI:10.26102/2310-6018/2022.37.2.021.

31. Mironov V.V., Gusarenko A.S., Yusupova N.I. The Invariance of The Virtual Data in The Situationally Oriented Database When Displayed on Heterogeneous Data Storages. Vestnik komp'yuternykh i informatsionnykh tekhnologiy = Herald of Computer and Information Technologies. 2017;(1(151)):29–36. DOI: 10.14489/VKIT.2017.01.PP.029-036 (In Russ.).

32. Course project "Databases". Available from: http://hsm.ugatu.su/artem/dbproj/ (accessed on 20.10.2022). (In Russ.).