Development of a software package for database management in a search service using organizational systems

The article discusses the implementation of a database management system, which is interesting because it allows for fast searches of static and unchangeable data, including large volumes of this data. To obtain the results, programs for processing and unifying files, combining and indexing them, as well as for searching by indexed data were developed from scratch. The methods of parallelization, binary search, interpolation search, mmap-mapping, clustering, caching, direct and reverse indexing, merging, LZ archiving and B-trees were used. A search engine was created that allows performing thousands of search queries per second and working with databases of several terabytes in size. The relevance of the study is due to the need to perform a large number of search operations on large arrays of data. In this regard, this article is aimed at disclosing and implementing the most effective mechanisms for such a search. The leading approach to the study of this problem is the practical implementation of various search algorithms and their further optimization to obtain the fastest search methods. Ready-made algorithms for data processing and further methods of searching on them are presented. The materials of the article are of practical value for specialists solving problems related to big data and performing search queries on them. At present, such development for improving databases is necessary due to the constantly increasing flow of digital information that must be correctly collected, processed, analyzed and stored.

1. Shevskiy V.S., Shichkina Yu.A. Technology for executing retrieval queries to a database based on the CW-tree data indexing method. Modeling, Optimization and Information Technology. 2021;9(1). (In Russ.). https://doi.org/10.26102/2310-6018/2021.32.1.014

2. Zobel J., Moffat A. Inverted files for text search engines. ACM Computing Surveys. 2006;38(2). https://doi.org/10.1145/1132956.1132959

3. Golitsyna O.L., Partyka T.L., Popov I.I. Osnovy proektirovaniya baz dannykh. Moscow: Forum; 2012. 415 p. (In Russ.).

4. Mokhov V.A. Binary optimization: problems and algorithms. Bulletin of Higher Educational Institutions. North Caucasus Region. Technical Sciences. 2022;(2):12–19. (In Russ.). https://doi.org/10.17213/1560-3644-2022-2-12-19

5. Demikhov M.A. The methods for fuzzy search in information systems. Modeling, Optimization and Information Technology. 2015;3(2). (In Russ.). URL: https://moit.vivt.ru/wp-content/uploads/2015/06/Demikhov_2_15_2.pdf

6. Hsu J.-Ch., Hsu Ch.-H., Chen S.C., Chung Ye.Ch. Correlation Aware Technique for SQL to NoSQL Transformation. In: 2014 7th International Conference on Ubi-Media Computing and Workshops, 12–14 July 2014, Ulaanbaatar, Mongolia. IEEE; 2014. pp. 43–46. https://doi.org/10.1109/U-MEDIA.2014.27

7. Pinto Y. A Framework for Systematic Database Denormalization. Global Journal of Computer Science and Technology. 2009;9(4):44–52.

8. Mehmood A. ASH Search: Binary Search Optimization. International Journal of Computer Applications. 2019;178(15):10–17. https://doi.org/10.5120/ijca2019918788

9. Demikhov M.A. The characteristics of search algorithms in modern search engines. Modeling, Optimization and Information Technology. 2015;3(2). (In Russ.). URL: https://moit.vivt.ru/wp-content/uploads/2015/06/Demikhov_2_15_1.pdf

10. Sedgewick R., Wayne K. Algorithms: Part I, 4th Edition. New Jersey: Pearson Education, Inc.; 2014. 932 p.

11. Novikov B.A., Gorshkova E.A., Grafeeva N.G. Osnovy tekhnologii baz dannykh. Moscow: DMK Press; 2020. 582 p. (In Russ.).