An incremental method for updating a multidimensional cube based on a disordered stream of information system log events

Information systems generate large volumes of event logs, which are used to analyze the operation of applications and services. In this case, events may arrive in the analytical circuit later than the moment of their actual occurrence and not in the original order. Such time inconsistency leads to errors in the construction of aggregated time-based indicators, while regular full recalculation of a multidimensional analytical cube requires significant computational costs. The purpose of the work is to develop an approach to updating a multidimensional cube based on the flow of log events, taking into account delays and violations of the order in which they arrive. The leading approach is to divide the final cube representation into a base cube and a compensation layer. Timely events update the base cube, while delayed events within a given compensation horizon add corrections to the compensation layer. Unique event identifiers are checked to exclude repeated event accounting. The paper presents a formal event model, describes rules for assigning events to cube cells by the occurrence time, proposes an incremental update algorithm, and provides a computational experiment on a software prototype. The experimental results show that the proposed method reduces aggregate errors compared with the window-based method and requires a smaller amount of repeated data processing compared with batch recalculation. The materials of the paper are of practical value for developing analytical pipelines for monitoring, audit, security event analysis, and user activity analysis.

1. Frolov V.A., Khayrullin R.Z., Afanasyev G.I. Analysis of storage formats for multidimensional data models in the context of multidimensional cubes. Computational nanotechnology. 2025;12(8):187–194. (In Russ.). https://doi.org/10.33693/2313-223X-2025-12-4-187-194

2. Neupokoeva E.V., Bystrov V.V. Analysis of OLAP solutions for studying resilienсe of regional socio-economic systems. Transactions of the Kola Science Centre of RAS. Series: Engineering Sciences. 2023;14(7):52–67. (In Russ.). https://doi.org/10.37614/2949-1215.2023.14.7.006

3. Akidau T., Begoli E., Chernyak S., et al. Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow. Proceedings of the VLDB Endowment. 2021;14(12):3135–3147. https://doi.org/10.14778/3476311.3476389

4. Fragkoulis M., Carbone P., Kalavri V., et al. A survey on the evolution of stream processing systems. The VLDB Journal. 2024;33:507–541. https://doi.org/10.1007/s00778-023-00819-8

5. Svingos C., Hernich A., Gildhoff H., et al. Foreign Keys Open the Door for Faster Incremental View Maintenance. Proceedings of the ACM on Management of Data. 2023;1(1):1–25. https://doi.org/10.1145/3588720

6. Cuzzocrea A., Moussa R., Vercelli G. An Innovative Lambda-Architecture-Based Data Warehouse Maintenance Framework for Effective and Efficient Near-Real-Time OLAP over Big Data. In: Big Data – BigData 2018: 7th International Congress, Held as Part of the Services Conference Federation, SCF 2018, June 25–30, 2018, Seattle, WA, USA. Cham: Springer; 2018. P. 149–165. https://doi.org/10.1007/978-3-319-94301-5_12

7. Tahir J., Mayer R., Doblander C., et al. How Reliable are Streams? End-to-End Processing-Guarantee Validation and Performance Benchmarking of Stream Processing Systems. Proceedings of the VLDB Endowment. 2024;18(3):585–598. https://doi.org/10.14778/3712221.3712227

8. Stupnikov S.A., Skvortsov N.A., Briukhov D.O. Advanced methods for implementation of incremental view maintenance in modern relational database management systems. Systems and Means of Informatics. 2025;35(1):95–110. (In Russ.). https://doi.org/10.14357/08696527250105

9. Loginovskiy O.V., Shinkarev A.A., Koval M.E. Development of architecture of message queue based information search systems for enterprise information systems. Applied Mathematics and Control Sciences. 2021;(1):119–140. (In Russ.). URL: https://doi.org/10.15593/2499-9873/2021.01.07

10. Tang B., Han S., Yiu M.L., et al. Extracting Top-K Insights from Multi-dimensional Data. In: Proceedings of the 2017 ACM International Conference on Management of Data: SIGMOD ’17, May 14–19, 2017, Chicago, IL, USA. New York: Association for Computing Machinery; 2017. P. 1509–1524. https://doi.org/10.1145/3035918.3035922

11. Dehne F., Kong Q., Rau-Chaplin A., et al. Scalable real-time OLAP on cloud architectures. Journal of Parallel and Distributed Computing. 2015;79–80:31–41. https://doi.org/10.1016/j.jpdc.2014.08.006

12. Samarev R.S. Survey of streaming processing field. Proceedings of the Institute for System Programming of the RAS. 2017;29(1):231–260. (In Russ.).