Game-theoretic models of resource coordination in distributed streaming data analysis systems

Modern distributed streaming data analysis systems such as Apache Spark face the fundamental problem of resource coordination in the context of the strategic behavior of computing nodes. Traditional scheduling algorithms (FIFO, Fair Scheduler) do not take into account that each executor strives to maximize its own local performance, which leads to systemic problems: "tragedies of shared resources", load imbalance due to data skew and an overall decrease in cluster efficiency. The article suggests an approach to solving this problem based on game-theoretic modeling. The research systematizes and adapts cooperative and non-cooperative game theory models for resource management tasks in the Apache Spark environment. As part of the cooperative approach, the Shapley Value algorithm has been formalized and adapted in detail, making it possible to quantify the contribution of each computing node to the overall performance of the system and ensure a fair distribution of computing resources among the coalition participants. To manage competition, an auction mechanism based on the Vickery principle (second price) has been developed, which encourages nodes to honestly state their needs. The practical significance of the work is confirmed by the development and implementation of a modular optimization subsystem integrated with the Prometheus/Grafana monitoring stack. Experimental results based on synthetic data demonstrate that the proposed approach reduces the average task execution time and improves load balancing compared to standard schedulers. The work contributes to the creation of self-optimizing distributed systems capable of operating effectively in conditions of competition for resources.

1. Zaharia M., Chowdhury M., Das T., et al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In: NSDI'12: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, 25–27 April 2012, San Jose, CA, USA. Berkeley: USENIX Association; 2012. URL: https://dl.acm.org/doi/10.5555/2228298.2228301

2. Nash J.F. Equilibrium Points in n-Person Games. Proceedings of the National Academy of Sciences. 1950;36(1):48–49. https://doi.org/10.1073/pnas.36.1.48

3. Shapley L.S. A Value for n-Person Games. In: Contributions to the Theory of Games II. Princeton: Princeton University Press; 1953. P. 307–317.

4. Leyton-Brown K., Shoham Y. Essentials of Game Theory. A Concise, Multidisciplinary Introduction. Morgan & Claypool Publishers; 2008. 88 p.

5. Marz N., Warren J. Big Data. Principles and Best Practices of Scalable Real-Time Data Systems. Moscow: Vil'yams; 2016. 368 p. (In Russ.).

6. Perrin J.-G. Spark in Action. Covers Apache Spark 3 with Examples in Java, Python, and Scala. Moscow: DMK Press; 2021. 636 p. (In Russ.).

7. Qiu H., Zhu K., Luong N.C., Yi Ch., Niyato D., Kim D.I. Applications of Auction and Mechanism Design in Edge Computing: A Survey. arXiv. URL: https://arxiv.org/abs/2105.03559 [Accessed 15th October 2025].

8. Taulli T. Artificial Intelligence Basics. A non-Technical Introduction. Saint Petersburg: BKhV-Peterburg; 2021. 288 p. (In Russ.).

9. Vickrey W. Counterspeculation, Auctions, and Competitive Sealed Tenders. Journal of Finance. 1961;16(1):8–37. https://doi.org/10.1111/J.1540-6261.1961.TB02789.X

10. Cardellini V., De Nitto Personé V., Di Valerio V., et al. A Game-Theoretic Approach to Computation Offloading in Mobile Cloud Computing. Mathematical Programming. 2016;157(2):421–449. https://doi.org/10.1007/s10107-015-0881-6