Experimental study of cooperative resource allocation in big data processing systems based on Shapley value and machine learning

Optimal and equitable allocation of computing resources in dynamic Big Data environments such as Apache Spark remains a challenge. Traditional planners often do not take into account the synergetic effects of cooperation between tasks, which leads to inefficiency and conflicts. The purpose of this work is to experimentally investigate and verify a hybrid approach to cooperative resource allocation based on the formal principles of cooperative game theory and adaptive machine learning capabilities. The paper formalizes a model of a cooperative game, where coalitions of parallel tasks are characterized by a utility function depending on the allocated resources. To ensure stability and fairness, equilibrium conditions (the core of the game) have been introduced, and the distribution is based on the Shapley value, which estimates the marginal contribution of each task. To overcome the analytical complexity of evaluating utility in real conditions, it is proposed to use ML models (gradient boosting, graph neural networks) trained on historical cluster metrics as approximators of the characteristic function of the game. An experimental bench based on Apache Spark with the Prometheus/Grafana monitoring system has been developed and deployed. Experiments have shown that the proposed approach provides a dynamic and balanced allocation of resources (CPU, memory), increases the stability of task coalitions, and improves overall distribution equity (Ginny index) compared to the baseline scenarios. Visualization of key metrics confirmed the achievement of states close to the core of the game. The study demonstrates the practical applicability and effectiveness of combining game-theoretic models and machine learning for intelligent resource management in distributed Big Data systems, paving the way for the creation of self-optimizing and cooperative orchestrators.

1. Blyakherov M.V., Petrova E.S. Game-theoretic models of resource coordination in distributed streaming data analysis systems. Modeling, Optimization and Information Technology. 2025;13(4). (In Russ.). https://doi.org/10.26102/2310-6018/2025.51.4.068

2. Kim S. Cooperative Game-Based Virtual Machine Resource Allocation Algorithms in Cloud Data Centers. Mobile Information Systems. 2020;2020. https://doi.org/10.1155/2020/9840198

3. Bertossi L., Kimelfeld B., Livshits E., Monet M. The Shapley Value in Database Management. arXiv. URL: https://arxiv.org/abs/2401.06234 [Accessed 26th December 2025].

4. Shapley L.S. A Value for n-Person Games. In: Contributions to the Theory of Games: Volume II. Princeton: Princeton University Press; 1953. P. 307–317. https://doi.org/10.1515/9781400881970-018

5. Colini-Baldeschi R., Scarsini M., Vaccari S. Variance Allocation and Shapley Value. arXiv. URL: https://arxiv.org/abs/1606.09424 [Accessed 26th December 2025].

6. Zhou J., Wen G., Lv Y., Yang T., Chen G. DRAG: Distributed Resource Allocation Games over Multiple Interacting Coalitions. arXiv. URL: https://arxiv.org/abs/2210.02919 [Accessed 26th December 2025].

7. Cardellini V., De Nitto Personé V., Di Valerio V., et al. A Game-Theoretic Approach to Computation Offloading in Mobile Cloud Computing. Mathematical Programming. 2016;157(2):421–449. https://doi.org/10.1007/s10107-015-0881-6

8. Razumovskii D.A., Volkov D.D., Stuchilin V.V. Architecture of a system for collecting and storing metrics on the resource usage of Spark applications in clustered big data processing systems. International Research Journal. 2025;(12). (In Russ.). https://doi.org/10.60797/IRJ.2025.162.81

9. Elradi M.D. Prometheus and Grafana: A Metrics-focused Monitoring Stack. Journal of Computer Allied Intelligence. 2025;3(3):28–39. https://doi.org/10.69996/jcai.2025015

10. Mehdi A., Bali M.K., Abbas S.I., Singh M. Unleashing the Potential of Grafana: A Comprehensive Study on Real-Time Monitoring and Visualization. In: 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), 06–08 July 2023, Delhi, India. IEEE; 2023. https://doi.org/10.1109/ICCCNT56998.2023.10306699