Theoretical foundations of monitoring big data changes in large-scale sparse unweighted networks with cloud processing

Networks are widely used to represent the interactive relationships between individual elements in complex big data systems, such as the cloud-based Internet. Determinable causes in these systems can lead to a significant increase or decrease in the frequency of interaction within the corresponding network, making it possible to identify such causes by monitoring the level of interaction within the network. One method for detecting changes is to first create a network graph by drawing an edge between each pair of nodes that have interacted within a specified time interval. The topological characteristics of the graph, such as degree, proximity, and mediation, can then be considered as one-dimensional or multidimensional data for online monitoring. However, the existing statistical process control (SPC) methods for unweighted networks almost do not take into account either the sparsity of the network or the direction of interaction between two network nodes, that is, pair interaction. By excluding inactive pair interactions, the proposed parameter estimation procedure provides higher consistency with lower computational costs than the alternative approach when the networks are large-scale and sparse. The matrices developed on the basis of a matrix probabilistic model for describing directed pair interactions within time-independent, unweighted big data networks with cloud processing significantly simplify parameter estimation, the effectiveness of which is increased by automatically eliminating pair interactions that do not actually occur. Then the proposed model is integrated into a multidimensional distribution function for online monitoring of the level of communication in the network.

1. Paranjape A., Benson A.R., Leskovec J. Motifs in Temporal Networks. In: WSDM '17: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, 06–10 February 2017, Cambridge, United Kingdom. New York: Association for Computing Machinery; 2017. P. 601–610. https://doi.org/10.1145/3018661.3018731

2. Li B., Solea E. A Nonparametric Graphical Model for Functional Data with Application to Brain Networks Based on fMRI. Journal of the American Statistical Association. 2018;113(524):1637–1655. https://doi.org/10.1080/01621459.2017.1356726

3. Yang H., Kumara S., Bukkapatnam S.T.S., Tsung F. The Internet of Things for Smart Manufacturing: A Review. IISE Transactions. 2019;51(11):1190–1216. https://doi.org/10.1080/24725854.2018.1555383

4. Zou N., Li J. Modeling and Change Detection of Dynamic Network Data by a Network State Space Model. IISE Transactions. 2017;49(1):45–57. https://doi.org/10.1080/0740817X.2016.1198065

5. Pandit Sh., Chau D.H., Wang S., Faloutsos Ch. Netprobe: A Fast and Scalable System for Fraud Detection in Online Auction Networks. In: WWW '07: Proceedings of the 16th International Conference on World Wide Web, 08–12 May 2007, Banff, Alberta, Canada. New York: Association for Computing Machinery; 2007. P. 201–210. https://doi.org/10.1145/1242572.1242600

6. McCulloh I., Carley K.M. Detecting Change in Longitudinal Social Networks. Journal of Social Structure. 2011;12(1). https://doi.org/10.21307/joss-2019-031

7. Woodall W.H., Zhao M.J., Paynabar K., Sparks R., Wilson J.D. An Overview and Perspective on Social Network Monitoring. IISE Transactions. 2017;49(3):354–365. https://doi.org/10.1080/0740817X.2016.1213468

8. Hosseini S.S., Noorossana R. Performance Evaluation of EWMA and CUSUM Control Charts to Detect Anomalies in Social Networks Using Average and Standard Deviation of Degree Measures. Quality and Reliability Engineering International. 2018;34(4):477–500. https://doi.org/10.1002/qre.2267

9. Marchette D. Scan Statistics on Graphs. WIREs Computational Statistics. 2012;4(5):466–473. https://doi.org/10.1002/wics.1217

10. Perry M.B. An EWMA Control Chart for Categorical Processes with Applications to Social Network Monitoring. Journal of Quality Technology. 2019;52(2):182–197. https://doi.org/10.1080/00224065.2019.1571343

11. Goldenberg A., Zheng A.X., Fienberg S.E., Airoldi E.M. A Survey of Statistical Network Models. Foundations and Trends® in Machine Learning. 2010;2(2):129–233. https://doi.org/10.1561/2200000005

12. Dong H., Chen N., Wang K. Modeling and Change Detection for Count-Weighted Multilayer Networks. Technometrics. 2020;62(2):184–195. https://doi.org/10.1080/00401706.2019.1625812

13. Azarnoush B., Paynabar K., Bekki J., Runger G. Monitoring Temporal Homogeneity in Attributed Network Streams. Journal of Quality Technology. 2016;48(1):28–43. https://doi.org/10.1080/00224065.2016.11918149

14. Gahrooei M.R., Paynabar K. Change Detection in a Dynamic Stream of Attributed Networks. Journal of Quality Technology. 2018;50(4):418–430. https://doi.org/10.1080/00224065.2018.1507558

15. Holland P.W., Leinhardt S. An Exponential Family of Probability Distributions for Directed Graphs. Journal of the American Statistical Association. 1981;76(373):33–50. https://doi.org/10.1080/01621459.1981.10477598

16. Hetmanskaya D.V. Investigation of Pair Interactions in Monitoring Unweighted Undirected Networks. Sistemy upravleniya i informatsionnye tekhnologii. 2025;(2-1):30–36. (In Russ.).