moitvivt Моделирование, оптимизация и информационные технологии Modeling, Optimization and Information Technology 2310-6018 Издательство 10.26102/2310-6018/2026.61.10.011 2555 Методы обучения с подкреплением для автоматической настройки и онлайн-адаптации регуляторов трудно настраиваемых динамических объектов: аналитический обзор Reinforcement learning methods for automatic tuning and online adaptation of controllers for hard-to-tune dynamic plants: an analytical review 0000-0002-5259-6841 Воробей Сергей Сергеевич Vorobey Sergey Sergeevich sv@rootml.com aff-1 Московский технический университет связи и информатики ООО "КуРэйт" Moscow Technical University of Communications and Informatics LLC QRate 01 01 2026 1 1 10.26102/2310-6018/2026.61.10.011 Copyright © Авторы, 2026 2026 This work is licensed under a Creative Commons Attribution 4.0 International License

Рассмотрена задача автоматической настройки и подстройки в процессе работы регуляторов сложных динамических объектов, для которых однократно рассчитанный набор параметров не сохраняет требуемого качества управления на всем сроке эксплуатации. Цель обзора состоит в том, чтобы установить границы применимости существующих методов настройки по единому критерию и обосновать направления дальнейших исследований по нерешенной научной проблеме. Поиск велся по реферативным базам и охватывал классические первоисточники и работы прикладного периода; отбирались рецензируемые источники, относящиеся к настройке регуляторов. Прослежено развитие подходов от эмпирических правил настройки пропорционально-интегрально-дифференциальных регуляторов и управления с активным подавлением возмущений до обучаемых правил настройки, построенных методами обучения с подкреплением. Для каждого семейства методов выделены достоинства и пределы применимости: классические правила рассчитывают фиксированные коэффициенты по одному эксперименту и теряют качество при дрейфе параметров объекта; обучаемые методы работают с непрерывными параметрами, но не гарантируют устойчивости замкнутого контура в процессе обучения и требуют большого числа взаимодействий с объектом. Разрозненные подходы сведены к единому критерию пригодности, а именно работе с непрерывным вектором параметров и выборочной эффективности в реальном времени, и систематизированы в четыре нерешенные проблемы: сохранение устойчивости при обучении, выборочную эффективность, перенос с модели на объект и дефицит подтверждения на натурных стендах. Каждая проблема сопоставлена с направлением дальнейших исследований. Полученное сопоставление показывает, что заявленная проблема остается открытой для всего класса трудно настраиваемых объектов и требует одновременного решения вопросов устойчивости, выборочной эффективности и натурной валидации.

This review addresses the problem of automatic tuning and online adaptation of controllers for complex dynamic plants for which a single computed parameter set does not preserve the required control quality over the whole service interval. The aim is to establish, under a single criterion, the applicability limits of the existing tuning methods and to substantiate directions for further research on an open scientific problem. The search covered classical primary sources and applied period works selected from abstract databases. The development of approaches is traced from empirical tuning rules for proportional-integral-derivative controllers and active disturbance rejection control to learned tuning rules built by reinforcement learning. For each family of methods the strengths and the applicability limits are identified: classical rules compute fixed coefficients from a single experiment and lose quality under plant parameter drift, whereas learned methods operate with continuous parameters but do not guarantee closed loop stability during learning and require a large number of interactions with the plant. The scattered approaches are reduced to a single suitability criterion, namely operation with a continuous parameter vector and sample efficiency in real time, and are systematised into four open problems: preserving stability during learning, sample efficiency, transfer from a model to the plant, and the shortage of validation on physical test benches. Each problem is matched to a direction for further research. The mapping shows that the stated problem remains open for the whole class of hard to tune plants and requires the simultaneous resolution of stability, sample efficiency, and physical validation.

 обучение с подкреплением автоматическая настройка регуляторов онлайн-адаптация устойчивость замкнутого контура выборочная эффективность популяционное обучение перенос с модели на объект натурная валидация reinforcement learning automatic controller tuning online adaptation closed loop stability sample efficiency population-based training sim to real transfer physical validation Исследование выполнено без спонсорской поддержки. The study was performed without external funding. References Minorsky N. Directional stability of automatically steered bodies. Journal of the American Society for Naval Engineers. 1922;34(2):280–309. https://doi.org/10.1111/j.1559-3584.1922.tb04958.x Ziegler J.G., Nichols N.B. Optimum settings for automatic controllers. Transactions of the ASME. 1942;64(8):759–765. https://doi.org/10.1115/1.4019264 Desborough L., Miller R. Increasing customer value of industrial control performance monitoring – Honeywell's experience. In: Chemical Process Control-VI: Assessment and New Directions for Research: Proceedings of the Sixth International Conference on Chemical Process Control, 07–12 January 2001, Tucson, AZ, USA. CACHE, AIChE; 2002. P. 169–189. Bauer M., Horch A., Xie L., Jelali M., Thornhill N. The current state of control loop performance monitoring – A survey of application in industry. Journal of Process Control. 2016;38:1–10. https://doi.org/10.1016/j.jprocont.2015.11.002 Åström K.J., Hägglund T. Advanced PID Control. ISA – Instrumentation, Systems, and Automation Society; 2006. 460 p. Gaing Z.-L. A particle swarm optimization approach for optimum design of PID controller in AVR system. IEEE Transactions on Energy Conversion. 2004;19(2):384–391. https://doi.org/10.1109/TEC.2003.821821 Fleming P.J., Purshouse R.C. Evolutionary algorithms in control systems engineering: A survey. Control Engineering Practice. 2002;10(11):1223–1241. https://doi.org/10.1016/S0967-0661(02)00081-3 Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction. Cambridge: MIT Press; 2018. 552 p. Dorf R.C., Bishop R.H. Modern Control Systems. Upper Saddle River: Prentice Hall; 2001. 831 p. Cohen G.H., Coon G.A. Theoretical consideration of retarded control. Transactions of the ASME. 1953;75:827–834. https://doi.org/10.1115/1.4015451 Åström K.J., Hägglund T. Automatic tuning of simple regulators with specifications on phase and amplitude margins. Automatica. 1984;20(5):645–651. https://doi.org/10.1016/0005-1098(84)90014-1 Åström K.J., Wittenmark B. Adaptive Control. Reading: Addison-Wesley; 1995. 574 p. Ioannou P., Fidan B. Adaptive Control Tutorial. Philadelphia: SIAM; 2006. 405 p. https://doi.org/10.1137/1.9780898718652 Han J. From PID to active disturbance rejection control. IEEE Transactions on Industrial Electronics. 2009;56(3):900–906. https://doi.org/10.1109/TIE.2008.2011621 Gao Zh. Scaling and bandwidth-parameterization based controller tuning. In: 2003 American Control Conference, 04–06 June 2003, Denver, CO, USA. IEEE; 2003. P. 4989–4996. https://doi.org/10.1109/ACC.2003.1242516 Gao Zh. Active disturbance rejection control: a paradigm shift in feedback control system design. In: 2006 American Control Conference, 14–16 June 2006, Minneapolis, MN, USA. IEEE; 2006. https://doi.org/10.1109/ACC.2006.1656579 Huang Y., Xue W. Active disturbance rejection control: methodology and theoretical analysis. ISA Transactions. 2014;53(4):963–976. https://doi.org/10.1016/j.isatra.2014.03.003 Khalil H.K. High-Gain Observers in Nonlinear Feedback Control. Philadelphia: SIAM; 2017. 323 p. https://doi.org/10.1137/1.9781611974867 Fu C., Tan W. Tuning of linear ADRC with known plant information. ISA Transactions. 2016;65:384–393. https://doi.org/10.1016/j.isatra.2016.06.016 Sutton R.H., Precup D., Singh S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence. 1999;112(1–2):181–211. https://doi.org/10.1016/S0004-3702(99)00052-1 Bradtke S., Duff M.O. Reinforcement learning methods for continuous-time Markov decision problems. In: Advances in Neural Information Processing Systems 7: Annual Conference on Neural Information Processing Systems, 28 November – 01 December 1994, Denver, CO, USA. Cambridge: MIT Press; 1995. P. 393–400. Watkins Ch.J.C.H., Dayan P. Q-learning. Machine Learning. 1992;8(3–4):279–292. https://doi.org/10.1007/BF00992698 Mnih V., Kavukcuoglu K., Silver D., et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–533. https://doi.org/10.1038/nature14236 Shi Q., Lam H.-K., Xiao B., Tsai Sh.-H. Adaptive PID controller based on Q-learning algorithm. CAAI Transactions on Intelligence Technology. 2018;3(4):235–244. https://doi.org/10.1049/trit.2018.1007 Wang F., Ren B., Liu Y., Cui B. Tracking moving target for 6 degree-of-freedom robot manipulator with adaptive visual servoing based on deep reinforcement learning PID controller. Review of Scientific Instruments. 2022;93(4):045108. https://doi.org/10.1063/5.0087561 Qin Y., Zhang W., Shi J., Liu J. Improve PID controller through reinforcement learning. In: 2018 IEEE CSAA Guidance, Navigation and Control Conference, 10–12 August 2018, Xiamen, China. IEEE; 2018. https://doi.org/10.1109/GNCC42960.2018.9019095 Sutton R.S., McAllester D.A., Singh S., Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems 12: Annual Conference on Neural Information Processing Systems, 29 November – 04 December 1999, Denver, CO, USA. Cambridge: MIT Press; 2000. P. 1057–1063. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 1992;8(3):229–256. https://doi.org/10.1007/BF00992696 Schulman J., Levine S., Abbeel P., Jordan M.I., Moritz Ph. Trust region policy optimization. In: Proceedings of the 32nd International Conference on Machine Learning, 06–11 July 2015, Lille, France. JMLR; 2015. P. 1889–1897. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. Proximal policy optimization algorithms. arXiv. URL: https://arxiv.org/abs/1707.06347 [Accessed 13th July 2026]. Lillicrap T.P., Hunt J.J., Pritzel A., et al. Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations, 02–04 May 2016, San Juan, Puerto Rico. 2016. URL: https://arxiv.org/abs/1509.02971 Lakhani A.I., Chowdhury M.A., Lu Q. Stability-preserving automatic tuning of PID control with reinforcement learning. Complex Engineering Systems. 2022;2(1):3. https://doi.org/10.20517/ces.2021.15 Arshad J., Khan A., Aftab M., et al. Deep deterministic policy gradient to regulate feedback control systems using reinforcement learning. Computers, Materials & Continua. 2022;71(1):1153–1169. https://doi.org/10.32604/cmc.2022.021917 Lee D., Lee S.J., Yim S.C. Reinforcement learning-based adaptive PID controller for DPS. Ocean Engineering. 2020;216(11). https://doi.org/10.1016/j.oceaneng.2020.108053 Lawrence N.P., Forbes M.G., Loewen Ph.D., McClement D.G., Backström J.U., Gopaluni R.Bh. Deep reinforcement learning with shallow controllers: An experimental application to PID tuning. Control Engineering Practice. 2022;121(5):105046. https://doi.org/10.1016/j.conengprac.2021.105046 Wang H., Ricardez-Sandoval L.A. A deep reinforcement learning-based PID tuning strategy for nonlinear MIMO systems with time-varying uncertainty. IFAC-PapersOnLine. 2024;58(14):887–892. https://doi.org/10.1016/j.ifacol.2024.08.449 Veerasamy G., Balaji S., Kadirvelu Th., Ramasamy V. Reinforcement Learning Based Adaptive PID Controller for a Continuous Stirred Tank Heater Process. Iranian Journal of Chemistry and Chemical Engineering. 2025;44(1):265–282. https://doi.org/10.30492/ijcce.2024.2029225.6615 Fujimoto S., van Hoof H., Meger D. Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning, 10–15 July 2018, Stockholm, Sweden. PMLR; 2018. P. 1582–1591. Haarnoja T., Zhou A., Abbeel P., Levine S. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, 10–15 July 2018, Stockholm, Sweden. PMLR; 2018. P. 1856–1865. Wang Y., Fang Sh., Hu J., Huang D. A novel active disturbance rejection control of PMSM based on deep reinforcement learning for more electric aircraft. IEEE Transactions on Energy Conversion. 2023;38(2):1461–1470. https://doi.org/10.1109/TEC.2023.3235927 Deisenroth M.P., Rasmussen C.E. PILCO: A model-based and data-efficient approach to policy search. In: Proceedings of the 28th International Conference on Machine Learning, 28 June – 02 July 2011, Bellevue, WA, USA. Madison: Omnipress; 2011. P. 465–472. Doerr A., Nguyen-Tuong D., Marco A., Schaal S., Trimpe S. Model-based policy search for automatic tuning of multivariate PID controllers. In: 2017 IEEE International Conference on Robotics and Automation, 29 May – 03 June 2017, Singapore. IEEE; 2017. P. 5295–5301. https://doi.org/10.1109/ICRA.2017.7989622 Chua K., Calandra R., McAllister R., Levine S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, 03–08 December 2018, Montréal, Canada. 2018. P. 4759–4770. Altman E. Constrained Markov Decision Processes. Boca Raton: CRC Press; 1999. 256 p. Stooke A., Achiam J., Abbeel P. Responsive safety in reinforcement learning by PID Lagrangian methods. In: Proceedings of the 37th International Conference on Machine Learning, 13–18 July 2020, Virtual Event. PMLR; 2020. P. 9133–9143. Achiam J., Held D., Tamar A., Abbeel P. Constrained policy optimization. In: Proceedings of the 34th International Conference on Machine Learning, 06–11 August 2017, Sydney, NSW, Australia. PMLR; 2017. P. 22–31. Berkenkamp F., Turchetta M., Schoellig A.P., Krause A. Safe model-based reinforcement learning with stability guarantees. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 04–09 December 2017, Long Beach, CA, USA. 2017. P. 908–918. Zhang M., Zhang H., Ji J., Yang Y., Sun Y. Enhance the safety in reinforcement learning by ADRC Lagrangian methods. arXiv. URL: https://arxiv.org/abs/2601.18142 [Accessed 13th July 2026]. Bergstra J., Bengio Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research. 2012;13(2):281–305. Jaderberg M., Dalibard V., Osindero S., et al. Population based training of neural networks. arXiv. URL: https://arxiv.org/abs/1711.09846 [Accessed 13th July 2026]. Dalibard V., Jaderberg M. Faster Improvement Rate Population Based Training. arXiv. URL: https://arxiv.org/abs/2109.13800 [Accessed 13th July 2026]. Tobin J., Fong R., Ray A., Schneider J., Zaremba W., Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 24–28 September 2017, Vancouver, Canada. IEEE; 2017. P. 23–30. https://doi.org/10.1109/IROS.2017.8202133 Zhao W., Queralta J.P., Westerlund T. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In: 2020 IEEE Symposium Series on Computational Intelligence, 01–04 December 2020, Canberra, ACT, Australia. IEEE; 2020. P. 737–744. https://doi.org/10.1109/SSCI47803.2020.9308468 Reutov A., Vorobey S., Katanskiy A., et al. Optical stabilization for laser communication satellite systems through proportional-integral-derivative (PID) control and reinforcement learning approach. Review of Scientific Instruments. 2025;96(3):035121. https://doi.org/10.1063/5.0239029 Henderson P., Islam R., Bachman Ph., Pineau J., Precup D., Meger D. Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence. 2018;32(1). https://doi.org/10.1609/aaai.v32i1.11694

The authors declare that there are no conflicts of interest present.