Методы обучения с подкреплением для автоматической настройки и онлайн-адаптации регуляторов трудно настраиваемых динамических объектов: аналитический обзор

Рассмотрена задача автоматической настройки и подстройки в процессе работы регуляторов сложных динамических объектов, для которых однократно рассчитанный набор параметров не сохраняет требуемого качества управления на всем сроке эксплуатации. Цель обзора состоит в том, чтобы установить границы применимости существующих методов настройки по единому критерию и обосновать направления дальнейших исследований по нерешенной научной проблеме. Поиск велся по реферативным базам и охватывал классические первоисточники и работы прикладного периода; отбирались рецензируемые источники, относящиеся к настройке регуляторов. Прослежено развитие подходов от эмпирических правил настройки пропорционально-интегрально-дифференциальных регуляторов и управления с активным подавлением возмущений до обучаемых правил настройки, построенных методами обучения с подкреплением. Для каждого семейства методов выделены достоинства и пределы применимости: классические правила рассчитывают фиксированные коэффициенты по одному эксперименту и теряют качество при дрейфе параметров объекта; обучаемые методы работают с непрерывными параметрами, но не гарантируют устойчивости замкнутого контура в процессе обучения и требуют большого числа взаимодействий с объектом. Разрозненные подходы сведены к единому критерию пригодности, а именно работе с непрерывным вектором параметров и выборочной эффективности в реальном времени, и систематизированы в четыре нерешенные проблемы: сохранение устойчивости при обучении, выборочную эффективность, перенос с модели на объект и дефицит подтверждения на натурных стендах. Каждая проблема сопоставлена с направлением дальнейших исследований. Полученное сопоставление показывает, что заявленная проблема остается открытой для всего класса трудно настраиваемых объектов и требует одновременного решения вопросов устойчивости, выборочной эффективности и натурной валидации.

1. Minorsky N. Directional stability of automatically steered bodies. Journal of the American Society for Naval Engineers. 1922;34(2):280–309. https://doi.org/10.1111/j.1559-3584.1922.tb04958.x

2. Ziegler J.G., Nichols N.B. Optimum settings for automatic controllers. Transactions of the ASME. 1942;64(8):759–765. https://doi.org/10.1115/1.4019264

3. Desborough L., Miller R. Increasing customer value of industrial control performance monitoring – Honeywell's experience. In: Chemical Process Control-VI: Assessment and New Directions for Research: Proceedings of the Sixth International Conference on Chemical Process Control, 07–12 January 2001, Tucson, AZ, USA. CACHE, AIChE; 2002. P. 169–189.

4. Bauer M., Horch A., Xie L., Jelali M., Thornhill N. The current state of control loop performance monitoring – A survey of application in industry. Journal of Process Control. 2016;38:1–10. https://doi.org/10.1016/j.jprocont.2015.11.002

5. Åström K.J., Hägglund T. Advanced PID Control. ISA – Instrumentation, Systems, and Automation Society; 2006. 460 p.

6. Gaing Z.-L. A particle swarm optimization approach for optimum design of PID controller in AVR system. IEEE Transactions on Energy Conversion. 2004;19(2):384–391. https://doi.org/10.1109/TEC.2003.821821

7. Fleming P.J., Purshouse R.C. Evolutionary algorithms in control systems engineering: A survey. Control Engineering Practice. 2002;10(11):1223–1241. https://doi.org/10.1016/S0967-0661(02)00081-3

8. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction. Cambridge: MIT Press; 2018. 552 p.

9. Dorf R.C., Bishop R.H. Modern Control Systems. Upper Saddle River: Prentice Hall; 2001. 831 p.

10. Cohen G.H., Coon G.A. Theoretical consideration of retarded control. Transactions of the ASME. 1953;75:827–834. https://doi.org/10.1115/1.4015451

11. Åström K.J., Hägglund T. Automatic tuning of simple regulators with specifications on phase and amplitude margins. Automatica. 1984;20(5):645–651. https://doi.org/10.1016/0005-1098(84)90014-1

12. Åström K.J., Wittenmark B. Adaptive Control. Reading: Addison-Wesley; 1995. 574 p.

13. Ioannou P., Fidan B. Adaptive Control Tutorial. Philadelphia: SIAM; 2006. 405 p. https://doi.org/10.1137/1.9780898718652

14. Han J. From PID to active disturbance rejection control. IEEE Transactions on Industrial Electronics. 2009;56(3):900–906. https://doi.org/10.1109/TIE.2008.2011621

15. Gao Zh. Scaling and bandwidth-parameterization based controller tuning. In: 2003 American Control Conference, 04–06 June 2003, Denver, CO, USA. IEEE; 2003. P. 4989–4996. https://doi.org/10.1109/ACC.2003.1242516

16. Gao Zh. Active disturbance rejection control: a paradigm shift in feedback control system design. In: 2006 American Control Conference, 14–16 June 2006, Minneapolis, MN, USA. IEEE; 2006. https://doi.org/10.1109/ACC.2006.1656579

17. Huang Y., Xue W. Active disturbance rejection control: methodology and theoretical analysis. ISA Transactions. 2014;53(4):963–976. https://doi.org/10.1016/j.isatra.2014.03.003

18. Khalil H.K. High-Gain Observers in Nonlinear Feedback Control. Philadelphia: SIAM; 2017. 323 p. https://doi.org/10.1137/1.9781611974867

19. Fu C., Tan W. Tuning of linear ADRC with known plant information. ISA Transactions. 2016;65:384–393. https://doi.org/10.1016/j.isatra.2016.06.016

20. Sutton R.H., Precup D., Singh S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence. 1999;112(1–2):181–211. https://doi.org/10.1016/S0004-3702(99)00052-1

21. Bradtke S., Duff M.O. Reinforcement learning methods for continuous-time Markov decision problems. In: Advances in Neural Information Processing Systems 7: Annual Conference on Neural Information Processing Systems, 28 November – 01 December 1994, Denver, CO, USA. Cambridge: MIT Press; 1995. P. 393–400.

22. Watkins Ch.J.C.H., Dayan P. Q-learning. Machine Learning. 1992;8(3–4):279–292. https://doi.org/10.1007/BF00992698

23. Mnih V., Kavukcuoglu K., Silver D., et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–533. https://doi.org/10.1038/nature14236

24. Shi Q., Lam H.-K., Xiao B., Tsai Sh.-H. Adaptive PID controller based on Q-learning algorithm. CAAI Transactions on Intelligence Technology. 2018;3(4):235–244. https://doi.org/10.1049/trit.2018.1007

25. Wang F., Ren B., Liu Y., Cui B. Tracking moving target for 6 degree-of-freedom robot manipulator with adaptive visual servoing based on deep reinforcement learning PID controller. Review of Scientific Instruments. 2022;93(4):045108. https://doi.org/10.1063/5.0087561

26. Qin Y., Zhang W., Shi J., Liu J. Improve PID controller through reinforcement learning. In: 2018 IEEE CSAA Guidance, Navigation and Control Conference, 10–12 August 2018, Xiamen, China. IEEE; 2018. https://doi.org/10.1109/GNCC42960.2018.9019095

27. Sutton R.S., McAllester D.A., Singh S., Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems 12: Annual Conference on Neural Information Processing Systems, 29 November – 04 December 1999, Denver, CO, USA. Cambridge: MIT Press; 2000. P. 1057–1063.

28. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 1992;8(3):229–256. https://doi.org/10.1007/BF00992696

29. Schulman J., Levine S., Abbeel P., Jordan M.I., Moritz Ph. Trust region policy optimization. In: Proceedings of the 32nd International Conference on Machine Learning, 06–11 July 2015, Lille, France. JMLR; 2015. P. 1889–1897.

30. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. Proximal policy optimization algorithms. arXiv. URL: https://arxiv.org/abs/1707.06347 [Accessed 13th July 2026].

31. Lillicrap T.P., Hunt J.J., Pritzel A., et al. Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations, 02–04 May 2016, San Juan, Puerto Rico. 2016. URL: https://arxiv.org/abs/1509.02971

32. Lakhani A.I., Chowdhury M.A., Lu Q. Stability-preserving automatic tuning of PID control with reinforcement learning. Complex Engineering Systems. 2022;2(1):3. https://doi.org/10.20517/ces.2021.15

33. Arshad J., Khan A., Aftab M., et al. Deep deterministic policy gradient to regulate feedback control systems using reinforcement learning. Computers, Materials & Continua. 2022;71(1):1153–1169. https://doi.org/10.32604/cmc.2022.021917

34. Lee D., Lee S.J., Yim S.C. Reinforcement learning-based adaptive PID controller for DPS. Ocean Engineering. 2020;216(11). https://doi.org/10.1016/j.oceaneng.2020.108053

35. Lawrence N.P., Forbes M.G., Loewen Ph.D., McClement D.G., Backström J.U., Gopaluni R.Bh. Deep reinforcement learning with shallow controllers: An experimental application to PID tuning. Control Engineering Practice. 2022;121(5):105046. https://doi.org/10.1016/j.conengprac.2021.105046

36. Wang H., Ricardez-Sandoval L.A. A deep reinforcement learning-based PID tuning strategy for nonlinear MIMO systems with time-varying uncertainty. IFAC-PapersOnLine. 2024;58(14):887–892. https://doi.org/10.1016/j.ifacol.2024.08.449

37. Veerasamy G., Balaji S., Kadirvelu Th., Ramasamy V. Reinforcement Learning Based Adaptive PID Controller for a Continuous Stirred Tank Heater Process. Iranian Journal of Chemistry and Chemical Engineering. 2025;44(1):265–282. https://doi.org/10.30492/ijcce.2024.2029225.6615

38. Fujimoto S., van Hoof H., Meger D. Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning, 10–15 July 2018, Stockholm, Sweden. PMLR; 2018. P. 1582–1591.

39. Haarnoja T., Zhou A., Abbeel P., Levine S. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, 10–15 July 2018, Stockholm, Sweden. PMLR; 2018. P. 1856–1865.

40. Wang Y., Fang Sh., Hu J., Huang D. A novel active disturbance rejection control of PMSM based on deep reinforcement learning for more electric aircraft. IEEE Transactions on Energy Conversion. 2023;38(2):1461–1470. https://doi.org/10.1109/TEC.2023.3235927

41. Deisenroth M.P., Rasmussen C.E. PILCO: A model-based and data-efficient approach to policy search. In: Proceedings of the 28th International Conference on Machine Learning, 28 June – 02 July 2011, Bellevue, WA, USA. Madison: Omnipress; 2011. P. 465–472.

42. Doerr A., Nguyen-Tuong D., Marco A., Schaal S., Trimpe S. Model-based policy search for automatic tuning of multivariate PID controllers. In: 2017 IEEE International Conference on Robotics and Automation, 29 May – 03 June 2017, Singapore. IEEE; 2017. P. 5295–5301. https://doi.org/10.1109/ICRA.2017.7989622

43. Chua K., Calandra R., McAllister R., Levine S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, 03–08 December 2018, Montréal, Canada. 2018. P. 4759–4770.

44. Altman E. Constrained Markov Decision Processes. Boca Raton: CRC Press; 1999. 256 p.

45. Stooke A., Achiam J., Abbeel P. Responsive safety in reinforcement learning by PID Lagrangian methods. In: Proceedings of the 37th International Conference on Machine Learning, 13–18 July 2020, Virtual Event. PMLR; 2020. P. 9133–9143.

46. Achiam J., Held D., Tamar A., Abbeel P. Constrained policy optimization. In: Proceedings of the 34th International Conference on Machine Learning, 06–11 August 2017, Sydney, NSW, Australia. PMLR; 2017. P. 22–31.

47. Berkenkamp F., Turchetta M., Schoellig A.P., Krause A. Safe model-based reinforcement learning with stability guarantees. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 04–09 December 2017, Long Beach, CA, USA. 2017. P. 908–918.

48. Zhang M., Zhang H., Ji J., Yang Y., Sun Y. Enhance the safety in reinforcement learning by ADRC Lagrangian methods. arXiv. URL: https://arxiv.org/abs/2601.18142 [Accessed 13th July 2026].

49. Bergstra J., Bengio Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research. 2012;13(2):281–305.

50. Jaderberg M., Dalibard V., Osindero S., et al. Population based training of neural networks. arXiv. URL: https://arxiv.org/abs/1711.09846 [Accessed 13th July 2026].

51. Dalibard V., Jaderberg M. Faster Improvement Rate Population Based Training. arXiv. URL: https://arxiv.org/abs/2109.13800 [Accessed 13th July 2026].

52. Tobin J., Fong R., Ray A., Schneider J., Zaremba W., Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 24–28 September 2017, Vancouver, Canada. IEEE; 2017. P. 23–30. https://doi.org/10.1109/IROS.2017.8202133

53. Zhao W., Queralta J.P., Westerlund T. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In: 2020 IEEE Symposium Series on Computational Intelligence, 01–04 December 2020, Canberra, ACT, Australia. IEEE; 2020. P. 737–744. https://doi.org/10.1109/SSCI47803.2020.9308468

54. Reutov A., Vorobey S., Katanskiy A., et al. Optical stabilization for laser communication satellite systems through proportional-integral-derivative (PID) control and reinforcement learning approach. Review of Scientific Instruments. 2025;96(3):035121. https://doi.org/10.1063/5.0239029

55. Henderson P., Islam R., Bachman Ph., Pineau J., Precup D., Meger D. Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence. 2018;32(1). https://doi.org/10.1609/aaai.v32i1.11694