Reinforcement learning methods for automatic tuning and online adaptation of controllers for hard-to-tune dynamic plants: an analytical review

This review addresses the problem of automatic tuning and online adaptation of controllers for complex dynamic plants for which a single computed parameter set does not preserve the required control quality over the whole service interval. The aim is to establish, under a single criterion, the applicability limits of the existing tuning methods and to substantiate directions for further research on an open scientific problem. The search covered classical primary sources and applied period works selected from abstract databases. The development of approaches is traced from empirical tuning rules for proportional-integral-derivative controllers and active disturbance rejection control to learned tuning rules built by reinforcement learning. For each family of methods the strengths and the applicability limits are identified: classical rules compute fixed coefficients from a single experiment and lose quality under plant parameter drift, whereas learned methods operate with continuous parameters but do not guarantee closed loop stability during learning and require a large number of interactions with the plant. The scattered approaches are reduced to a single suitability criterion, namely operation with a continuous parameter vector and sample efficiency in real time, and are systematised into four open problems: preserving stability during learning, sample efficiency, transfer from a model to the plant, and the shortage of validation on physical test benches. Each problem is matched to a direction for further research. The mapping shows that the stated problem remains open for the whole class of hard to tune plants and requires the simultaneous resolution of stability, sample efficiency, and physical validation.

1. Minorsky N. Directional stability of automatically steered bodies. Journal of the American Society for Naval Engineers. 1922;34(2):280–309. https://doi.org/10.1111/j.1559-3584.1922.tb04958.x

2. Ziegler J.G., Nichols N.B. Optimum settings for automatic controllers. Transactions of the ASME. 1942;64(8):759–765. https://doi.org/10.1115/1.4019264

3. Desborough L., Miller R. Increasing customer value of industrial control performance monitoring – Honeywell's experience. In: Chemical Process Control-VI: Assessment and New Directions for Research: Proceedings of the Sixth International Conference on Chemical Process Control, 07–12 January 2001, Tucson, AZ, USA. CACHE, AIChE; 2002. P. 169–189.

4. Bauer M., Horch A., Xie L., Jelali M., Thornhill N. The current state of control loop performance monitoring – A survey of application in industry. Journal of Process Control. 2016;38:1–10. https://doi.org/10.1016/j.jprocont.2015.11.002

5. Åström K.J., Hägglund T. Advanced PID Control. ISA – Instrumentation, Systems, and Automation Society; 2006. 460 p.

6. Gaing Z.-L. A particle swarm optimization approach for optimum design of PID controller in AVR system. IEEE Transactions on Energy Conversion. 2004;19(2):384–391. https://doi.org/10.1109/TEC.2003.821821

7. Fleming P.J., Purshouse R.C. Evolutionary algorithms in control systems engineering: A survey. Control Engineering Practice. 2002;10(11):1223–1241. https://doi.org/10.1016/S0967-0661(02)00081-3

8. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction. Cambridge: MIT Press; 2018. 552 p.

9. Dorf R.C., Bishop R.H. Modern Control Systems. Upper Saddle River: Prentice Hall; 2001. 831 p.

10. Cohen G.H., Coon G.A. Theoretical consideration of retarded control. Transactions of the ASME. 1953;75:827–834. https://doi.org/10.1115/1.4015451

11. Åström K.J., Hägglund T. Automatic tuning of simple regulators with specifications on phase and amplitude margins. Automatica. 1984;20(5):645–651. https://doi.org/10.1016/0005-1098(84)90014-1

12. Åström K.J., Wittenmark B. Adaptive Control. Reading: Addison-Wesley; 1995. 574 p.

13. Ioannou P., Fidan B. Adaptive Control Tutorial. Philadelphia: SIAM; 2006. 405 p. https://doi.org/10.1137/1.9780898718652

14. Han J. From PID to active disturbance rejection control. IEEE Transactions on Industrial Electronics. 2009;56(3):900–906. https://doi.org/10.1109/TIE.2008.2011621

15. Gao Zh. Scaling and bandwidth-parameterization based controller tuning. In: 2003 American Control Conference, 04–06 June 2003, Denver, CO, USA. IEEE; 2003. P. 4989–4996. https://doi.org/10.1109/ACC.2003.1242516

16. Gao Zh. Active disturbance rejection control: a paradigm shift in feedback control system design. In: 2006 American Control Conference, 14–16 June 2006, Minneapolis, MN, USA. IEEE; 2006. https://doi.org/10.1109/ACC.2006.1656579

17. Huang Y., Xue W. Active disturbance rejection control: methodology and theoretical analysis. ISA Transactions. 2014;53(4):963–976. https://doi.org/10.1016/j.isatra.2014.03.003

18. Khalil H.K. High-Gain Observers in Nonlinear Feedback Control. Philadelphia: SIAM; 2017. 323 p. https://doi.org/10.1137/1.9781611974867

19. Fu C., Tan W. Tuning of linear ADRC with known plant information. ISA Transactions. 2016;65:384–393. https://doi.org/10.1016/j.isatra.2016.06.016

20. Sutton R.H., Precup D., Singh S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence. 1999;112(1–2):181–211. https://doi.org/10.1016/S0004-3702(99)00052-1

21. Bradtke S., Duff M.O. Reinforcement learning methods for continuous-time Markov decision problems. In: Advances in Neural Information Processing Systems 7: Annual Conference on Neural Information Processing Systems, 28 November – 01 December 1994, Denver, CO, USA. Cambridge: MIT Press; 1995. P. 393–400.

22. Watkins Ch.J.C.H., Dayan P. Q-learning. Machine Learning. 1992;8(3–4):279–292. https://doi.org/10.1007/BF00992698

23. Mnih V., Kavukcuoglu K., Silver D., et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–533. https://doi.org/10.1038/nature14236

24. Shi Q., Lam H.-K., Xiao B., Tsai Sh.-H. Adaptive PID controller based on Q-learning algorithm. CAAI Transactions on Intelligence Technology. 2018;3(4):235–244. https://doi.org/10.1049/trit.2018.1007

25. Wang F., Ren B., Liu Y., Cui B. Tracking moving target for 6 degree-of-freedom robot manipulator with adaptive visual servoing based on deep reinforcement learning PID controller. Review of Scientific Instruments. 2022;93(4):045108. https://doi.org/10.1063/5.0087561

26. Qin Y., Zhang W., Shi J., Liu J. Improve PID controller through reinforcement learning. In: 2018 IEEE CSAA Guidance, Navigation and Control Conference, 10–12 August 2018, Xiamen, China. IEEE; 2018. https://doi.org/10.1109/GNCC42960.2018.9019095

27. Sutton R.S., McAllester D.A., Singh S., Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems 12: Annual Conference on Neural Information Processing Systems, 29 November – 04 December 1999, Denver, CO, USA. Cambridge: MIT Press; 2000. P. 1057–1063.

28. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 1992;8(3):229–256. https://doi.org/10.1007/BF00992696

29. Schulman J., Levine S., Abbeel P., Jordan M.I., Moritz Ph. Trust region policy optimization. In: Proceedings of the 32nd International Conference on Machine Learning, 06–11 July 2015, Lille, France. JMLR; 2015. P. 1889–1897.

30. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. Proximal policy optimization algorithms. arXiv. URL: https://arxiv.org/abs/1707.06347 [Accessed 13th July 2026].

31. Lillicrap T.P., Hunt J.J., Pritzel A., et al. Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations, 02–04 May 2016, San Juan, Puerto Rico. 2016. URL: https://arxiv.org/abs/1509.02971

32. Lakhani A.I., Chowdhury M.A., Lu Q. Stability-preserving automatic tuning of PID control with reinforcement learning. Complex Engineering Systems. 2022;2(1):3. https://doi.org/10.20517/ces.2021.15

33. Arshad J., Khan A., Aftab M., et al. Deep deterministic policy gradient to regulate feedback control systems using reinforcement learning. Computers, Materials & Continua. 2022;71(1):1153–1169. https://doi.org/10.32604/cmc.2022.021917

34. Lee D., Lee S.J., Yim S.C. Reinforcement learning-based adaptive PID controller for DPS. Ocean Engineering. 2020;216(11). https://doi.org/10.1016/j.oceaneng.2020.108053

35. Lawrence N.P., Forbes M.G., Loewen Ph.D., McClement D.G., Backström J.U., Gopaluni R.Bh. Deep reinforcement learning with shallow controllers: An experimental application to PID tuning. Control Engineering Practice. 2022;121(5):105046. https://doi.org/10.1016/j.conengprac.2021.105046

36. Wang H., Ricardez-Sandoval L.A. A deep reinforcement learning-based PID tuning strategy for nonlinear MIMO systems with time-varying uncertainty. IFAC-PapersOnLine. 2024;58(14):887–892. https://doi.org/10.1016/j.ifacol.2024.08.449

37. Veerasamy G., Balaji S., Kadirvelu Th., Ramasamy V. Reinforcement Learning Based Adaptive PID Controller for a Continuous Stirred Tank Heater Process. Iranian Journal of Chemistry and Chemical Engineering. 2025;44(1):265–282. https://doi.org/10.30492/ijcce.2024.2029225.6615

38. Fujimoto S., van Hoof H., Meger D. Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning, 10–15 July 2018, Stockholm, Sweden. PMLR; 2018. P. 1582–1591.

39. Haarnoja T., Zhou A., Abbeel P., Levine S. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, 10–15 July 2018, Stockholm, Sweden. PMLR; 2018. P. 1856–1865.

40. Wang Y., Fang Sh., Hu J., Huang D. A novel active disturbance rejection control of PMSM based on deep reinforcement learning for more electric aircraft. IEEE Transactions on Energy Conversion. 2023;38(2):1461–1470. https://doi.org/10.1109/TEC.2023.3235927

41. Deisenroth M.P., Rasmussen C.E. PILCO: A model-based and data-efficient approach to policy search. In: Proceedings of the 28th International Conference on Machine Learning, 28 June – 02 July 2011, Bellevue, WA, USA. Madison: Omnipress; 2011. P. 465–472.

42. Doerr A., Nguyen-Tuong D., Marco A., Schaal S., Trimpe S. Model-based policy search for automatic tuning of multivariate PID controllers. In: 2017 IEEE International Conference on Robotics and Automation, 29 May – 03 June 2017, Singapore. IEEE; 2017. P. 5295–5301. https://doi.org/10.1109/ICRA.2017.7989622

43. Chua K., Calandra R., McAllister R., Levine S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, 03–08 December 2018, Montréal, Canada. 2018. P. 4759–4770.

44. Altman E. Constrained Markov Decision Processes. Boca Raton: CRC Press; 1999. 256 p.

45. Stooke A., Achiam J., Abbeel P. Responsive safety in reinforcement learning by PID Lagrangian methods. In: Proceedings of the 37th International Conference on Machine Learning, 13–18 July 2020, Virtual Event. PMLR; 2020. P. 9133–9143.

46. Achiam J., Held D., Tamar A., Abbeel P. Constrained policy optimization. In: Proceedings of the 34th International Conference on Machine Learning, 06–11 August 2017, Sydney, NSW, Australia. PMLR; 2017. P. 22–31.

47. Berkenkamp F., Turchetta M., Schoellig A.P., Krause A. Safe model-based reinforcement learning with stability guarantees. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 04–09 December 2017, Long Beach, CA, USA. 2017. P. 908–918.

48. Zhang M., Zhang H., Ji J., Yang Y., Sun Y. Enhance the safety in reinforcement learning by ADRC Lagrangian methods. arXiv. URL: https://arxiv.org/abs/2601.18142 [Accessed 13th July 2026].

49. Bergstra J., Bengio Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research. 2012;13(2):281–305.

50. Jaderberg M., Dalibard V., Osindero S., et al. Population based training of neural networks. arXiv. URL: https://arxiv.org/abs/1711.09846 [Accessed 13th July 2026].

51. Dalibard V., Jaderberg M. Faster Improvement Rate Population Based Training. arXiv. URL: https://arxiv.org/abs/2109.13800 [Accessed 13th July 2026].

52. Tobin J., Fong R., Ray A., Schneider J., Zaremba W., Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, 24–28 September 2017, Vancouver, Canada. IEEE; 2017. P. 23–30. https://doi.org/10.1109/IROS.2017.8202133

53. Zhao W., Queralta J.P., Westerlund T. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In: 2020 IEEE Symposium Series on Computational Intelligence, 01–04 December 2020, Canberra, ACT, Australia. IEEE; 2020. P. 737–744. https://doi.org/10.1109/SSCI47803.2020.9308468

54. Reutov A., Vorobey S., Katanskiy A., et al. Optical stabilization for laser communication satellite systems through proportional-integral-derivative (PID) control and reinforcement learning approach. Review of Scientific Instruments. 2025;96(3):035121. https://doi.org/10.1063/5.0239029

55. Henderson P., Islam R., Bachman Ph., Pineau J., Precup D., Meger D. Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence. 2018;32(1). https://doi.org/10.1609/aaai.v32i1.11694