یادگیری تقویتی (RL) در سامانه‌های انرژی: مروری بر بهینه‌سازی تطبیقی، چالش‌های کنونی و مسیرهای آینده

نوع مقاله : مروری

نویسندگان

1 دانشیار، دانشکدۀ مهندسی انرژی و منابع پایدار، رئیس مؤسسۀ فناوری‌های نرم، دانشکدگان علوم و فناوری‌های میان‌رشته‌ای، دانشگاه تهران، تهران، ایران

2 دانشجوی دکتری مهندسی سیستم‌های انرژی، دانشکدۀ مهندسی انرژی و منابع پایدار، دانشکدگان علوم و فناوری‌های میان‌رشته‌ای، دانشگاه تهران، تهران، ایران

چکیده

صنعت با افزایش تقاضای جهانی برای انرژی و پیچیدگی روزافزون سیستم‌های انرژی، به‌ویژه در زمینۀ منابع تجدیدپذیر و شبکه‌های هوشمند، نیاز به روش‌های هوشمند برای بهینه‌سازی تولید، توزیع و مصرف انرژی بیش از پیش احساس می‌شود. یادگیری تقویتی (RL)، به‌ عنوان یکی از شاخه‌های پیشرفتۀ هوش مصنوعی، با توانایی یادگیری سیاست‌های بهینه از طریق تعامل پویا با محیط و سازگاری با عدم قطعیت‌ها، راهکارهای نوینی برای مدیریت سیستم‌های انرژی ارائه کرده است. این مقاله به بررسی مفاهیم پایۀ یادگیری تقویتی، مانند فرایندهای تصمیم‌گیری مارکوف و الگوریتم‌های مرتبط، مزایا و معایب این روش، کاربردهای عملی آن در مدیریت شبکه‌های هوشمند، بهینه‌سازی ذخیره‌سازی انرژی و مدیریت خودروهای الکتریکی می‌پردازد. همچنین، RL با سایر روش‌های بهینه‌سازی، نظیر یادگیری ماشین نظارت‌شده، الگوریتم‌های تکاملی و مدل‌های ریاضی سنتی مقایسه شده و جهت‌گیری‌های آیندۀ آن، از جمله ادغام با فناوری‌های نوین مانند اینترنت اشیا و بلاک‌چین، بررسی می‌شود. تمرکز ویژه‌ای بر پتانسیل RL در حل چالش‌های بومی ایران، مانند خاموشی‌های مکرر و ناکارایی شبکه‌های توزیع، ارائه شده است تا راهکارهایی برای پایداری انرژی در سطح ملی پیشنهاد شود.

کلیدواژه‌ها

موضوعات

عنوان مقاله [English]

Reinforcement Learning (RL) in Energy Systems: A Review of Adaptive Optimization, Current Challenges, and Future Directions

نویسندگان [English]

  • Amirali Saifoddin 1
  • Ehsan Abdolvand 2
  • Mohammadali Allahrabbi Shirazi 2
1 Associate Professor, School of Energy Engineering and Sustainable Resources, Head of the Institute of Soft Technologies, College of Interdisciplinary Sciences and Technologies, University of Tehran, Tehran, Iran
2 PhD Student in Energy Systems Engineering, School of Energy Engineering and Sustainable Resources, College of Interdisciplinary Sciences and Technologies, University of Tehran, Tehran, Iran
چکیده [English]

With the increasing global demand for energy and the increasing complexity of energy systems, especially in the field of renewable resources and smart grids, the need for intelligent methods to optimize energy production, distribution, and consumption is increasingly felt. Reinforcement learning (RL), as an advanced branch of artificial intelligence, has provided new solutions for energy system management with the ability to learn optimal policies through dynamic interaction with the environment and adapt to uncertainties. This paper reviews the basic concepts of reinforcement learning, such as Markov decision processes and related algorithms, the advantages and disadvantages of this method, its practical applications in smart grid management, energy storage optimization, and electric vehicle management. RL is also compared with other optimization methods, such as supervised machine learning, evolutionary algorithms, and traditional mathematical models, and its future directions, including integration with new technologies such as the Internet of Things and blockchain, are reviewed. A special focus is placed on the potential of RL in solving Iran's endemic challenges, such as frequent blackouts and inefficient distribution networks, to propose solutions for energy sustainability at the national level.

کلیدواژه‌ها [English]

  • Energy systems
  • adaptive optimization
  • reinforcement learning
  • energy system management

مراجع

  • Almutairi A, Abiyo S, Hyejian J. Secured and Smart System for Energy Management in Microgrids Using Deep Reinforcement Learning. IEEE Transactions on Consumer Electronics. 2025 Jun 4.
  • Vamvakas D, Michailidis P, Korkas C, Kosmatopoulos E. Review and evaluation of reinforcement learning frameworks on smart grid applications. Energies. 2023 Jul 12;16(14):5326.
  • Moaven, M., Allahrabbi Shirazi, M., & Ghodusinejad, M. H. (2025). Systematic Analysis of the Impact of Life Cycle Assessment of Materials on Urban Heat Island Mitigation: A Path Toward Sustainable Urban Policy Using the PRISMA Method. Urban Development Policy Making, 2(1), 95-110.
  • Sutton RS, Barto AG. Reinforcement learning: An introduction. Cambridge: MIT press; 1998 Mar 1.
  • Glavic M, Fonteneau R, Ernst D. Reinforcement learning for electric power system decision and control: Past considerations and perspectives. IFAC-PapersOnLine. 2017 Jul 1;50(1):6918-27.
  • Jahani MT, Nazarian P, Safari A, Haghifam MR. Multi-objective optimization model for optimal reconfiguration of distribution networks with demand response services. Sustainable Cities and Society. 2019 May 1;47:101514.
  • Maeda I, DeGraw D, Kitano M, Matsushima H, Sakaji H, Izumi K, Kato A. Deep reinforcement learning in agent based financial market simulation. Journal of Risk and Financial Management. 2020 Apr 11;13(4):71.
  • Russell SJ, Norvig P. Artificial intelligence: A modern approach;[the intelligent agent book]. Prentice hall; 1995.
  • Gupta A, Badr Y, Negahban A, Qiu RG. Energy-efficient heating control for smart buildings with deep reinforcement learning. Journal of Building Engineering. 2021 Feb 1;34:101739.
  • Kumar A, Maulik A, Chinmaya KA. Energy Management Strategies for Active Distribution Networks and Microgrids–A Comprehensive Survey. IETE Technical Review. 2025 Jun 28:1-40.
  • Liang, Jiaxin, et al. "A review of multi-agent reinforcement learning algorithms." Electronics 14.4 (2025): 820.
  • Shojaeighadikolaei, Amin, et al. "Distributed Energy Management and Demand Response in Smart Grids: A Multi-Agent Deep Reinforcement Learning Framework." arXiv preprint arXiv:2211.15858 (2022).
  • Arwa EO, Folly KA. Reinforcement learning techniques for optimal power control in grid-connected microgrids: A comprehensive review. Ieee Access. 2020 Nov 17;8:208992-9007.
  • Duan J, Yi Z, Shi D, Lin C, Lu X, Wang Z. Reinforcement-learning-based optimal control of hybrid energy storage systems in hybrid AC–DC microgrids. IEEE Transactions on Industrial Informatics. 2019 Jan 31;15(9):5355-64.
  • Latoń D, Grela J, Ożadowicz A. Applications of Deep Reinforcement Learning for Home Energy Management Systems: A Review. Energies. 2024 Dec 20;17(24):6420.
  • Saifoddin A, Mirzaei N, Allahrabbi Shirazi M, Yousefi H. Comparative Applications of Supervised and Unsupervised Machine Learning Models in Energy Systems. Journal of Energy Management and Technology. 2025 Dec 1;9(4):284-90.
  • Subramanya R, Sierla SA, Vyatkin V. Exploiting battery storages with reinforcement learning: a review for energy professionals. IEEE Access. 2022 May 18;10:54484-506.
  • Vázquez-Canteli JR, Nagy Z. Reinforcement learning for demand response: A review of algorithms and modeling techniques. Applied energy. 2019 Feb 1;235:1072-89.
  • Michailidis P, Michailidis I, Kosmatopoulos E. Reinforcement Learning for Electric Vehicle Charging Management: Theory and Applications. Energies. 2025 Oct 1;18(19):5225.
  • Duan Y, Chen X, Houthooft R, Schulman J, Abbeel P. Benchmarking deep reinforcement learning for continuous control. InInternational conference on machine learning 2016 Jun 11 (pp. 1329-1338). PMLR.
  • Giannelos S. Reinforcement Learning in Energy Finance: A Comprehensive Review. Energies (19961073). 2025 Jun 1;18(11).
  • Mohammaddini S, Yousefi H, Abdoos M, Shirazi MA, Hajinezhad A. Techno-economic simulation of solar flat plate collector systems for building hot water demand supply. Energy. 2025 Oct 16:138933.
  • Singh D, Shah OA, Arora S. Adaptive control strategies for effective integration of solar power into smart grids using reinforcement learning. Energy Storage and Saving. 2024 Dec 1;3(4):327-40.
  • Wang D, Zheng W, Wang Z, Wang Y, Pang X, Wang W. Comparison of reinforcement learning and model predictive control for building energy system optimization. Applied Thermal Engineering. 2023 Jun 25;228:120430.
  • Dulac-Arnold G, Evans R, van Hasselt H, Sunehag P, Lillicrap T, Hunt J, Mann T, Weber T, Degris T, Coppin B. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679. 2015 Dec 24.
  • Stavrev S, Ginchev D. Reinforcement learning techniques in optimizing energy systems. Electronics. 2024 Apr 12;13(8):1459.
  • Bui VH, Das S, Hussain A, Hollweg GV, Su W. A critical review of safe reinforcement learning techniques in smart grid applications. arXiv preprint arXiv:2409.16256. 2024 Sep 24.
  • Jin, Ming. "Reinforcement Learning Meets the Power Grid: A Contemporary Survey with Emphasis on Safety and Multi-agent Challenges." Foundations and Trends® in Electric Energy Systems 8.3-4 (2025): 169-316.
  • Bui, Van-Hai, et al. "A critical review of safe reinforcement learning strategies in power and energy systems." Engineering Applications of Artificial Intelligence 143 (2025): 110091.
  • Tabas, Daniel, and Baosen Zhang. "Computationally efficient safe reinforcement learning for power systems." 2022 American Control Conference (ACC). IEEE, 2022.
  • Ceusters, Glenn, et al. "Safe reinforcement learning for multi-energy management systems with known constraint functions." Energy and AI 12 (2023): 100227.
  • Zhang H, Sun X, Lee MH, Moon J. Deep reinforcement learning-based active network management and emergency load-shedding control for power systems. IEEE Transactions on Smart Grid. 2023 Aug 8;15(2):1423-37.
  • Guan Y, Ma W, Che L, Shahidehpour M. Model-Based Safe Reinforcement Learning for Active Distribution Network Scheduling. IEEE Transactions on Smart Grid. 2025 Mar 17.
  • Allahrabbi Shirazi, M. A., Goldoust, A., Khatami, M., Abedi, E., & Janfeshan, M. H. (2024). Modeling and simulation of a solar tracker with bifacial panel: a case study of Tehran city. Journal of Sustainable Energy Systems, 3(3), 271-287.
  • Dang Q, Wu D, Boulet B. A q-learning based charging scheduling scheme for electric vehicles. In2019 IEEE Transportation Electrification Conference and Expo (ITEC) 2019 Jun 19 (pp. 1-5). IEEE.
  • Xu S, Fu Y, Wang Y, Yang Z, Huang C, O’Neill Z, Wang Z, Zhu Q. Efficient and assured reinforcement learning-based building HVAC control with heterogeneous expert-guided training. Scientific reports. 2025 Mar 5;15(1):7677.
  • Samadi E, Badri A, Ebrahimpour R. Q-Learning-Oriented Distributed Energy Management of Grid-Connected Microgrid. In2021 29th Iranian Conference on Electrical Engineering (ICEE) 2021 May 18 (pp. 318-322). IEEE.
  • Ministry of Energy / Tavanir Company. Research Priorities of Iran’s Ministry of Energy in 2021. Tehran. 2021; Available: https://gsme.semnan.ac.ir/uploads/23/2021/Aug/21/tavanir_1.docx (In persia).
  • Asghari Oskoei M, Fallahi F, Doostizadeh M, Moshiri S. Reinforcement Learning Applied to Multi Agent Modelling, the Case of the Iranian Power Market. Iranian Energy Economics. 2017;7(25):1-40.
  • Mousavi Ziabari Z, Azmi R. Demand Forecasting for Dynamic Pricing in Smart Electricity Markets. 10th Conference on Information Technology and Knowledge (IKT2019). Tehran. 2019.
  • Mohsenzadeh‐Yazdi, Hossein, Hamed Kebriaei, and Farrokh Aminifar. "Multi‐agent reinforcement learning in a new transactive energy mechanism." IET Generation, Transmission & Distribution 18.18 (2024): 2943-2955
  • Lissa P, Deane C, Schukat M, Seri F, Keane M, Barrett E. Deep reinforcement learning for home energy management system control. Energy and AI. 2021 Mar 1;3:100043.
  • Salmi C, Senoussi NE, Chaal D, Boudjadi M. An in-depth Investigation Into the Application of Flash Memory In a Business Intelligence Database Environment.
  • Su Y, Yue S, Qiu L, Chen J, Wang R, Tan M. Energy management for scalable battery swapping stations: A deep reinforcement learning and mathematical optimization cascade approach. Applied Energy. 2024 Jul 1;365:123212.
  • Li P, Hao J, Tang H, Fu X, Zhen Y, Tang K. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey on hybrid algorithms. IEEE Transactions on evolutionary computation. 2024 Aug 14.
  • Mulayim, Ozan Baris, et al. "Comparative Field Deployment of Reinforcement Learning and Model Predictive Control for Residential HVAC." arXiv preprint arXiv:2510.01475 (2025).
  • Gokhale, Gargya, Bert Claessens, and Chris Develder. "PhysQ: a physics informed reinforcement learning framework for building control." arXiv preprint arXiv:2211.11830 (2022).
  • Xiong, Rui, Jiayi Cao, and Quanqing Yu. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle." Applied energy 211 (2018): 538-548.
  • Im, Jaedong, et al. "Reinforcement learning-based energy management system in the complex electric tariff environment." International Journal of Electrical Power & Energy Systems 172 (2025): 111038.
  • Zhang, Haifeng, et al. "Resilient dispatching optimization of power system driven by deep reinforcement learning model." Discover Artificial Intelligence 5.1 (2025): 189.
  • Xu Y, Li Y, Gao W. Comparative Analysis of Reinforcement Learning Approaches for Multi-Objective Optimization in Residential Hybrid Energy Systems. Buildings. 2024;14(9):2645. doi: 10.3390/buildings14092645.
  • Naziri A, Tahavvor AR, Shirazi MA, Zahedi R. Feasibility study on heat recovery from gas turbine exhaust for absorption chiller operation and efficiency enhancement using neural networks. Thermal Science and Engineering Progress. 2025;104210.
  • Shirazi MA, Zahedi R, Yousefi H, Aslani A. Environmental and damage assessment of electric vehicles compared to internal combustion engine vehicles under various ambient temperature scenarios using the LCA approach. Energy Nexus. 2025 Nov 19;100606.
  • Allah Rabbi Shirazi MA. Idea generation and examination of environmental challenges of floating solar photovoltaic power plants on wetlands and its economic advantage for local communities. Journal of Sustainable Energy Systems. 2024;3(1):39-51.
  • Pallonetto F, De Rosa M, Milano F, Finn DP. Demand response algorithms for smart-grid ready residential buildings using machine learning models. Applied energy. 2019 Apr 1;239:1265-82.
  • Zhang Z, Zhang D, Qiu RC. Deep reinforcement learning for power system applications: An overview. CSEE Journal of Power and Energy Systems. 2019 Oct 7;6(1):213-25.
  • Ozan AT, Kamalaruban P. Applications of reinforcement learning in energy systems. Renewable and Sustainable Energy Reviews. 2021 Mar 1;137:110618.
فصلنامه سیستم های انرژی پایدار
دوره 5، شماره 3
تیر 1405
صفحه 527-547

فایل ها

هم رسانی

ارجاع به این مقاله

آمار