Journal of sustainable Energy Systems

Journal of sustainable Energy Systems

Development of a Deep Reinforcement Learning Framework for Microgrid Energy Management under High Renewable Energy Penetration Scenarios

Document Type : Original Article

Authors
Mona Mirrazavi 1
Hossein Yousefi 2
Ahmad Hajinezhad 3
1 M.Sc. Student in Energy Systems Engineering, Faculty of Energy Engineering and Sustainable Resources, College of Interdisciplinary Science and Technologies, University of Tehran, Tehran, Iran
2 Full Professor of Energy Systems Engineering, Faculty of Energy Engineering and Sustainable Resources, College of Interdisciplinary Science and Technologies, University of Tehran, Tehran, Iran
3 Associate Professor of Energy Systems Engineering, Faculty of Energy Engineering and Sustainable Resources, College of Interdisciplinary Science and Technologies, University of Tehran, Tehran, Iran
10.22059/ses.2026.413067.1228
Abstract
With the increasing penetration of renewable energy resources in microgrids, intelligent energy management has become increasingly important due to the intermittent nature of solar and wind generation. This study presents a deep reinforcement learning-based energy management framework for a grid-connected microgrid consisting of photovoltaic panels, a wind turbine, a battery energy storage system, a diesel generator, and the utility grid. The main novelty of this work lies in combining Double Deep Q-Network (Double DQN) with Prioritized Experience Replay (PER) and designing a multi-objective reward function that simultaneously considers operating cost, carbon emissions, supply reliability, and battery state-of-charge management before peak-price hours. This structure enables the agent to learn not only an economic operating policy but also decisions that adapt to the temporal patterns of renewable generation and electricity prices. The proposed model is evaluated over a 24-hour scheduling horizon after 2,000 training episodes. The results show satisfactory convergence, reducing the daily operating cost from approximately $10 to $3.44, corresponding to a 66% cost reduction. In addition, total carbon emissions decrease from nearly 41 kg CO₂ to 16.6 kg CO₂, representing about a 60% reduction. Moreover, load shedding is completely eliminated, and the demand is fully supplied throughout the day. Battery behavior analysis indicates that the agent charges the battery during periods of high renewable generation and discharges it during peak-price hours. The results confirm the effectiveness and scalability of the proposed method for energy management of renewable-rich microgrids.
Keywords
Deep Reinforcement Learning
Energy Management
Microgrid
Renewable Energy Sources
Subjects
[1] Olabi A, Elsaid K, Obaideen K, Abdelkareem MA, Rezk H, Wilberforce T, et al. Renewable energy systems: Comparisons, challenges and barriers, sustainability indicators, and the contribution to UN sustainable development goals. International Journal of Thermofluids. 2023;20:100498.
[2] Yajloo AB, Hamedani EA. Simulation of a solar-based small-scale green hydrogen production unit in Iran: A techno-economic-feasibility analysis. Results in Engineering. 2025:106734.
[3] Erdiwansyah f, Mahidin f, Husin H, Nasaruddin f, Zaki M, Muhibbuddin f. A critical review of the integration of renewable energy sources with various technologies. Protection and control of modern power systems. 2021;6(1):3.
[4] Ang T-Z, Salem M, Kamarol M, Das HS, Nazari MA, Prabaharan N. A comprehensive study of renewable energy sources: Classifications, challenges and suggestions. Energy strategy reviews. 2022;43:100939.
[5] Abbasian Hamedani E, Bahrami Yajloo A, Talebi S. A comprehensive review on carbon capture, transportation, storage, and utilization technologies; part Ⅰ: Carbon capture technologies. Advances in Energy Sciences and Technologies. 2025;1(1):119-32.
[6] Albarakati AJ, Boujoudar Y, Azeroual M, Eliysaouy L, Kotb H, Aljarbouh A, et al. Microgrid energy management and monitoring systems: A comprehensive review. Frontiers in Energy Research. 2022;10:1097858.
[7] Liu D, Zang C, Zeng P, Li W, Wang X, Liu Y, et al. Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties. Frontiers in Energy Research. 2023;11:1163053.
[8] Zia MF, Elbouchikhi E, Benbouzid M. Microgrids energy management systems: A critical review on methods, solutions, and prospects. Applied energy. 2018;222:1033-55.
[9] Shakya AK, Pillai G, Chakrabarty S. Reinforcement learning algorithms: A brief survey. Expert Systems with Applications. 2023;231:120495.
[10]                       Nian R, Liu J, Huang B. A review on reinforcement learning: Introduction and applications in industrial process control. Computers & Chemical Engineering. 2020;139:106886.
[11]           Matsuo Y, LeCun Y, Sahani M, Precup D, Silver D, Sugiyama M, et al. Deep learning, reinforcement learning, and world models. Neural Networks. 2022;152:267-75.
[12]           Wang H-n, Liu N, Zhang Y-y, Feng D-w, Huang F, Li D-s, et al. Deep reinforcement learning: a survey. Frontiers of Information Technology & Electronic Engineering. 2020;21(12):1726-44.
[13]           Gronauer S, Diepold K. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review. 2022;55(2):895-943.
[14]           Hamedani EA, Khodaparast P, Hosseini E, Mahmudy T, Yajloo AB. A mini-review of energy hub: Concept, components, classifications, and applications. Energy Reports. 2026;15:108886.
[15]           Sharma S, Ali I. Efficient energy management and cost optimization using multi-objective grey wolf optimization for EV charging/discharging in microgrid. e-Prime-Advances in Electrical Engineering, Electronics and Energy. 2024;10:100804.
[16]           Yousri D, Ousama A, Fathy A, Babu TS, Allam D. Managing the exchange of energy between microgrid elements based on multi-objective enhanced marine predators algorithm. Alexandria Engineering Journal. 2022;61(11):8487-505.
[17]           Vijaykumar G. Integrated energy management and harmonic mitigation in a microgrid using sea gull-ANN MPPT and advanced multi-level inverter. Ain Shams Engineering Journal. 2025;16(7):103397.
[18]           Mei Y, Li B, Wang H, Wang X, Negnevitsky M. Multi-objective optimal scheduling of microgrid with electric vehicles. Energy Reports. 2022;8:4512-24.
[19]           Babu VV, Roselyn JP, Sundaravadivel P. Multi-objective genetic algorithm based energy management system considering optimal utilization of grid and degradation of battery storage in microgrid. Energy Reports. 2023;9:5992-6005.
[20]           Aeggegn DB, Nyakoe GN, Wekesa C. An energy management system for multi-microgrid system considering uncertainties using multi-objective multi-verse optimization. Energy Reports. 2025;13:286-302.
[21]           Osama A, Allam D, Fathy A, Abdelaziz AY, Kim W-W, Hong J, et al. Optimal energy managing and power scheduling of microgrid in grid-connected mode using modified multi-objective manta ray technique. Energy Reports. 2025;13:4781-99.
[22]           Rochd A, Raihani A, Mahir O, Kissaoui M, Laamim M, Lahmer A, et al. Swarm Intelligence-driven Multi-objective optimization for microgrid energy management and trading considering DERs and EVs integration: case studies from green energy park, Morocco. Results in Engineering. 2025;25:104400.
[23]           Lamari M, Amrane Y, Boudour M, Boussahoua B. Multi‐objective economic/emission optimal energy management system for scheduling micro‐grid integrated virtual power plant. Energy Science & Engineering. 2022;10(8):3057-74.
[24]           Ali L, Muyeen S, Bizhani H, Ghosh A. A multi‐objective optimization for planning of networked microgrid using a game theory for peer‐to‐peer energy trading scheme. IET Generation, Transmission & Distribution. 2021;15(24):3423-34.
[25]           Kamarposhti MA, Colak I, Shokouhandeh H, Iwendi C, Padmanaban S, Band SS. Optimum operation management of microgrids with cost and environment pollution reduction approach considering uncertainty using multi‐objective NSGAII algorithm. IET Renewable Power Generation. 2025;19(1):e12579.
[26]           Momen S, Nikoukar J, Gandomkar M. Multi-objective optimization of energy consumption in microgrids considering CHPs and renewables using improved shuffled frog leaping algorithm. Journal of Electrical Engineering & Technology. 2023;18(3):1539-55.
[27]           Wang Z, Luo Y, Wu W, Cao L, Li Z. Multi-objective optimization models for power load balancing in distributed energy systems. Energy Informatics. 2025;8(1):104.
[28]           Wei W, Wang H, Hou K, Ji L. Multi‐objective optimal configuration of stand‐alone microgrids based on Benders decomposition considering power supply reliability. IET Energy Systems Integration. 2022;4(2):281-95.
[29]           Ahmadi B, Hoogsteen G, Zawadzki P, Gerards ME, Radziszewska W, Bykuć S, et al. A novel multi-objective approach to user-centric energy management systems. Energy Reports. 2025;14:185-204.
[30]           Niknami A, Askari MT, Ahmadi MA, Nik MB, Moghaddam MS. Resilient day-ahead microgrid energy management with uncertain demand, EVs, storage, and renewables. Cleaner Engineering and Technology. 2024;20:100763.
[31]           Veisi M, Adabi F, Kavousi‐Fard A, Karimi M. A novel comprehensive energy management model for multi‐microgrids considering ancillary services. IET Generation, Transmission & Distribution. 2022;16(23):4710-25.
[32]           Hosseini E, García-Triviño P, Horrillo-Quintero P, Carrasco-Gonzalez D, García-Vázquez CA, Sarrias-Mena R, et al. A novel reinforcement learning-based multi-objective energy management system for multi-energy microgrids integrating electrical, hydrogen, and thermal elements. Electric Power Systems Research. 2025;242:111474.
[33]           Cui Y, Xu Y, Li Y, Wang Y, Zou X. Deep reinforcement learning based optimal energy management of multi-energy microgrids with uncertainties. CSEE Journal of Power and Energy Systems. 2024.
[34]           Alabdullah MH, Abido MA. Microgrid energy management using deep Q-network reinforcement learning. Alexandria Engineering Journal. 2022;61(11):9069-78.
[35]           Zheng Y, Jia J, An D. Energy management for microgrids with hybrid hydrogen-battery storage: A reinforcement learning framework integrated multi-objective dynamic regulation. Processes. 2025;13(8):2558.
[36]           Yao F, Zhao W, Forshaw M, Zhou W. A unified data-driven approach under deep reinforcement learning with direct control responses for microgrid operations. Knowledge-Based Systems. 2025;325:113844.
[37]           Olivares DE, Mehrizi-Sani A, Etemadi AH, Cañizares CA, Iravani R, Kazerani M, et al. Trends in microgrid control. IEEE Transactions on smart grid. 2014;5(4):1905-19.
[38]           Rossow M. Electrification of Aircraft: Challenges, Barriers, and Potential Impacts. Continuing Education and Development, Inc, USA. 2021.
[39]           Das I, Cañizares CA. Renewable energy integration in diesel-based microgrids at the Canadian arctic. Proceedings of the IEEE. 2019;107(9):1838-56.
[40]           Basu M. Economic environmental dispatch using multi-objective differential evolution. Applied soft computing. 2011;11(2):2845-53.
[41]           Sutton RS, Barto AG. Reinforcement learning: An introduction: MIT press Cambridge; 1998.
[42]           Ye Y, Wang H, Chen P, Tang Y, Strbac G. Safe deep reinforcement learning for microgrid energy management in distribution networks with leveraged spatial–temporal perception. IEEE Transactions on Smart Grid. 2023;14(5):3759-75.
[43]           Van Hasselt H, Guez A, Silver D, editors. Deep reinforcement learning with double q-learning. Proceedings of the AAAI conference on artificial intelligence; 2016.
[44]           Schaul T, Quan J, Antonoglou I, Silver D. Prioritized experience replay. arXiv preprint arXiv:151105952. 2015.
[45]           Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. nature. 2015;518(7540):529-33.
[46]           Green MA. Solar cells: operating principles, technology, and system applications. 1981.
[47]           Anup K, Whale J, Urmee T. Urban wind conditions and small wind turbines in the built environment: A review. Renewable energy. 2019;131:268-83.
[48]           Sibley J, Richards R, Krikke R, Forney S, editors. The caterpillar 3176 heavy duty diesel engine. SAE International Truck and Bus Meeting and Exposition; 1988: SAE Technical Paper.
[49]           Aeggegn DB, Agajie TF, Workie YG, Khan B, Fopah-Lele A. Feasibility and techno-economic analysis of PV-battery priority grid tie system with diesel resilience: A case study. Heliyon. 2023;9(9).
[50]           Hiraishi T, Krug T, Tanabe K, Srivastava N, Baasansuren J, Fukuda M, et al. 2013 supplement to the 2006 IPCC guidelines for national greenhouse gas inventories: Wetlands: IPCC Switzerland; 2014.
[51]           Birol F. World energy outlook 2006. International Energy Agency. 2009.
[52]           Atcitty S, Neely J, Ingersoll D, Akhil A, Waldrip K. Battery energy storage system.  Power Electronics for Renewable and Distributed Energy Systems: A Sourcebook of Topologies, Control and Integration: Springer; 2013. p. 333-66.
[53]           Lai X, Yao J, Jin C, Feng X, Wang H, Xu C, et al. A review of lithium-ion battery failure hazards: Test standards, accident analysis, and safety suggestions. Batteries. 2022;8(11):248.
[54]           Truong CN, Naumann M, Karl RC, Müller M, Jossen A, Hesse HC. Economics of residential photovoltaic battery systems in Germany: The case of Tesla’s Powerwall. Batteries. 2016;2(2):14.
[55]           Ramdas A, McCabe K, Das P, Sigrin BO. California Time-of-Use (TOU) transition: Effects on distributed wind and solar economic potential. National Renewable Energy Laboratory (NREL), Golden, CO (United States); 2019.
[56]           Hamedani EA, Araghi AR, Hajinezhad A, Yousefi H. A Scenario-Based Techno-Economic Optimization Framework for Power-to-X Residential Energy Hubs Across Diverse Climate Conditions. Energy. 2026:141155.
[57]           Bahrami Yajloo A, Abbasian Hamedani E, Maleki P, Hosseinpour M. Optimization of economic dispatch for distributed generation-based power networks. Advances in Energy Sciences and Technologies. 2025;1(3):266-79.
[58]           Wang Y, Qiu D, Sun M, Strbac G, Gao Z. Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach. Applied Energy. 2023;335:120759.
[59]           Leite GMC, Jiménez-Fernández S, Salcedo-Sanz S, Marcelino CG, Pedreira CE. Solving an energy resource management problem with a novel multi-objective evolutionary reinforcement learning method. Knowledge-Based Systems. 2023;280:111027.
[60]           Domínguez-Barbero D, García-González J, Sanz-Bobi MÁ, García-Cerrada A. Energy management of a microgrid considering nonlinear losses in batteries through Deep Reinforcement Learning. Applied Energy. 2024;368:123435.
[61]           Xiong B, Zhang L, Hu Y, Fang F, Liu Q, Cheng L. Deep reinforcement learning for optimal microgrid energy management with renewable energy and electric vehicle integration. Applied Soft Computing. 2025;176:113180.
Journal of sustainable Energy Systems
Volume 5, Issue 4
Autumn 2026
Pages 763-786