Reinforcement Learning in Micro- and Smart Grids: Safe, Data-Driven Operating Strategies for Complex Energy Systems
Reinforcement Learning in Micro- and Smart Grids: Safe, Data-Driven Operating Strategies for Complex Energy Systems
The research project is funded and supported by the Committee for Research and Junior Academics at Paderborn University. It was awarded the research prize of the university in 2019.
Project acronym: RLMG
Funding period: 1 year (2020/01/01 – 2020/12/31)
Project partner: Department of Intelligent Systems and Machine Learning
Project profile: https://www.upb.de/en/research/fk/research-award-winners/
Motivation
The transformation of the current energy system to a sustainable structure characterized by renewable energies is a central social challenge of the 21st century. To achieve this, the inherent volatility of renewable energy sources requires a move away from conventional, hierarchically structured top-down energy networks towards flexible, cross-sectoral and intelligent energy systems. Therefore, in the course of the energy transition, so-called micro- and smart grids (MSG) represent an important solution component to ensure a clean, efficient and cost-effective energy supply in the future. MSG is the concept of a local grid consisting of energy sources (e.g. wind power), energy storage (e.g. battery) and energy consumers of different sectors (e.g. electricity, heat, mobility). The local integration of regenerative energies by means of MSGs, e.g. within industrial companies or residential areas, relieves the energy networks and, thus, reduces the need for cost- and resource-intensive network expansion.
The central hurdle to the establishment of MSGs is safe operation. MSGs are highly heterogeneous, complex and have a significant stochastic component, which is caused by the uncertainty of consumer behavior and renewable power plants. Classical methods of control engineering are not considered to be effective here. In contrast, reinforcement learning (RL) is a data-driven operating concept from the field of machine learning (ML), which has been able to celebrate promising success with similarly complex and stochastic problems (e.g. stock exchange trading). Nevertheless, research into MSG-specific ML strategies is still fraught with a high risk, since the security and availability of energy networks must meet the highest requirements: even a single wrong decision can lead to a complete system failure (black-out). In the absence of mathematically provable guarantees, the use of adaptive, data-driven methods of ML, whose behavior is fundamentally unpredictable, is extremely challenging in this context.
Project Objectives
The project should answer the following questions: Are RL-based operation strategies in principle able to control complex, heterogeneous and stochastic MSGs under highest security and availability requirements? Which MSG-specific methodological extensions are to be developed for reinforcement learning in order to achieve this goal?
There are basically two possible variants for operating MSGs. In the centralized approach, an agent (controller) derives control commands for all components in the MSG (e.g. battery storage) based on information about the entire system. In contrast, in the distributed concept, each entity decides on its own operating behavior (multi-agent system, MAS). This approach tends to be more robust and reduces the risk of a total failure as it is not dependent on a central component. However, the large number of decision makers increases system complexity, which increases the risk of wrong decisions in MAS as a whole.
In the course of the project, both approaches are to be examined with regard to their suitability, since no statements can yet be made on the basis of the available literature on related tasks. In order to achieve the primary goal of safe MSG operation, existing RL methods have to be extended in such a way that the learned models meet certain boundary conditions and fulfil given guarantees. In the case of decentralized control, this means the development of safe methods of MAS-RL, a very difficult problem for which there are no established approaches in the literature so far. In addition, further objectives are to be included in the investigation, e.g. minimizing operating and investment costs or maximizing regenerative energy to minimize environmental pollution. This goes hand in hand with the extension of RL methods to the case of multi-criteria optimization, for which no viable solution concepts exist so far either.
For numerical investigations a scalable model environment representing different MSG topologies is created. The goal is to automatically design arbitrary MSGs in order to evaluate the RL strategies under investigation with respect to their practicability and generalizability. In the long term, a transfer to the real test environment Microgrid-Laboratory, which is currently under construction in the LEA department, is planned.
Contact:
Dr.-Ing. Oliver Wallscheid

Leistungselektronik und Elektrische Antriebstechnik (LEA)
Oberingenieur - Mitarbeiter - Leitung Fachgruppe Elektrische Antriebstechnik & Intelligente Energiesysteme
Terminvereinbarung
33098 Paderborn
Liste im Research Information System öffnen
2023
S. Peitz, J. Stenner, V. Chidananda, O. Wallscheid, S.L. Brunton, K. Taira, in: arXiv:2301.10737, 2023
A. Brosch, O. Wallscheid, J. Böcker, IEEE Transactions on Power Electronics (2023), pp. 1-14
2022
N. Förster, P. Rehlaender, O. Wallscheid, F. Schafmeister, J. Böcker, in: Proc. 37th IEEE Applied Power Electronics Conference (APEC), IEEE, 2022
A. Brosch, O. Wallscheid, J. Böcker, IEEE Transactions on Power Electronics (2022)
A. Brosch, O. Wallscheid, J. Böcker, IEEE Transactions on Power Electronics (2022)
J. Lange, D. Schmies, K.S.C. Stille, J. Böcker, O. Wallscheid, in: EPE'22 ECCE Europe, IEEE, 2022
M. Stender, M. Becker, O. Wallscheid, J. Böcker, in: 48th Annual Conference of the Industrial Electronics Society (IECON), 2022
N. Förster, P. Rehlaender, O. Wallscheid, F. Schafmeister, J. Böcker, in: 2022 IEEE Applied Power Electronics Conference and Exposition (APEC), IEEE, 2022
N. Förster, T. Piepenbrock, P. Rehlaender, O. Wallscheid, F. Schafmeister, J. Böcker, in: PCIM Europe 2022; International Exhibition and Conference for Power Electronics, Intelligent Motion, Renewable Energy and Energy Management, 2022, pp. 1-10
B. Haucke-Korber, M. Schenke, O. Wallscheid, in: IKMT 2022; 13. GMM/ETG-Symposium, 2022, pp. 1-6
W. Kirchgässner, O. Wallscheid, J. Böcker, Engineering Applications of Artificial Intelligence (2022), 117, 105537
W. Kirchgässner, O. Wallscheid, J. Böcker, in: 2022 International Power Electronics Conference (IPEC-Himeji 2022- ECCE Asia), IEEE, 2022
P. Rehlaender, O. Wallscheid, F. Schafmeister, J. Böcker, IEEE Transactions on Power Electronics (2022), 37(11), pp. 13413-13427
W. Kirchgässner, D. Wöckinger, O. Wallscheid, G. Bramerdorfer, J. Böcker, in: IKMT 2022; 13. GMM/ETG-Symposium, 2022, pp. 1-6
A. Brosch, J. Rauhaus, O. Wallscheid, J. Böcker, D. Zimmer, IEEE Open Journal of Industry Applications (2022)
2021
A. Brosch, O. Wallscheid, J. Böcker, IEEE Open Journal of Industry Applications (2021), 2, pp. 47–63
E.G. Gedlu, O. Wallscheid, J. Böcker, in: 2021 IEEE International Electric Machines & Drives Conference (IEMDC), 2021, pp. 1–8
G. Book, A. Traue, P. Balakrishna, A. Brosch, M. Schenke, S. Hanke, W. Kirchgässner, O. Wallscheid, IEEE Open Journal of Power Electronics (2021), pp. 187-201
D. Weber, S. Heid, H. Bode, J. Lange, E. Hüllermeier, O. Wallscheid, IEEE Access (2021), 9, pp. 35654–35669
M. Stender, O. Wallscheid, J. Böcker, IEEE Transactions on Power Electronics (2021), 36(11), pp. 13261-13274
M. Stender, O. Wallscheid, J. Böcker, in: 2021 IEEE 19th International Power Electronics and Motion Control Conference (PEMC), IEEE, 2021
M. Stender, O. Wallscheid, J. Böcker, in: IECON 2021 – 47th Annual Conference of the IEEE Industrial Electronics Society, IEEE, 2021
M. Stender, O. Wallscheid, J. Böcker, in: 2021 IEEE 19th International Power Electronics and Motion Control Conference (PEMC), IEEE, 2021
S. Hanke, O. Wallscheid, J. Böcker, IET Digital Library, 2021
O. Wallscheid, IEEE Open Journal of Industry Applications (2021)
W. Kirchgässner, O. Wallscheid, J. Böcker, IEEE Transactions on Energy Conversion (2021), 36(3), pp. 2059 - 2067
P. Balakrishna, G. Book, W. Kirchgässner, M. Schenke, A. Traue, O. Wallscheid, Journal of Open Source Software (2021), 2498
M. Schenke, O. Wallscheid, IEEE Open Journal of the Industrial Electronics Society (2021), pp. 388-400
M. Schenke, O. Wallscheid, arXiv preprint arXiv:2105.08990 (2021)
A. Brosch, O. Wallscheid, J. Böcker, IEEE Transactions on Industrial Informatics (2021)
W. Kirchgässner, O. Wallscheid, J. Böcker, in: arXiv preprint arXiv:2103.16323, 2021
2020
A. Traue, G. Book, W. Kirchgässner, O. Wallscheid, IEEE Transactions on Neural Networks and Learning Systems (2020), pp. 1-10
W. Kirchgässner, O. Wallscheid, J. Böcker, IEEE Transactions on Power Electronics (2020), 36(7), pp. 7480-7488
A. Brosch, S. Hanke, O. Wallscheid, J. Böcker, IEEE Transactions on Power Electronics (2020), pp. 2179-2190
K.S.C. Stille, D. Weber, J. Lange, T. Vogt, O. Wallscheid, J. Böcker, in: 2020 International Symposium on Power Electronics, Electrical Drives, Automation and Motion (SPEEDAM), IEEE, 2020
M. Stender, O. Wallscheid, J. Böcker, IEEE Transactions on Industrial Electronics (2020), 68(9), pp. 8646-8656
M. Stender, O. Wallscheid, J. Böcker, in: 2020 International Symposium on Power Electronics, Electrical Drives, Automation and Motion (SPEEDAM), IEEE, 2020
S. Heid, D. Weber, H. Bode, E. Hüllermeier, O. Wallscheid, Journal of Open Source Software (2020), 5(54), pp. 2435
H. Bode, S. Heid, D. Weber, E. Hüllermeier, O. Wallscheid, arXiv preprint arXiv:2005.04869 (2020)
E.G. Gedlu, O. Wallscheid, J. Böcker, in: The 10th International Conference on Power Electronics, Machines and Drives (PEMD 2020), 2020, pp. 937–942
O. Wallscheid, E.F.B. Ngoumtsa, IEEE Transactions on Power Electronics (2020), 35(12), pp. 13563–13572
S. Hanke, O. Wallscheid, J. Böcker, arXiv preprint arXiv:2003.06268 (2020)
W. Kirchgässner, O. Wallscheid, J. Böcker, arXiv preprint arXiv:2001.06246 (2020)
M. Stender, O. Wallscheid, J. Böcker, 2020
2019
M. Schenke, W. Kirchgässner, O. Wallscheid, IEEE Transactions on Industrial Informatics (2019), pp. 4650-4658
W. Kirchgässner, O. Wallscheid, J. Böcker, in: 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE), 2019
W. Kirchgässner, O. Wallscheid, J. Böcker, in: 2019 IEEE International Electric Machines & Drives Conference (IEMDC), 2019
M. Stender, O. Wallscheid, J. Böcker, in: 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE), IEEE, 2019
S. Hanke, S. Peitz, O. Wallscheid, J. Böcker, M. Dellnitz, in: 2019 IEEE International Symposium on Predictive Control of Electrical Drives and Power Electronics (PRECEDE), 2019, pp. 1–6
O. Wallscheid, M.S. Shafiq, J. Böcker, in: 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE), 2019, pp. 402–407
O. Wallscheid, E.F.B. Ngoumtsa, J. Böcker, in: 2019 IEEE International Electric Machines & Drives Conference (IEMDC), 2019, pp. 2188–2195
S. Hanke, O. Wallscheid, J. Böcker, in: 2019 IEEE International Electric Machines & Drives Conference (IEMDC), 2019, pp. 2210–2216
D.E.E. Gaona, O. Wallscheid, J. Böcker, IEEE Transactions on Industrial Electronics (2019)
2018
D. Weber, K.S.C. Stille, O. Wallscheid, J. Böcker, in: 2018 International Symposium on Power Electronics, Electrical Drives, Automation and Motion (SPEEDAM), IEEE, 2018
D. Weber, M.I. Rafsan Jani, M. Grabo, O. Wallscheid, T. Klaus, S. Krauter, J. Böcker, in: World Conference on Photovoltaic Energy Conversion (WCPEC-7), 45th IEEE PVSC, 28th PVSEC, 34th EU PVSEC., 2018
O. Wallscheid, M. Schenke, J. Böcker, in: 2018 21st International Conference on Electrical Machines and Systems (ICEMS), 2018, pp. 1181–1186
O. Wallscheid, M. Schenke, J. Böcker, in: 2018 IEEE 18th International Power Electronics and Motion Control Conference (PEMC), 2018, pp. 793–799
S. Hanke, S. Peitz, O. Wallscheid, S. Klus, J. Böcker, M. Dellnitz, arXiv preprint arXiv:1804.00854 (2018)
2017
O. Wallscheid, W. Kirchgässner, J. Böcker, in: 2017 International Joint Conference on Neural Networks (IJCNN), 2017
D. Gaona, O. Wallscheid, J. Böcker, in: 2017 IEEE 12th International Conference on Power Electronics and Drive Systems (PEDS), 2017, pp. 1–126
D. Gaona, O. Wallscheid, J. Böcker, in: 2017 IEEE 12th International Conference on Power Electronics and Drive Systems (PEDS), 2017, pp. 599–606
S. Hanke, O. Wallscheid, J. Böcker, in: 2017 IEEE International Symposium on Predictive Control of Electrical Drives and Power Electronics (PRECEDE), 2017, pp. 101–106
O. Wallscheid, A. Specht, J. Böcker, IEEE Transactions on Industrial Electronics (2017), 64(5), pp. 3921–3929
D. Gaona, O. Wallscheid, J. Böcker, in: 2017 IEEE Southern Power Electronics Conference (SPEC), 2017, pp. 1–6
O. Wallscheid, J. Böcker, in: 2017 IEEE International Electric Machines and Drives Conference (IEMDC), 2017, pp. 1–8
O. Wallscheid, J. Böcker, in: 2017 IEEE International Symposium on Predictive Control of Electrical Drives and Power Electronics (PRECEDE), 2017, pp. 31–36
T. Vogt, D. Weber, O. Wallscheid, J. Böcker, in: International Joint Conference on Neural Networks (IJCNN), 2017, pp. 3228–3235
O. Wallscheid, Shaker Verlag, 2017
2016
O. Wallscheid, T. Huber, W. Peters, J. Böcker, EPE Journal (2016)(25), pp. 1–10
O. Wallscheid, U. Ammann, J. Böcker, in: PCIM Europe 2016; International Exhibition and Conference for Power Electronics, Intelligent Motion, Renewable Energy and Energy Management, 2016, pp. 1–8
2015
O. Wallscheid, J. Böcker, IEEE Transactions on Energy Conversion (2015), 31(1), pp. 354–365
O. Wallscheid, M. Meyer, J. Böcker, in: 2015 IEEE 11th International Conference on Power Electronics and Drive Systems, 2015, pp. 981–985
O. Wallscheid, J. Böcker, in: European Conference on Power Electronics and Applications (EPE), 2015
O. Wallscheid, J. Böcker, in: IEEE International Electric Machines and Drives Conference (IEMDC), 2015
W. Peters, O. Wallscheid, J. Böcker, in: European Conference on Power Electronics and Applications (EPE), 2015
2014
O. Wallscheid, T. Huber, W. Peters, J. Böcker, in: IECON 2014-40th Annual Conference of the IEEE Industrial Electronics Society, 2014, pp. 811–818
A. Specht, O. Wallscheid, J. Böcker, in: 2014 International Power Electronics Conference (IPEC-Hiroshima 2014-ECCE ASIA), 2014, pp. 1501–1507
2013
O. Wallscheid, J. Böcker, in: ETG-Fachbericht-Internationaler ETG-Kongress 2013–Energieversorgung auf dem Weg nach 2050, 2013
A. Specht, S. Ober-Blöbaum, O. Wallscheid, C. Romaus, J. Böcker, in: Electric Machines & Drives Conference (IEMDC), 2013 IEEE International, 2013, pp. 1411–1417
2012
W. Peters, O. Wallscheid, J. Böcker, in: IECON 2012-38th Annual Conference on IEEE Industrial Electronics Society, 2012, pp. 2877–2882
2011
O. Wallscheid, W. Peters, J. Böcker, Aktuelle Technik (2011), 5(1), pp. 30–33
W. Peters, O. Wallscheid, J. Böcker, in: IECON 2011-37th Annual Conference of the IEEE Industrial Electronics Society, 2011, pp. 534–538