Control Theory and Reinforcement Learning: Connections and Challenges - Spring School

Prof. Dr. Bert Kappen completed his PhD in theoretical particle physics in 1987 at the Rockefeller University in New York. From 1987 until 1989 he worked as a scientist at the Philips Research Laboratories in Eindhoven, the Netherlands. Since 2004 he is full professor on machine learning and neural networks at the science faculty of the Radboud University. In 1998, he co-founded the company Smart Research that commercializes applications of neural networks and machine learning.

Bert Kappen conducts research on neural networks, Bayesian machine learning, stochastic control theory and computational neuroscience. Currently, he is investigating ways to use quantum mechanics for a new generation of quantum machine learning algorithms and control methods for quantum computing.

Dr. Debabrota Basu works on designing theoretically-grounded and practically-efficient algorithms for bandits and Reinforcement Learning. He also conducts active research on robustness, privacy, and fairness in machine learning, in brief responsible AI. He actively works on developing robust, private, and fair Reinforcement Learning for algorithmic decision making in health, education, and agro-ecology. He was awarded ANR young researcher grant for studying impacts of responsible AI constraints in sequential decision making. In the ACM EAAMO 2022 conference, his work on fair college admissions got the best student paper award. In IJCAI 2023, he presented the tutorial on "Auditing Bias of Machine Learning Algorithms". Till now, he has delivered multiple talks to general audiences, policymakers, and lawmakers in Europe, Asia, and USA on frontiers and opportunities of algorithmic auditing. He has been elected as a scholar of European Learning Society (ELLIS) in 2024.

Dr. Frans A. Oliehoek is Associate Professor at Delft University of Technology, where he is a leader of the sequential decision making group, a scientific director of the Mercury machine learning lab, and director and co-founder of the ELLIS Unit Delft. He received his Ph.D. in Computer Science (2010) from the University of Amsterdam (UvA), and held positions at various universities including MIT, Maastricht University and the University of Liverpool. Frans' research interests revolve around intelligent systems that learn about their environment via interaction, building on techniques from machine learning, AI and game theory. He has served as PC/SPC/AC at top-tier venues in AI and machine learning, and currently serves as associate editor for JAIR and AIJ. He is a Senior Member of AAAI, and was awarded a number of personal research grants, including a prestigious ERC Starting Grant.

Prof. Dr. Maryam Kamgarpour holds a Doctor of Philosophy in Engineering from the University of California, Berkeley and a Bachelor of Applied Science from University of Waterloo, Canada. Her research is on safe decision-making and control under uncertainty, game theory and mechanism design, mixed integer and stochastic optimization and control. Her theoretical research is motivated by control challenges arising in intelligent transportation networks, robotics, power grid systems and healthcare. She is the recipient of NASA High Potential Individual Award, NASA Excellence in Publication Award, and the European Union Starting Grant (2016-2021), European Control Award (2024).

Prof. Dr. Sean Meyn was raised by the beach in Southern California. Following his BA in mathematics at UCLA, he moved on to pursue a PhD with Peter Caines at McGill University. After about 20 years as a professor of ECE at the University of Illinois, in 2012 he moved to beautiful Gainesville. He is now Professor and Robert C. Pittman Eminent Scholar Chair in the Department of Electrical and Computer Engineering at the University of Florida, and director of the Laboratory for Cognition and Control. He also holds an Inria International Chair to support research with colleagues in France. His interests span many aspects of stochastic control, stochastic processes, information theory, and optimization. For the past decade, his applied research has focused on engineering, markets, and policy in energy systems.

Caio Kalil Lauand, Univ. of Florida, USA.
Mustafa Mert Çelikok, TU Delft, NL.
Guillaume Pourcel, Univ. of Groningen, NL & Inria, France.
Reabetswe Nkhumise, Univ. of Sheffield, UK.

08:30 - 09:00: Registration
09:00 - 09:15: Welcome by Prof Ton de Kok, Director, CWI, and organizers
09:15 - 10:45: Lecture, Bert Kappen: Deterministic, discrete-time & continuous-time control
10:45 - 11:15: Break
11:15 - 13:00: Tutorial, Bert Kappen & TAs: Deterministic, discrete-time & continuous-time control
13:00 - 14:00: Lunch Break
14:00 - 15:30: Lecture, Bert Kappen: Stochastic optimal control
15:30 - 16:00: Break
16:00 - 17:45: Tutorial, Bert Kappen & TAs: Stochastic optimal control
17:45 - 18:30: Social

08:30 - 09:00: Walk-in coffee/tea
09:00 - 10:30: Lecture, Frans Oliehoek: Generalized policy iteration (value iteration & policy iteration), planning, partial observability, abstractions
10:30 - 11:00: Break
11:00 - 12:45: Tutorial, Frans Oliehoek & TAs: Generalized policy iteration, planning, partial observability, abstractions
12:45 - 14:00: Lunch Break and Posters
14:00 - 15:30: Lecture, Debabrota Basu: From linear to kernel methods to function approximation (LSPI, LSVI), Policy gradient, Actor-critic
15:30 - 16:00: Break
16:00 - 17:45: Tutorial, Debabrota Basu & TAs: From linear to kernel methods to function approximation (LSPI, LSVI), Policy gradient, Actor-critic
17:45 - 20:00: Games evening

08:30 - 09:00: Walk-in coffee/tea
09:00 - 10:30: Lecture, Sean Meyn: Stochastic Approximation
10:30 - 11:00: Break
11:00 - 12:45: Tutorial, Sean Meyn & TAs: Stochastic Approximation
12:45 - 14:00: Lunch Break and Posters
14:00 - 15:30: Lecture, Sean Meyn: Q-learning & TD-learning (advanced)
15:30 - 16:00: Break
16:00 - 17:45: Tutorial, Sean Meyn & TAs: Q-learning & TD-learning (advanced)
18:30 - 20:30: Dinner

08:30 - 09:00: Walk-in coffee/tea
09:00 - 10:30: Lecture, Bert Kappen: Path integral control
10:30 - 11:00: Break
11:00 - 12:45: Tutorial, Bert Kappen & TAs: Path integral control
12:45 - 14:00: Lunch Break and Posters
14:00 - 15:30: Lecture, Sean Meyn: Exploration
15:30 - 16:00: Break
16:00 - 17:30: Lecture, Maryam Kamgarpour: Safety in RL and control - state space approach

08:30 - 09:00: Walk-in coffee/tea
09:00 - 10:30: Lecture, Maryam Kamgarpour: Multi-agent games: Linear Quadratic games
10:30 - 11:00: Break
11:00 - 12:30: Lecture, Frans Oliehoek: Multi-agent RL / decentralized control
12:30 - 14:00: Lunch Break and Posters
14:00 - 15:30: Lecture, Debabrota Basu: Optimism (UCRL, PSRL, randomized VI)
15:30 - 16:00: Break
16:00 - 17:45: Tutorial, Sean Meyn, Debabrota Basu, Aditya Gilra & TAs: Bringing it all together & open questions
17:45 - 18:00: Closing remarks