Masterclass on Machine learning for inverse problems; A Bayesian perspective.
Inverse problems are ubiquitous in science and engineering. Traditionally, they are formulated using a (physics-based) forward model that simulates the relation between parameters and observations. Typically, these observations are incomplete and noisy. Moreover, the model is often over-parametrized and thus not all parameters can be inferred from the observations alone. Taking into account the nature of the measurement process and prior information on the parameters leads to a regularized data-fitting problem. Solving this variational problem gives an estimate of the parameters. A Bayesian interpretation of this formulation leads to a posterior distribution on the parameters, allowing one to define uncertainties of the estimates as well.
In recent years, data-driven methods have been proposed to replace this model-driven paradigm. In this masterclass, we review some results on Bayesian statistics and recent machine-learning approaches for solving inverse problems and estimating uncertainties. The masterclass is aimed to Phd students in mathematics, physics, earth sciences, working on inverse problems and who are interested in diving a little deeper into the Bayesian perspective and machine learning techniques. Registration Registration is now closed, but if you are interested we can put you on the waiting list in case additional spots open up. Please send an email to Tristan van Leeuwen to do so.
Program The event is scheduled to take place at the CWI in Amsterdam. More info on the schedule will follow soon.
Thursday May 12
09:00 - 09:15 Welcome and overview - Tristan van Leeuwen (CWI)
09:15 - 10:15 The Careful Bayesian - Peter Grünwald (CWI and Leiden University)
10:15 - 10:30 Break
10:30 - 13:00 Supervised learning for calibrating financial models - Kees Oosterlee (UU) and Balint Negyesi (TU Delft)
13:00 - 14:00 Lunch
14:00 - 16:30 Generative models for inverse problems and uncertainty quantification - Nikolaj Mücke (CWI)
16:30 - 17:30 Drinks
Friday May 13
09:00 - 09:30 Welcome and overview - Tristan van Leeuwen (CWI)
09:30 - 12:00 Machine learning for imaging - Daan Pelt (Leiden U.) and Dirk Schut (CWI)
12:00 - 13:30 Lunch
13:30 - 16:00 Learning physics from data - Syver Agdestein (CWI)
16:00 - 16:15 Closing remarks - Tristan van Leeuwen (CWI)
AbstractsThe Careful Bayesian - Peter Grünwald (CWI and Leiden University)
Bayesian inference is one of the main paradigms for statistics and machine learning. Yet it can behave badly if the model is misspecified (`wrong but useful'). We provide an example from our own work (G. and Van Ommen 2017): a simple regression problem in which the 'wrongness' of the model is seemingly innocuous yet the posterior fails to concentrate even for large samples, leading to extreme overfitting and under-estimation of uncertainty in practice. This problem goes away if we exponentiate the likelihood with the "right" "learning rate", which essentially amounts to making the prior more and the data less important - a generalization of Bayesian inference that is becoming more and more popular within machine learning. We also look at the case where the problem is only with the prior distribution on the parameters and not with the model itself. We review a recent real-world example on satellite collision probabilities (Balch et al. 2019) which shows that bad things may still happen.
Supervised learning for calibrating financial models - Kees Oosterlee (UU) andBalint Negyesi (TU Delft)
In this presentation, we will start with some basics, explaining supervised learning, the connection
to regression models and the single layer perceptron. The focus will then be towards "classical" fully connected neural networks.
We will discuss the individual method components and give reasons for some specific choices.
Our work in financial mathematics is related to financial option valuation, so we'll explain the setting and context in some detail after which the use of supervised learning and neural networks to value financial options is being explained. Supervised learning is learning with labels, so we will first explain how we can obtain reference prices and how the neural network can subsequently learn to price an option. This is an important building block towards financial model calibration, in which we learn the financial model parameters.
In a connected hands-on session, Dr. Shuaiqiang Liu will explain some of the basics of PyTorch regarding regression and fully connected neural networks. He will also explain in some detail how we can use neural networks to calibrate financial models in a robust and efficient way. This latter aspects is explained based on a high level Keras implementation.
Generative models for inverse problems and uncertainty quantification - Nikolaj Mucke (CWI)
This lecture consists of two parts. In part one, we discuss generative models and in part two their application to uncertainty quantification in inverse problems.
Generative models are models that aim to approximate a probability distribution from training data samples. When trained, one can use the generative model to generate new data that resembles the training data by sampling from the learned distribution. Such models have received a lot of attention due to great performance achieved when using deep neural networks in the field of image generation. For this reason, generative modeling has been adopted in many other fields of research such as astronomy, fluid dynamics, structural mechanics, finance, etc. In this talk, we are going to present the general framework of generative modeling and discuss various approaches based on deep learning. There will be a special focus on generative adversarial networks (GANs)
Inverse problems are notoriously difficult to solve due to problems such as noisy and sparse observations, ill-posedness, and nonlinear and high-dimensional forward maps. Furthermore, if one is not only interested in the estimate but also the uncertainty, the problem complexity increases even further due to the necessity of including distributional information. The Bayesian framework is typically utilized for solving such problems. However, since the posterior distribution is typically intractable to derive analytically, we must resort to numerical methods such as variational inference or Markov Chain Monte Carlo methods. However, such methods are often computationally expensive. To alleviate the drawbacks of conventional methods we will present how generative models can used in inverse problem solving. By using the distributional information learned in generative models, they are also a highly suitable tool for uncertainty quantification.
Learning physics from data - Syver Agdestein (CWI)
Dynamical systems governed by partial differential equations (PDE) can be expensive to simulate.
In this workshop, we will look at different tools to learn representations for the underlying physics using high-fidelity data.
Differentiable programming provides flexible ways to infer differential operators in different functional bases, both continuous
and discrete. Recent tools provides the modeler with the possibility of trying out different architectures for encouraging
conservation of physical properties and good predictions. We will look at
- One-dimensional PDEs with different terms and boundary conditions
- Different discretizations and projections
- Formulating parametrized reduced order models, replacing individual terms by simple operators or neural networks
- Minimizing prediction errors by using different loss functions, regularizations, optimizers, and backpropagation trough differentiable ODE solvers.
