Model order reduction for Bayesian approach to inverse problems
 NgocHien Nguyen^{1}Email author,
 Boo Cheong Khoo^{2} and
 Karen Willcox^{3}
DOI: 10.1186/2196116612
© Nguyen et al.; licensee Springer. 2014
Received: 9 October 2013
Accepted: 23 November 2013
Published: 29 April 2014
Abstract
This work presents an approach to solve inverse problems in the application of water quality management in reservoir systems. One such application is contaminant cleanup, which is challenging because tasks such as inferring the contaminant location and its distribution require large computational efforts and data storage requirements. In addition, real systems contain uncertain parameters such as wind velocity; these uncertainties must be accounted for in the inference problem. The approach developed here uses the combination of a reducedorder model and a Bayesian inference formulation to rapidly determine contaminant locations given sparse measurements of contaminant concentration. The system is modelled by the coupled NavierStokes equations and convectiondiffusion transport equations. The Galerkin finite element method provides an approximate numerical solutionthe ’full model’, which cannot be solved in realtime. The proper orthogonal decomposition and Galerkin projection technique are applied to obtain a reducedorder model that approximates the full model. The Bayesian formulation of the inverse problem is solved using a Markov chain Monte Carlo method for a variety of source locations in the domain. Numerical results show that applying the reducedorder model to the source inversion problem yields a speedup in computational time by a factor of approximately 32 with acceptable accuracy in comparison with the full model. Application of the inference strategy shows the potential effectiveness of this computational modeling approach for managing water quality.
Keywords
Bayesian Convectiondiffusion equation NavierStokes equations Markov chain Monte Carlo Inverse problem Proper orthogonal decomposition Reducedorder modelBackground
Hydrodynamic processes such as contaminant transport in lakes and reservoirs have a direct impact on water quality. The contaminants will appear, spread out, and decrease in concentration, etc. because of some processes such as convection, diffusion, time rate release of contaminants, and distance of travel. To simulate such processes, a coupled system of partial differential equations (PDEs) including the NavierStokes equations (NSEs) and contaminant transport equations needs to be solved. A better understanding of these processes is important in managing water resources effectively.
The direct or forward problems compute the distribution of contaminant directly from given input information such as contaminant location, contaminant properties, fluid flow properties, boundary conditions, initial conditions, etc. On the contrary, the inverse problems infer the unknown physical parameters, boundary conditions, initial conditions, or geometry given a set of measured data. These known data can be obtained experimental or computational. In realistic applications, data are not perfect because of error due to sensor noise. In addition, the model may contain some uncertain parameters such as wind velocity. Thus, inverse problems are generally stochastic problems.
A Bayesian inference approach to inverse problems is one way to deal with the stochastic problem. This approach takes into account both the information of the model parameters with uncertainty and the inaccuracy of data in terms of a probability distribution [1, 2]. The Bayesian approach provides a general framework for the formulation of wide variety of problems such as climate modeling [3], contaminant transport modeling [4–6], and heat transfer [7]. Under the Bayesian framework, simulated solutions need to be evaluated repeatedly over different samples of the input parameters. There are available sampling strategies associated with Bayesian computation such as Markov chain Monte Carlo (MCMC) methods [8–11]. However, if we use traditional PDE discretization methods, such as finite element or finite volume methods, the resulting numerical models describing the system will be very large and expensive to solve.
This paper presents an efficient computational approach to solve the statistical inverse problem. The approach uses the combination of a reducedorder model and a Bayesian inference formulation. The Galerkin finite element method [12] provides an approximate numerical solution  the ‘full model’ or ‘forward model’. The MCMC method is applied to solve the Bayesian inverse problems. In particular, we are interested in inferring an arbitrary source location in a timedependent convectiondiffusion transport equation, given a velocity field and a set of measured data of the concentration field at sparse spatial and temporal locations. We obtain the velocity field by solving the NavierStokes equations.
Solution of largescale inverse problems can be accelerated by applying model order reduction [13–15]. The idea is to project the largescale governing equations onto the subspace spanned by a reducedspace basis, yielding a loworder dynamical system. Proper orthogonal decomposition (POD) is the most popular method to find the reduced basis for a given set of data. The snapshot POD method, which was proposed by Sirovich [16], provide an efficient way for determining the POD basis vectors. Snapshots are solutions of a numerical simulation at selected times or choices of the parameters. The choice of snapshots should ensure that the resulting POD basis captures the most important characteristics of the system.
Since the convection term in contaminant transport equations is a velocitydependent term, we cannot apply the standard POD reduction framework. The evaluation of the velocitydependent convection term in the reduced system still depends on the full finite element dimension and has the same complexity as the fullorder system. Thus, a system with velocitydependent convection term needs an additional treatment to obtain an efficient reducedorder model.
This paper is outlined as follows. In the section ‘Methods’, we introduce the mathematical model, the finite element approximate technique, and the Bayesian inference formulation and its solution using an MCMC method. In section ‘Model order reduction’, we describe the model order reduction technique and the treatment of the velocitydependent convection term. In the section ‘Results and discussion’, we use the numerical example to demonstrate the solution of the statistical inverse problem and the reducedorder model performance. We provide some concluding remarks in the final section.
Methods
In this section, the mathematical model, finite element approximation technique, Bayesian inference formulation, and MCMC method are briefly introduced.
Transport model
where Re is the Reynolds number, p is the pressure field, and f_{ c } is the body force. So, for any given f and u, one can solve for the solution c(x,t;θ).
Finite element approximations
The finite element method [12] associated with the stabilized secondorder fractionalstep method is applied to solve the incompressible NavierStokes equations (5 to 8). The detailed discussion and derivation of this method is beyond the scope of this paper. For more details, please refer to [17, 18].
Here ϕ_{ i } with i=1,⋯,N is the finite element basis and $\theta \in {\mathbb{R}}^{d}$ is the coordinate vector of the source location.
Bayesian inference to inverse problems
Here, K is the number of time steps over which we collect the output data.
Markov chain Monte Carlo for posterior sampling
 1.
Initialize the chain θ ^{0} and set n=0
 2.
Repeat

n=n+1

Generate a proposal point θ^{∗}∼q(θ^{∗}θ)

Generate U from a uniform U(0,1) distribution

Update the state to θ^{n+1} as$\begin{array}{l}{\theta}^{n+1}=\left\{\begin{array}{cc}{\theta}^{\ast},& \text{if}\phantom{\rule{2.77626pt}{0ex}}\beta <\mathrm{U}\\ {\theta}^{n},& \text{otherwise}\end{array}\right.\end{array}$
 3.
Until n=N _{mcmc}→ stop.
N_{mcmc} is the total number of samples, π(θ) is the target distribution, q(θ^{∗}θ) is a proposal distribution, and U is a random number.
The discretized model in the form of ordinary differential equations (ODEs) (Equations 9 to 11) may have very large dimensions and be expensive to solve. The MCMC method requires evaluating repeatedly the solution of the forward model (many thousands or even millions of times). Hence, these simulations can be a computationally expensive undertaking. In such situations, the reducedorder model is needed to approximate the largescale model, which allows efficient simulations.
Model order reduction
This section presents the model order reduction framework. This includes the reduction via projection, the proper orthogonal decomposition method, the additional treatment for the velocitydependent convection term, and the error estimation.
Reduction via projection
The model reduction task is to find a suitable basis V so that m≪N and the reducedorder model yields accurate results. This study will consider POD as the method to compute the basis.
Proper orthogonal decomposition
POD provides a method to compute the reducedorder basis V and construct the loworder system. Here, we briefly describe the general POD method (more details may be found in [16]).
where ε_{ E }(%) is the required amount of energy, typically taken to be 99% or higher.
After obtaining the POD basis vectors for the contaminant, we have defined our reducedorder system. However, as mentioned earlier, the dimension of the system of Equations 24 to 26 is reduced only in state (concentration). The reduced convection matrix (C_{ r }(u)) in Equation 29 still depends on the full dimension of the velocity field as in Equation 14. This means that we have to recompute the full convection matrix (C(u)) at each timestep before projecting it onto the reducedspace basis to obtain the reduced convection matrix (C_{ r }(u)). Hence, we need an additional treatment to avoid this computational cost.
Linear expansion techniques for velocitydependent term
where N_{ u }≪T is the number of POD basis vectors use to represent the velocity.
Here, C_{ r m }(u_{ m }(x)) and C_{ r k }(Ψ_{ k }(x)),k=1,⋯,N_{ u } are the reducedorder forms of the fullorder convection matrices C(u_{ m }(x)) and C(Ψ_{ k }(x)), respectively. They are computed only once in the offline step. Thus, the first case of the reducedorder model (with Equation 29) is only reduced in state (concentration), but the second case (with Equation 38) is now reduced in both state (concentration) and parameter (velocity).
Error estimation
Here, c(t^{ k };θ),c_{ r }(t^{ k };θ),1≤k≤T are the full and reduced solutions, while y(t^{ k };θ) and y_{ r }(t^{ k };θ),1≤k≤T are the full and reduced outputs of interest, respectively.
Results and discussion
We consider a numerical example based on a 2D mathematical model. The velocity field in the reservoir is obtained by solving a 2D laterally averaged NavierStokes model. The Bayesian formulation of the inverse problem is then solved to determine an uncertain contaminant source location. We compare the effectiveness of solving the inverse problem using both the full model and reduced model techniques.
Model setup
A timedependent velocity field is obtained from the 2D laterally averaged system, which is given in Equations 5 to 8, where the body force f_{ c }= [f_{ c x }; f_{ c z }]^{ T }= [ 0; g]^{ T } with g the gravitational acceleration.
The velocity on remaining boundaries are set to zero. Here, V_{ a } is the wind speed at 10 m above the water surface. In this example, we assumed that V_{ a }=2 m/s for the entire simulation time. The Reynolds number is set to Re=5.0e^{5}, and a mixing length turbulence model is used [20]. The full model system is given in Equations 9 to 10. The CrankNicolson method [21] is used to discretize the system in time.
Here, to simplify the problem, we choose the number of sources to be n_{ s }=1, located at θ_{1}=(x_{ c },z_{ c }), with strength h_{1}=0.2 and width σ_{s 1}=0.05. The active time of the source is t_{01}∈ [ 0,t_{off}] with t_{off}=10. The inflow boundary and other solid boundaries satisfy a homogeneous Dirichlet condition on the contaminant concentration; the outflow boundaries and free surface boundary satisfy a homogeneous Neumann condition. The diffusivity coefficient is assumed to be constant, κ=0.005. Thus, the Péclet number $\text{Pe}=\frac{\parallel \mathbf{u}\parallel \mathrm{L}}{\kappa}=100$, where the length of the inflow section is used as the characteristic length L=0.5. The contaminant concentration is assumed to be zero at initial time t_{0}=0.
The outputs of interest are the values of contaminant solution c(x,t;θ) at selected sensor locations in the computational domain. These sensors are located on a 4×4 uniform grid covering the reservoir domain as shown in Figure 1b.
Model order reduction
To perform the model order reduction, we need to obtain the POD expansion of velocity field.
To generate the snapshots needed for the POD basis to represent the contaminant concentration, we choose S location samples in the computational domain. In this example, we generate random input locations $\theta \in \mathcal{D}\subset {\mathbb{R}}^{2}$ for source term f(x,t;θ).
The properties of various reducedorder models
N _{ s }  m  ε _{ROM1}  ε _{ROM2}  $\frac{{t}_{\text{full}}}{{t}_{\text{ROM1}}}$  $\frac{{t}_{\text{full}}}{{t}_{\text{ROM2}}}$ 

10  153  3.25e1  4.32e1  0.801  111 
20  220  9.81e2  9.74e2  0.799  56 
30  257  8.03e3  6.75e3  0.795  32 
40  284  7.53e3  7.08e3  0.785  26 
50  308  3.39e3  2.12e2  0.776  22 
60  328  5.83e3  6.08e3  0.767  19 
70  342  1.41e2  1.21e2  0.764  18 
80  355  9.09e3  8.21e3  0.763  15 
Figure 6b shows relative errors of ROMs for different numbers of POD basis vectors. With the energy taken to 99.99999%, the ROM resulted in 434 POD basis vectors with relative error around 4×10^{−3}. In the tradeoff between accuracy results and computational efforts (please refer to Table 1), we can choose our ROM with 99.999% of energy captured which resulting in the size of m=257 POD basis vectors and an acceptably small error.
Inverse problems
MCMC is now used to solve the inverse problem for a variety of source locations using the ROM2 solver above. By accounting for both measured noise and uncertain information in model parameters, Bayesian inference to inverse problem leads to a wellposed problem resulting in a posterior distribution of the unknown [1, 2]. Once obtaining the posterior distribution sample, all statistical questions related to the unknown could be answered with sample averages.
To represent the behavior of uncertain variables such as wind velocity, we generate noisy data at sensor locations. The ‘real’ data (y_{real}) is generated synthetically by adding noise to the ideal data (y_{ideal}) as in Equation 18. The noise is assumed to be Gaussian η∼N(0,σ^{2}I). Figure 8b shows the noisy data at sensor locations with σ=0.3.
We use the snapshots with S=30 samples above to generate several reducedorder models with different energies. To estimate the relative error of the inverse problem solutions (ε_{ θ }), we choose the solution corresponding to the largest order as a ‘truth’ solution. We do the MCMC simulation with the starting point θ_{ini}=(0.821; −0.955)^{ T }. The total number of MCMC samples is set to N_{mcmc}=5000. The initial burnin period is set to N_{burnin}=500. After this stage, data is saved to compute summary statistics of source locations.
Estimated inverse solutions for different numbers of POD basis
ε_{ E }(%)  m  θ _{1}  θ _{2}  ε _{ θ }  t_{mcmc}(s) 

99.0  61  0.788  0.595  5.31e1  1.75e+3 
90.9  111  0.484  0.491  6.95e2  2.74e+3 
99.99  178  0.451  0.471  1.05e2  7.55e+3 
99.999  257  0.443  0.467  2.42e3  8.71e+4 
99.9999  343  0.446  0.466  0  5.11e+5 
Estimated inverse solutions for different MCMC starting points
Initial  θ _{1ini}  θ _{2ini}  θ _{1−M}  θ _{2−M}  t_{mcmc}(s) 

Ini1  0.821  0.955  0.451  0.467  8.71e+4 
Ini2  0.322  0.451  0.458  0.463  4.53e+4 
Ini3  1.222  0.451  0.452  0.464  9.80e+4 
Ini4  0.323  1.313  0.456  0.465  1.05e+5 
Conclusion
This study has applied successfully the combination of a model order reduction technique based on the POD and a Bayesian inference approach to solve an inverse problem that seeks to identify an uncertain contaminant location. Applying an additional POD expansion to approximate the velocity results in a reducedorder model in both state (concentration) and parameter (velocity). The resulting reducedorder model is efficient for solution of the forward problem. Solution of the Bayesian formulation of the inverse problem using the reducedorder solver is much more rapid than using the full model, yielding the probability density of the source location in reasonable computational times. The computational time of using a reducedorder model with m=257 degrees of freedom is about a factor of 32 times lower than using the full model with size N=1,377. This reduction is important in realtime water quality management applications because it reduces time cost and storage requirements.
Endnote
^{1} The simulations were performed on a personal computer (PC) with processor Intel(R) Core(TM)2 Duo CPU E8200 @2.66GHz 2.66GHz, RAM 3.25GB, 32bit Operating System.
Declarations
Authors’ Affiliations
References
 Tarantola A: Inverse problem theory and methods for model parameter estimation. SIAM, Philadelphia. 2004.Google Scholar
 Sivia DS, Skilling J: Data analysis: a Bayesian tutorial, Second edition. Oxford University Press Inc., New York; 2006.Google Scholar
 Jackson C, Sen MK, Stoffa PL: An efficient stochastic Bayesian approach to optimal parameter and uncertainty estimation for climate model predictions. J Climate 2004, 17: 2828–2841. 10.1175/15200442(2004)017<2828:AESBAT>2.0.CO;2View ArticleGoogle Scholar
 Snodgrass MF, Kitanidis PK: A geostatistical approach to contaminant source identification. Water Resour Res 1997,34(4):537–546.View ArticleGoogle Scholar
 Woodbury AD, Ulrych TJ: A fullBayesian approach to the groundwater inverse problem for steady state flow. Water Resour Res 2000,36(1):159–171. 10.1029/1999WR900273View ArticleGoogle Scholar
 Marzouk YM, Najm HN, Rahn LA: Stochastic spectral methods for efficient Bayesian solution of inverse problems. J Comput Phys 2007,224(2):560–586. 10.1016/j.jcp.2006.10.010MathSciNetView ArticleGoogle Scholar
 Wang J, Zabaras N: Hierarchical Bayesian models for inverse problems in heat conduction. Inverse Probl 2005, 21: 183–206. 10.1088/02665611/21/1/012MathSciNetView ArticleGoogle Scholar
 Metropolis N, Ulam S: The Monte Carlo method. J Am Stat Assoc 1949, 44: 335–341. 10.1080/01621459.1949.10483310MathSciNetView ArticleGoogle Scholar
 Metropolis N, Rosenbluth AW, Rosenbluth MN, Teller AH, Teller E: Equations of state calculations by fast computing machines. J Chem Phys 1953, 21: 1087–1092. 10.1063/1.1699114View ArticleGoogle Scholar
 Hastings WK: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 1970, 57: 97–109. 10.1093/biomet/57.1.97View ArticleGoogle Scholar
 Gelman A, Carlin J, Stern H, Rubin D: Bayesian data analysis. Second Edition, Chapman &Hall/CRC Texts in Statistical Science; 2003.Google Scholar
 Zienkiewicz OC, Morgan K: Finite elements and approximation. Wiley, New York; 1983.Google Scholar
 Antoulas AC: Approximation of largescale dynamical systems. SIAM Advances in Design and Control, Philadelphia; 2005.View ArticleGoogle Scholar
 Lieberman C, Willcox K, Ghattas O: Parameter and state model reduction for largescale statistical inverse problems. SIAM J Sci Comput 2010,32(5):2523–2542. 10.1137/090775622MathSciNetView ArticleGoogle Scholar
 Lieberman C, Willcox K: Goaloriented inference: approach, linear theory, and application to advectiondiffusion. SIAM J Sci Comput 2012,34(4):1880–1904. 10.1137/110857763MathSciNetView ArticleGoogle Scholar
 Sirovich L: Turbulence and the dynamics of coherent structures. Part 1: coherent structures. Q Appl Math 1987,45(3):561–571.MathSciNetGoogle Scholar
 Codina R, Blasco J: Stabilized finite element method for the transient NavierStokes equations based on a pressure gradient projection. Comput Methods Appl Mech Engrg 2000, 182: 277–300. 10.1016/S00457825(99)001942MathSciNetView ArticleGoogle Scholar
 Ma X, Zabaras N: A stabilized stochastic finite element secondorder projection method for modeling natural convection in random porous media. J Comput Phys 2008, 227: 8448–8471. 10.1016/j.jcp.2008.06.008MathSciNetView ArticleGoogle Scholar
 Jaynes ET: Information theory and statistical mechanics. Phys Rev 1957,104(4):620–630.MathSciNetView ArticleGoogle Scholar
 Wilcox CD: Turbulence Modeling for CFD, DCW Industries, La Cañada, California. 1993.Google Scholar
 Crank J, Nicolson P: A practical method for numerical evaluation of solutions of partial differential equations of the heatconduction type. Adv Comput Math 1996,6(1):207–226. 10.1007/BF02127704MathSciNetView ArticleGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.