New Method Turns High-Variability Data Into Parameter Distributions Without Inverting Models

Simulation models are only as good as their parameters, and a new method promises to estimate those parameters from sprawling, highly variable data sets at a fraction of the usual computational cost. In a paper published on 9 December 2024 in the SpringerNature journal Computational Science and Engineering, Vincent Wagner, Benjamin Castellaz, Lars Kaiser, Sebastian Höpfl and Nicole Radde of the University of Stuttgart introduce Eulerian Parameter Inference, or EPI, a technique that solves a stochastic inverse problem through a probabilistic change of variables. The method requires only data with large empirical variance and a deterministic simulation model that maps parameters to simulation results, and it never inverts the simulation model at all.

The problem EPI addresses is familiar to anyone who has tried to calibrate a model against thousands of measurements. The naive, point-wise approach searches for a set of parameters such that each simulation output resembles the corresponding data point. For a data set of n measurements, that means n separate inversions of the simulation function, which translate to numerical optimisation when the function cannot be inverted analytically. For large data sets, the authors write, this approach becomes prohibitively expensive. Their alternative treats the measurements as independent samples from an underlying probability distribution and asks for the parameter distribution that, when pushed through the model, reproduces that data distribution.

The mathematical core of the approach is a change-of-variables formula that traces back to Euler. If the simulation model is differentiable and injective, meaning it maps distinct parameters to distinct outputs, the parameter density at any point equals the data density evaluated at the model output multiplied by the absolute value of the model derivative at that point. Most intriguingly, as the authors put it, this probability density formulation essentially solves an inverse problem without ever inverting the simulation function; the associated probability density of one parameter point is obtained by only calculating forward simulations and respective model derivatives. Evaluating the parameter density at a single point costs roughly one forward simulation and one derivative evaluation.

The idea of using stochastic inverse problems for parameter inference dates to 1930, when Fisher described it in the context of fiducial inference, and it was later generalised by Hannig and colleagues. Butler and co-workers turned the idea into a practical measure-theoretic method, but that approach focuses on cases where the model maps from more parameter dimensions to fewer data dimensions, so that the inverse is set-valued and a statistician must choose a measure on the parameter space to pick one particular solution. The authors argue that choosing such a measure may not be the most transparent of assumptions. EPI complements this line of work in the opposite regime, where there are more data than parameter dimensions, and delivers a unique solution whenever the model is a smooth, injective mapping and the data lie within its image.

To generalise the formula to cases where the data dimension exceeds the parameter dimension, the authors assume the model can be extended to a smooth map between open neighbourhoods in the two spaces. The parameter density then involves the data density evaluated on the model’s image, a lower-dimensional manifold embedded in the data space, multiplied by a generalised Jacobian determinant, the square root of the determinant of the transposed Jacobian times the Jacobian. The data density on the manifold is obtained simply by evaluating the restriction of a kernel density estimate in the embedding space to the manifold. The Jacobian itself is computed with automatic differentiation, which, as Griewank showed in 1989, costs on average only about 1.5 times the effort of a single model evaluation, with a worst case factor of 5. Overall, the computational effort for evaluating the parameter density with EPI is on average only 2.5 times the cost of evaluating the model, independent of the parameter and data dimensions.

The authors demonstrate the method on three examples. The first uses an arithmetic climate model and real-world weather station data. Annual average temperatures from 3168 weather stations around the world, collected through the meteostat Python library, are described by a model in which temperature is proportional to the cosine of the latitude, calibrated with estimated temperatures of 30 degrees Celsius at the equator and minus 30 degrees near the poles. The data are interpreted as samples from an underlying temperature density, estimated with kernel density estimation using Gaussian kernels with bandwidths according to Silverman. Markov Chain Monte Carlo sampling then provides samples from the inferred latitude distribution, which accumulate in the range from 0 to 75 degrees with a maximum at about 50 degrees. A comparison with the true latitudes shows that they are similar to the inferred latitudes, and the model captures the relationship between temperature and latitude reasonably well.

The second example demonstrates real-world applicability with a differential equation model of COVID-19. The authors model infection data from the first wave in different districts of Germany over a 15-week period from March to June 2020, using incidence data from the Robert Koch Institute for 411 districts. A susceptible-exposed-infectious-removed model with three rate parameters describes how individuals move between compartments, with the population normalised to 1000 individuals and initial conditions of 999 susceptible, 0 exposed, 1 infectious and 0 removed. Incidence values after 1, 2, 5 and 15 weeks in each district form four-dimensional data vectors, so the model maps three parameters to four data dimensions. EPI returns samples of a complex, three-variate distribution with strongly correlated dimensions. The pushforward of the inferred parameter sample accurately resembles the data at all observed time points, and although the similarity is less satisfactory at an unseen tenth week, the inferred parameters can still predict those unseen data.

The third example pushes the method to a partial differential equation and to artificial data. Heat transport in a thin metal plate is described by the two-dimensional anisotropic heat equation, whose diffusion matrix contains three conductivity parameters. Because manufacturing tolerances and material differences may lead to differences in the heat conductivity tensor, a manufacturer might want to estimate the underlying distribution of that tensor from a finite sample of temperature measurements. The authors infer the three parameters from temperatures evaluated at five locations on each plate at a specified final time, giving five-dimensional data. To guarantee stochastically independent parameters, they generate artificial data by drawing 5000 parameter samples from shifted Beta distributions and simulating the corresponding temperatures. Under the assumption of independent parameters, the estimation of the full joint parameter distribution can be reduced to a series of one-dimensional problems, breaking the curse of dimensionality. Evaluating the marginalised formula on an equidistant grid of 100 parameter points per dimension requires only 300 model evaluations in total, and the inferred marginal distributions align closely with the true ones.

The method is implemented in the eulerpi Python package, an efficient, parallel implementation that harnesses automatic differentiation through JAX and just-in-time compilation. Users can choose between grid-based inference, which is advantageous in low dimensions, and a state-of-the-art affine-invariant ensemble sampler from the emcee package, which directly provides parameter samples and the pushforward along the way. The package offers native support for Systems Biology Markup Language models, robust testing, active development and availability on the Python Package Index, and all figures in the article can be reproduced with the published model and plotting files.

The authors are candid about limitations. The solved stochastic inverse problem has a unique solution only if the simulation model is a diffeomorphism, meaning it must be differentiable with respect to the parameters and injectively map from the parameter space to the image space. Both properties are not always easy to assess and might be violated. If the model is non-injective, multiple parameter distributions represent the data equally well, and EPI’s solution locally overestimates the parameter and data density at points of non-injectivity, a correction the authors describe as an ongoing effort in their research group. They expect differentiability to be the assumption more likely violated in practice, for example if bifurcations occur in dynamic models, and they note that extremely ill-conditioned problems such as chaotic maps amplify even the most minor density approximation errors and prevent meaningful inference. Density estimation also becomes increasingly challenging in higher dimensions, and the authors suggest that future research could investigate methods better suited for high dimensions than kernel density estimation. They also distinguish EPI from Bayesian approaches, which require a stochastic embedding and a measurement error model to define a likelihood function; EPI’s injectivity assumption prevents the use of explicit error models, and measurement errors instead express themselves as a broader parameter distribution. Where the inherent data variability originates from fluctuations of underlying model parameters, the authors argue, it is desirable to extract that information during the parameter estimation process, and EPI preserves the information encoded in the variability of modern, large-scale data sets.

Subject of Research: A probabilistic change-of-variables method for inferring model parameter distributions from high-variability data

Article Title: Eulerian Parameter Inference: a probabilistic change of variables for model-based inference with high-variability data sets

Article References: Wagner, V., Castellaz, B., Kaiser, L., Höpfl, S., & Radde, N. (2024). Eulerian Parameter Inference: a probabilistic change of variables for model-based inference with high-variability data sets. Computational Science and Engineering, 1(1), Article 4. https://doi.org/10.1007/s44207-024-00004-x

Image Credits: AI Generated

DOI: 10.1007/s44207-024-00004-x

Keywords: Eulerian Parameter Inference, stochastic inverse problem, change of variables, parameter inference, high-variability data, kernel density estimation, automatic differentiation, SEIR model, COVID-19, heat equation, Python package, simulation models