Message Passing-based Inference for Gaussian Processes

Gaussian Processes (GPs) have long been a powerful toolkit in the development of machine learning algorithms. GPs provide a nonparametric approach to Bayesian inference that requires few model-specific structure assumptions from human experts. Applications of GPs range from regression and classification to time-series forecasting, spatial modeling, and active learning. Their ability to provide uncertainty estimates makes them especially attractive in domains where safety, interpretability, or data efficiency are crucial, such as robotics, climate modeling, and healthcare. However, the adoption of GPs is hindered in practice by the technical complexity of their inference procedures. Although various sparse approximations alleviate the cubic complexity of full GP inference, practitioners often still need to manually derive model-specific update rules, especially when embedding GPs into larger, more expressive Bayesian models. While recent software developments have improved the accessibility of Gaussian processes, their application in large-scale settings or more complex formulations remains challenging and often requires specialized expertise. To overcome this, two essential features are needed: inference automation and composability. Inference automation allows practitioners to define flexible probabilistic models without manually computing posterior approximations. Composability enables models to be assembled from smaller, reusable components or modules. This aligns with modern trends in probabilistic programming, where model development is often iterative, modular, and collaborative. Factor graphs, in particular Forney-style factor graphs (FFGs), naturally offer both of these features. FFGs represent factorized probabilistic models as networks of nodes (the "factors"), where each node encapsulates a specific component of the joint distribution as a well-defined computational unit. Inference is performed via message passing, in which localized computations propagate along edges to approximate variational posteriors. Importantly, the message-passing rules can be precomputed and stored for each node type, enabling efficient and fully automated inference once the graph is defined. This design philosophy is similar to how neural network layers are reused in deep learning. Unfortunately, GPs have not been naturally developed to fit within this composable framework. While inference in GPs can be expressed in terms of finite-dimensional marginals over observed data, their tightly coupled covariance structure makes it difficult to represent them as finite, self-contained nodes within a factor graph. This dissertation addresses that gap by proposing a set of contributions that integrate GPs, including their sparse and structured variants, into the FFG framework, thereby unlocking the modularity and reusability that have long been standard for parametric models.