Beyond Bayesian model averaging over paths in probabilistic programs with stochastic support

Conference item

The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.

Authors: Reichelt, T; Ong, L; Rainforth, T

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Journal of Machine Learning Research
• Host title: Proceedings of The 27th International Conference on Artificial Intelligence and Statistics
• Pages: 829-837
• Series: Proceedings of Machine Learning Research
• Series number: 238
• Publication date: 2024-04-18
• Acceptance date: 2024-03-20
• Event title: 27th International Conference on Artificial Intelligence and Statistics (AISTATS 2024)
• Event location: Valencia, Spain
• Event website: https://aistats.org/aistats2024/
• Event start date: 2024-05-02
• Event end date: 2024-05-04
• EISSN: 2640-3498
• Language: English