Non-exchangeable random partition models for microclustering

Journal article

Many popular random partition models, such as the Chinese restaurant process and its two-parameter extension, fall in the class of exchangeable random partitions, and have found wide applicability in various fields. While the exchangeability assumption is sensible in many cases, it implies that the size of the clusters necessarily grows linearly with the sample size, and such feature may be undesirable for some applications. We present here a flexible class of non-exchangeable random partition models which are able to generate partitions whose cluster sizes grow sublinearly with the sample size, and where the growth rate is controlled by one parameter. Along with this result, we provide the asymptotic behaviour of the number of clusters of a given size, and show that the model can exhibit a power-law behaviour, controlled by another parameter. The construction is based on completely random measures and a Poisson embedding of the random partition, and inference is performed using a Sequential Monte Carlo algorithm. Experiments on real datasets emphasise the usefulness of the approach compared to a two-parameter Chinese restaurant process.

Authors: Di Benedetto, G; Caron, F; Teh, YW

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Institute of Mathematical Statistics
• Journal: Annals of Statistics
• Volume: 49
• Issue: 4
• Pages: 1931-1957
• Publication date: 2021-09-29
• Acceptance date: 2020-07-09
• DOI: 10.1214/20-AOS2003
• ISSN: 0090-5364
• Language: English
• Keywords: FFR