Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles
Authors: Eric Slyman, Mehrab Tanjim, Kushal Kafle, Stefan Lee
Deep-Dive Summary:
Original Abstract: Multimodal large language models (MLLMs) are increasingly used to evaluate
text-to-image (TTI) generation systems, providing automated judgments based on
visual and textual context. However, these “judge” models often suffer from
biases, overconfidence, and inconsistent performance across diverse image
domains. While prompt ensembling has shown promise for mitigating these issues
in unimodal, text-only settings, our experiments reveal that standard
ensembling methods fail to generalize effectively for TTI tasks. To address
these limitations, we propose a new multimodal-aware method called Multimodal
Mixture-of-Bayesian Prompt Ensembles (MMB). Our method uses a Bayesian prompt
ensemble approach augmented by image clustering, allowing the judge to
dynamically assign prompt weights based on the visual characteristics of each
sample. We show that MMB improves accuracy in pairwise preference judgments and
greatly enhances calibration, making it easier to gauge the judge’s true
uncertainty. In evaluations on two TTI benchmarks, HPSv2 and MJBench, MMB
outperforms existing baselines in alignment with human annotations and
calibration across varied image content. Our findings highlight the importance
of multimodal-specific strategies for judge calibration and suggest a promising
path forward for reliable large-scale TTI evaluation.
PDF Link: 2509.08777v1
部分平台可能图片显示异常，请以我的博客内容为准
