The European Laboratory for Learning and Intelligent Systems (ELLIS) recently added The University of Manchester as a partner of its global members who strive towards a meaningful contribution to securing Europe’s sovereignty and leadership in the research field of modern artificial intelligence (AI).
Manchester’s ELLIS unit brings together experts in AI fundamentals with experts in the application of AI in other fields. Their research focus will be probabilistic modelling and Bayesian inference, AI technologies that work better with people, ML for digital health and medicine, and privacy-preserving ML.
ELLIS Manchester is hosting a seminar from Wenlong Chen and Yingzhen Li from Imperial College London. Their talk talk is titled 'Calibrating transformers via sparse Gaussian processes'.
Abstract:
Transformer models have become standard building blocks for popular machine learning applications such as large-scale generative models including large language models (LLMs). Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It draws connection between dot-product based attentions with sparse Gaussian processes (SGP), which are then used to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection. Future works include adapt SGPA to auto-regressive prediction and improve its scalability in order to apply it to LLM
Dr Yingzhen Li is a Senior Lecturer in Machine Learning at Imperial College London, UK. Before that she worked at Microsoft Research Cambridge and Disney Research. She received her PhD from the University of Cambridge. Yingzhen is passionate about building reliable machine learning systems with probabilistic methods, and her published work has been applied in industrial systems and implemented in popular deep learning frameworks. She is a regularly invited speaker at international machine learning conferences and summer schools, and she gave an invited tutorial on approximate inference at NeurIPS 2020. Her work on Bayesian ML has also been recognised in AAAI 2023 New Faculty Highlights. She has co-organised many international research workshops on probabilistic inference and deep generative models. She regularly serves as Area Chair for ICML, ICLR and NeurIPS, and currently she is a Program Chair for AISTATS 2024. When not at work, Yingzhen enjoys reading, hiking, video games, and following news on latest technology developments.
Wenlong Chen is a PhD student in Machine Learning at Imperial College London, UK. Before that, he obtained his MPhil degree in Machine Learning and Machine Intelligence from the University of Cambridge. Wenlong mainly studies uncertainty estimation and Bayesian deep learning. Meanwhile, he also works on generative models and responsible AI.
