Events at The University of Manchester
  • University home
  • Events
  • Home
  • Exhibitions
  • Conferences
  • Lectures and seminars
  • Performances
  • Events for prospective students
  • Sustainability events
  • Family events
  • All Events

ELLIS Manchester | Wenlong Chen and Yingzhen Li: Calibrating transformers via sparse Gaussian processes

image
Dates:14 February 2024
Times:13:00 - 14:00
What is it:Seminar
Organiser:Faculty of Science and Engineering
Who is it for:University staff, External researchers, Current University students
Speaker:Wenlong Chen, Yingzhen Li
See travel and contact information
Add to your calendar

More information

  • ELLIS Manchester

Other events

  • In category "Seminar"
  • In group "(DF) Data Science and AI"
  • By Faculty of Science and Engineering

The European Laboratory for Learning and Intelligent Systems (ELLIS) recently added The University of Manchester as a partner of its global members who strive towards a meaningful contribution to securing Europe’s sovereignty and leadership in the research field of modern artificial intelligence (AI).

Manchester’s ELLIS unit brings together experts in AI fundamentals with experts in the application of AI in other fields. Their research focus will be probabilistic modelling and Bayesian inference, AI technologies that work better with people, ML for digital health and medicine, and privacy-preserving ML.

ELLIS Manchester is hosting a seminar from Wenlong Chen and Yingzhen Li from Imperial College London. Their talk talk is titled 'Calibrating transformers via sparse Gaussian processes'.

Abstract:

Transformer models have become standard building blocks for popular machine learning applications such as large-scale generative models including large language models (LLMs). Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It draws connection between dot-product based attentions with sparse Gaussian processes (SGP), which are then used to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection. Future works include adapt SGPA to auto-regressive prediction and improve its scalability in order to apply it to LLM

Dr Yingzhen Li is a Senior Lecturer in Machine Learning at Imperial College London, UK. Before that she worked at Microsoft Research Cambridge and Disney Research. She received her PhD from the University of Cambridge. Yingzhen is passionate about building reliable machine learning systems with probabilistic methods, and her published work has been applied in industrial systems and implemented in popular deep learning frameworks. She is a regularly invited speaker at international machine learning conferences and summer schools, and she gave an invited tutorial on approximate inference at NeurIPS 2020. Her work on Bayesian ML has also been recognised in AAAI 2023 New Faculty Highlights. She has co-organised many international research workshops on probabilistic inference and deep generative models. She regularly serves as Area Chair for ICML, ICLR and NeurIPS, and currently she is a Program Chair for AISTATS 2024. When not at work, Yingzhen enjoys reading, hiking, video games, and following news on latest technology developments.

Wenlong Chen is a PhD student in Machine Learning at Imperial College London, UK. Before that, he obtained his MPhil degree in Machine Learning and Machine Intelligence from the University of Cambridge. Wenlong mainly studies uncertainty estimation and Bayesian deep learning. Meanwhile, he also works on generative models and responsible AI.

Speakers

Wenlong Chen

Organisation: Imperial College London

Yingzhen Li

Organisation: Imperial College London

Travel and Contact Information

Find event

Emmeline Suite
Christabel Pankhurst Building
Dover Street
Manchester

Contact event

ELLIS Manchester

ellis@manchester.ac.uk

Contact us

  • +44 (0) 161 306 6000

Find us

The University of Manchester
Oxford Rd
Manchester
M13 9PL
UK

Connect with the University

  • Facebook page for The University of Manchester
  • X (formerly Twitter) page for The University of Manchester
  • YouTube page for The University of Manchester
  • Instagram page for The University of Manchester
  • TikTok page for The University of Manchester
  • LinkedIn page for The University of Manchester

  • Privacy /
  • Copyright notice /
  • Accessibility /
  • Freedom of information /
  • Charitable status /
  • Royal Charter Number: RC000797
  • Close menu
  • Home
    • Featured events
    • Today's events
    • The Whitworth events
    • Manchester Museum events
    • Jodrell Bank Discovery Centre events
    • Martin Harris Centre events
    • The John Rylands Library events
    • Exhibitions
    • Conferences
    • Lectures and seminars
    • Performances
    • Events for prospective students
    • Sustainability events
    • Family events
    • All events