content type videopublished November 6, 2023
Apple Workshop on Machine Learning for Health : Web3 and Decentralized AI (DecAI)
AuthorsRamesh Raskar (MIT)
Related readings and updates.
Metric-Dependent Annotation Saturation for Learning from Label Distributions
June 23, 2026research area Data Science and Annotation, research area Speech and Natural Language Processing
When annotators disagree on a label, the disagreement itself carries signal—and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation—whether the model identifies which items elicit…
Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
June 23, 2026research area Data Science and Annotation, research area Speech and Natural Language Processing
LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find…
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.