content type videopublished July 20, 2023
research area Speech and Natural Language Processing

NLU Workshop Talk: Model-Aided Human Annotation at Scale

AuthorsHadas Kotek

Related readings and updates.

Theoretical Limits of Language Model Alignment

November 12, 2026research area Fairness, research area Methods and Algorithmsconference NeurIPS

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NN alignment, which selects the highest-reward output among NN independent samples. Despite their widespread use, the fundamental limits of reward…

Read more

Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment

November 10, 2026research area Human-Computer Interaction, research area Methods and AlgorithmsWorkshop at NeurIPS

This paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026.

We evaluate how good large language model (LLM) judges are for personalized writing tasks. This is important when a user’s preferred writing style is known (e.g., “upbeat”) and an LLM judge is used to evaluate whether generated text adheres to this preference. We examine how the writing task, the combination of generator and judge LLM, the evaluation objective,…

Read more