research area Human-Computer Interaction, research area Methods and Algorithmsconference ICML
content type paperpublished June 2025

Aligning LLMs by Predicting Preferences from User Writing Samples

AuthorsStéphane Aroca-Ouellette‡†, Natalie Mackraz, Barry-John Theobald, Katherine Metcalf

View publication

Accommodating human preferences is essential for creating aligned LLM agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs acting as writing agents to infer a description of user preferences. Agent alignment then comes from conditioning on the inferred preference description. However, existing methods often produce generic preference descriptions that fail to capture the unique and individualized nature of human preferences. This paper introduces PROSE, a method designed to enhance the precision of preference descriptions inferred from user writing samples. PROSE incorporates two key elements: (1) iterative refinement of inferred preferences, and (2) verification of inferred preferences across multiple user writing samples. We evaluate PROSE with several LLMs (i.e., Qwen2.5 7B and 72B Instruct, GPT-mini, and GPT-4o) on a summarization and an email writing task. We find that PROSE more accurately infers nuanced human preferences, improving the quality of the writing agent's generations over CIPHER (a state-of-the-art method for inferring preferences) by 33%. Lastly, we demonstrate that ICL and PROSE are complementary methods, and combining them provides up to a 9% improvement over ICL alone.

  • ‡ Work done while at Apple
  • † University of Colorado Boulder

Related readings and updates.

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

October 23, 2024research area Data Science and Annotation, research area Methods and Algorithmsconference NeurIPS

The goal of aligning language models to human preferences requires data that reveal these preferences. Ideally, time and money can be spent carefully collecting and tailoring bespoke preference data to each downstream application. However, in practice, a select few publicly available preference datasets are often used to train reward models for reinforcement learning from human feedback (RLHF). While new preference datasets are being introduced…
Read more

Symbol Guided Hindsight Priors for Reward Learning from Human Preferences

December 1, 2022research area Human-Computer Interaction, research area Methods and Algorithmsconference NeurIPS

This paper was accepted at the "Human in the Loop Learning Workshop" at NeurIPS 2022. Specification of reward functions for Reinforcement Learning is a challenging task which is bypassed by the framework of Preference Based Learning methods which instead learn from preference labels on trajectory queries. These methods, however, still suffer from high requirements of preference labels and often would still achieve low reward recovery. We present…
Read more