To address these challenges, we wanted to call attention to a [latent text diffusion model](/research/planner) that we introduced in the fall of 2023. The model synergizes non-autoregressive latent semantic diffusion with autoregressive generation to overcome the hurdles faced by its predecessors. Specifically, we hope to conduct research to improve the experience of users who benefit from more diversified and controlled text generation. By adopting a latent diffusion approach (as discussed in [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2212.09462) and [Latent Diffusion for Language Generation](https://arxiv.org/abs/2212.09462), PLANNER mitigates computational expenses typically associated with similar models, while simultaneously delivering superior diversity and cohesiveness, and reduce the repetition level of generated text, particularly in longer blocks of text and paragraphs, which have traditionally posed a challenge for text generation models. 

Our model, PLANNER, extends its benefit to various text generation tasks such as semantic generation, text completion, and summarization, with extensive evaluations of fluency, diversity, and repetition mitigation. 

<Figure id="figure1" 
caption={<>Figure 1: A three-stage model for text generation. We begin with a variational paragraph embedder in stage 1 and evolve the coarse text through our latent diffusion model, PLANNER, for a finer coherent result in stage 3.</>}><Figure1 title="A two-part model for text generation." theme="dark-gray" />
</Figure>

In stage 1 of [Figure 1](#figure1), a variational paragraph embedder encodes paragraphs into a series of latent codes. The encoder E and decoder D construct a bidirectional mapping between the discrete data space and the latent code space. The paragraph embeddings z are extracted by taking the first k hidden state vectors of dimension h from the final layer of E, which are fed into the initial steps of the decoder, which is trained to reconstruct the original text x. BOS and EOS represent “beginning of sentence” and “end of sentence” tokens, respectively.

In stage 2 of [Figure 1](#figure1), these latent codes z are processed by a transformer-based latent diffusion model (as discussed in the work [Scalable Diffusion Models with Transformers](https://arxiv.org/abs/2212.09748)) for training, so that it can generate new latent codes over time during inference time, simulating the evolution of text from coarse to fine. Finally, in stage 3 the decoder D translates these evolving latent codes into coherent text.

Our PLANNER latent diffusion model considers the conditioning signal as raw text, such as preceding context or the document to be summarized. We applied a conditional feature encoder τ to the input and used the hidden states at the last layer as y. We fed y and the time embedding t into the latent diffusion model through two channels, namely cross-attention and adaptive layer normalization. The aim of our research is to use existing text samples, such as an email or a summary of a document, to help generate longer texts that are both cohesive and readable. Examples in the following two figures are taken from a public dataset of text samples related to hotel reviews. 
 
<Figure id="figure2" theme="dark-gray"
caption={<>Figure 2: Compare the fine-tuned GPT-2 large model (the most relevant model at the time of research) results in column at the left with the PLANNER results at the right when generating text from a repetitive prompt (shown as “Prefix" in the figure). On the left, the GPT-2 model, despite using top-p sampling, still yields text with self-reinforced repetition. On the right, data from 512 generation roll-outs illustrate that the new method produces a wider variety of first 1-grams, showcasing its ability to generate more diversified text unaffected by the poorly devised prompt.</>}>
<Figure2 title="compares two language models" theme="dark-gray"/>
</Figure>
 
[Figure 2](#figure2) compares two language models: a fine-tuned GPT-2 large model and our method. It showcases how each model handles a prompt designed to evaluate their ability to generate diversified text from a repetitive cue. We decided to select GPT-2 because it was the most relevant model at the time of conducting research. Starting with the fine-tuned GPT-2 large model, this model has been initialized using GPT-2 large, which has 774 million parameters. As for publicly available versions of GPT-2, OpenAI has released different sizes of GPT-2 models, including a large version that is accessible for researchers and developers. However, the particular fine-tuned version we used in our paper, [PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model](/research/planner), may include proprietary dataset adjustments and may not be directly available.

- _FT_ stands for fine-tuning, which is the process of taking a pre-trained model and training it further on a new dataset to specialize its knowledge.
- _Greedy decoding_ is a method where, at each step in generating text, the model picks the word with the highest probability.
- _Top-p sampling_ is a technique where the model chooses from the top p percent of probable words, allowing for more randomness and potential creativity in its output, as addressed in the work [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751) 
- _512 generation rollouts_ refers to the number of times the model generates text to test its capabilities. In this context, it means the model was used to generate text, starting from the prompt, 512 times for evaluation.
- _N-grams_ are sequences of N tokens.

The percentage numbers in the n-gram columns indicate the frequency of each n-gram’s appearance within the generated text by a specific method. A lower maximum percentage suggests that there is a larger variety of different n-grams, which is typically seen as desirable for the generation of text that is less repetitive and more diverse.

"More diversified" implies that the generated sequences of words (n-grams) are more varied and less repetitive compared to the repetitive n-grams generated by other methods or models. This diversification generally indicates a higher quality of text generation that is more likely to generate useful and novel content for users.

Lastly, we observed accumulative errors in traditional autoregressive models, such as the ones in GPT-2, where the model gets stuck in a loop and produces repetitive or unhelpful output. In the context given, the repeated phrase “awful hotel” in the generated text from GPT-2 is an example of such an accumulative error. 

<Figure id="figure3" caption={<>Figure 3: This hotel review text generated by a diffusion model progresses over 10 steps, from a vague to a more distinct and richly detailed positive sentiment about the hotel experience. This development follows a coarse-to-fine approach, starting from general commendation and culminating in a vibrant and specific final review that praises the bartender and the establishment’s ambiance and amenities. </>}>
<Figure3 />
</Figure>

[Figure 3](#figure3) illustrates the gradual evolution of generated text over a series of 10 steps. The model begins with coarse initial predictions (represented in Figure 3 as step 1, the initial state) and progresses by performing repeated processing steps to denoise and improve the text.

The reader should envision this scenario not as a snapshot of text being entered or prompted by an iPhone user but as a systematic process by which a language model refines an initially vague or broad expression into a more detailed and specific review text. At step 1, the text is a rough suggestion of what the user might want to express — it is terse and lacks detail. As time progresses, the model fine-tunes the text, introducing more specific descriptions, sentiment, and sophisticated language. By step 10, the end state, the generated text resembles a thoughtfully composed review that one might expect from an experienced reviewer who gives particular attention to various aspects of their hotel stay.

Thus, [Figure 3](#figure3) shows how the PLANNER model's generation progresses from coarse to fine, giving readers a step-by-step visualization of how the text is iteratively enhanced to improve readability, specificity, and overall quality. The scenario starts with a minimal outline of positive sentiment and, over time, develops into a fleshed-out testimonial with vivid details emerging at each subsequent step.

## Conclusion 

The PLANNER model represents an advancement in the pursuit of improved natural language. Tackling the challenge of accumulative errors in traditional autoregressive models, our model leverages latent semantic diffusion to generate text that's fluent, controlled, and diversified.

## Acknowledgments 

Many people contributed to this work, including Richard Bai, Ronan Collobert, Zhe Gan, David Grangier, Edouard Grave, Tatiana Likhomanenko, Barry Theobald, Yinfei Yang, and Yizhe Zhang.

## Apple Resources 

Xu, Jin, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. “Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation.” [[link.]](/research/analyzing-mitigating-repetitions)

Zhang, Yizhe, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. 2023. “PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model.” [[link.]](/research/planner)

## External References 

Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." [[link.]](https://arxiv.org/abs/1506.03099)

Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” [[link.]](https://arxiv.org/abs/1904.09751)

Hu, Zhiting, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. “Toward Controlled Generation of Text.” [[link.]](https://arxiv.org/abs/1703.00955)

Keskar, Nitish Shirish, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. “CTRL: A Conditional Transformer Language Model for Controllable Generation.” [[link.]](https://arxiv.org/abs/1909.05858)

Lovelace, Justin, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q. Weinberger. 2023. “Latent Diffusion for Language Generation." [link.]](https://doi.org/10.48550/arXiv.2212.09462)

Peebles, William, and Saining Xie. 2022. “Scalable Diffusion Models with Transformers.” [[link.]](https://arxiv.org/abs/2212.09748) 

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” [[link.]](https://arxiv.org/abs/2112.10752)

In the fast-evolving world of natural language processing (NLP), there is a strong demand for generating coherent and controlled text, as referenced in the work <a href="https://arxiv.org/abs/1703.00955" target="_blank" aria-label="Toward Controlled Generation of Text. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Toward Controlled Generation of Text.</a> Traditional autoregressive models such as GPT, which have long been the industry standard, possess inherent limitations that sometimes manifest as repetitive and low-quality outputs, as seen in the work <a href="https://arxiv.org/abs/1904.09751" target="_blank" aria-label="The Curious Case of Neural Text Degeneration. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">The Curious Case of Neural Text Degeneration.</a> This is primarily due to a phenomenon known as "exposure bias," as seen in the work <a href="https://arxiv.org/abs/1506.03099" target="_blank" aria-label="Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.</a> This imperfection arises due to a mismatch between how these models are trained and their actual use during inference, often leading to error accumulation during text generation.

 Enhancing Paragraph Generation with a Latent Language Diffusion Model

Recent research has explored <a href="https://pubmed.ncbi.nlm.nih.gov/32706685/" target="_blank" aria-label="clinical monitoring, - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">clinical monitoring,</a> <a href="https://arxiv.org/abs/1802.02511" target="_blank" aria-label="cardiovascular events, - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">cardiovascular events,</a> and even clinical lab values from <a href="https://pubmed.ncbi.nlm.nih.gov/34031607/" target="_blank" aria-label="wearables data. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">wearables data.</a> As adoption increases, wearables data may become crucial in public health applications like <a href="https://www.thelancet.com/journals/landig/article/PIIS2589-7500(19)30222-5/fulltext" target="_blank" aria-label="disease monitoring - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">disease monitoring</a> and the <a href="https://pubmed.ncbi.nlm.nih.gov/34711377/" target="_blank" aria-label="design of epidemiological studies. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">design of epidemiological studies.</a>

Personalizing Health and Fitness with Hybrid Modeling

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events at ICASSP 2024. Stop by the Apple booth from April 16 to 19 from 8:20 AM to 6:00 PM UTC at Booth D1 in the COEX Convention Center Exhibition Hall.

### Monday, April&nbsp;15

<ul class="links-stacked">
<li>POSTER</li>
<li><a href=https://machinelearning.apple.com/research/world-english-language-model>Towards a World-English Language Model</a></li>
<li>1:30 PM - 3:30 PM UTC, Poster Zone 1B</li>
<li>Rricha Jalota, Lyan Verwimp, Markus Nussbaum-Thom, Amr Mousa, Arturo Argueta, Youssef Oualil</li>
</ul>

<ul class="links-stacked">
<li>POSTER PRESENTATION</li>
 <li><a href=https://sites.google.com/view/hscma2024>Workshop on Hands Free Communication and Microphone Arrays (HSCMA)</a></li>
 <li>4:00 PM UTC, Room 205</li>
 <li><a href=/research/multichannel-voice-trigger>
Multichannel Voice Trigger Detection based on Transform-average-concatenate</a></li>
<li>Takuya Higuchi, Avamarie Brueggeman (The University of Texas at Dallas), Masood Delfarah, Stephen Shum</li>
</ul>


### Wednesday, April&nbsp;17
<ul class="links-stacked">
<li>LUNCHEON</li>
<li>Women in Signal Processing (WiSP)</a></li>
<li>11:40 AM - 1:40 PM UTC, Room 402</li>
<li>Panos Georgiou and Clara Borrelli will be representing Apple at the Women in Signal Processing Luncheon.</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/leveraging-large-language-models>Leveraging Large Language Models for Exploiting ASR Uncertainty</a></li>
 <li>1:10 PM - 3:10 PM UTC, Poster Zone 1A</li>
<li>Pranay Dighe, Yi Su, Daniel Zheng, Yunshu Liu, Vineet Garg, Xiaochuan Niu, Ahmed Tewfik</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/llm-device-directed-speech-detection>A Multimodal Approach to Device-Directed Speech Detection with Large Language Models</a></li>
 <li>1:10 PM - 3:10 PM UTC, Poster Zone 1B</li>
<li>Dominik Wagner (FAU), Alex Churchill, Siddharth Sigtia, Panos Georgiou, Matt Mirsamadi, Aarshee Mishra, Erik Marchi</li>
</ul>

<ul class="links-stacked">
<li>ORAL PRESENTATION</li>
 <li><a href=https://machinelearning.apple.com/research/ctc-based>Personalization of CTC-based End-to-End Speech Recognition Using Pronunciation-Driven Subword Tokenization</a></li>
<li>1:10 PM - 1:30 PM UTC, Room 102</li>
<li>Zhihong Lei, Ernie Pusateri, Michael Han, Leo Liu, Mingbin Xu, Tim Ng, Zhen Huang, Ruchir Travadi, Darien Zhang, Mirko Hannemann, Man-Hung Siu</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/resource-constrained>Resource-constrained stereo singing voice cancellation</a></li>
 <li>1:30 PM - 3:30 PM UTC, Poster Zone 4A</li>
<li>Clara Borrelli, Dogac Basaran, Matthias Mauch , Matthew McVicar, James Rae, Mehrez Souden</li>
</ul>



### Thursday, April&nbsp;18

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/corpus-synthesis>Corpus Synthesis for Zero-shot ASR Domain Adaptation using Large Language Models</a></li>
 <li>8:20 - 10:20 AM UTC, Poster Zone 1A</li>
<li>Hsuan Su (National Taiwan University), Ting-Yao Hu, Hema Koppula, Raviteja Vemulapalli, Rick Chang, Karren Yang, Gautam Varma Mantena, Oncel Tuzel</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/modality-dropout>Modality Drop-out for Multimodal Device Directed Speech Detection Using Verbal and Non-Verbal Features</a></li>
 <li>8:30 - 10:30 AM UTC, Poster Zone 5C</li>
<li>Gautam Krishna, Sameer Dharur, Oggi Rudovic, Pranay Dighe, Saurabh Adya, Ahmed Hussen Abdelaziz, Ahmed Tewfik</li>
</ul>

<ul class="links-stacked">
<li>JOB FAIR</li>
<li>Student and Young Professionals Luncheon</a></li>
<li>12:00 - 2:00 PM UTC, Room E5 - E6</li>
<li>Kisun You, Evan Yamasaki, and Alex Acero will be representing Apple at the Student Job Fair and Luncheon.</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/investigating-salient-representation>Investigating Salient Representations and Label Varience in Dimensional Speech Emotion Analysis</a></li>
 <li>1:10 - 3:10 PM UTC, Poster Zone 5A</li>
<li>Vikramjit Mitra, Jingping Nie, Erdrin Azemi</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/flexible-keyword>Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding</a></li>
 <li>4:30 - 6:30 PM UTC, Poster Zone 1B</li>
<li>Kumari Nishu, Minsik Cho, Paul Dixon, Devang Naik</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=https://machinelearning.apple.com/research/vision-inspired>Improving vision-inspired keyword spotting using a streaming conformer encoder with input-dependent dynamic depth</a></li>
 <li>4:30 - 6:30 PM UTC, Poster Zone 3C</li>
<li>Alexandre Bittar (Ecole Polytechnique Fédérale de Lausanne, Switzerland), Paul Dixon, Mohammad Samragh Razlighi, Kumari Nishu, Devang Naik</li>
</ul>

### Friday, April&nbsp;19

<ul class="links-stacked">
<li>ORAL PRESENTATION</li>
 <li><a href=https://machinelearning.apple.com/research/streaming-anchor-loss>Streaming Anchor Loss: Augmenting Supervision with Temporal Significance</a></li>
 <li>1:10 - 1:30 PM UTC, Room 103</li>
<li>Oggy Sarawgi, Jack Berkowitz, Vineet Garg, Arnav Kundu, Minsik Cho, Sai Srujana Buddi, Saurabh Adya, Ahmed Tewfik</li>
</ul>

<ul class="links-stacked">
<li>ORAL PRESENTATION</li>
 <li><a href=https://machinelearning.apple.com/research>Dialog modeling in audiobook synthesis</a></li>
 <li>2:10 - 2:30 PM UTC, Room 104</li>
<li>Cheng-Chieh Yeh, Reza Shirani, Weicheng Zhang, Tuomo Raitio, Ramya Rasipuram, Ladan Golipour, David Winarsky</li>
</ul>

</div>

## Accepted Papers

[Corpus Synthesis for Zero-shot ASR Domain Adaptation using Large Language Models](https://machinelearning.apple.com/research/corpus-synthesis)

*Hsuan Su (National Taiwan University), Ting-Yao Hu, Hema Koppula, Raviteja Vemulapalli, Rick Chang, Karren Yang, Gautam Varma Mantena, Oncel Tuzel*
 
[Dialog modeling in audiobook synthesis](https://machinelearning.apple.com/research)

*Cheng-Chieh Yeh, Reza Shirani, Weicheng Zhang, Tuomo Raitio, Ramya Rasipuram, Ladan Golipour, David Winarsky*

[Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding](https://machinelearning.apple.com/research/flexible-keyword)

*Kumari Nishu, Minsik Cho, Paul Dixon, Devang Naik*

[Improving vision-inspired keyword spotting using a streaming conformer encoder with input-dependent dynamic depth](https://machinelearning.apple.com/research/vision-inspired)

*Alexandre Bittar (Ecole Polytechnique Fédérale de Lausanne, Switzerland), Paul Dixon, Mohammad Samragh Razlighi, Kumari Nishu, Devang Naik*

[Leveraging Large Language Models for Exploiting ASR Uncertainty](https://machinelearning.apple.com/research/leveraging-large-language-models)

*Pranay Dighe, Yi Su, Daniel Zheng, Yunshu Liu, Vineet Garg, Xiaochuan Niu, Ahmed Tewfik*

[Modality Drop-out for Multimodal Device Directed Speech Detection Using Verbal and Non-Verbal Features](https://machinelearning.apple.com/research/modality-dropout)

*Gautam Krishna, Sameer Dharur, Oggi Rudovic, Pranay Dighe, Saurabh Adya, Ahmed Hussen Abdelaziz, Ahmed Tewfik*

[Personalization of CTC-based End-to-End Speech Recognition Using Pronunciation-Driven Subword Tokenization](https://machinelearning.apple.com/research/ctc-based)

*Zhihong Lei, Ernie Pusateri, Michael Han, Leo Liu, Mingbin Xu, Tim Ng, Zhen Huang, Ruchir Travadi, Darien Zhang, Mirko Hannemann, Man-Hung Siu*

[Streaming Anchor Loss: Augmenting Supervision with Temporal Significance](https://machinelearning.apple.com/research/streaming-anchor-loss)

*Oggy Sarawgi, Jack Berkowitz, Vineet Garg, Arnav Kundu, Minsik Cho, Sai Srujana Buddi, Saurabh Adya, Ahmed Tewfik*

[Towards a World-English Language Model](https://machinelearning.apple.com/research/world-english-language-model)

*Rricha Jalota, Lyan Verwimp, Markus Nussbaum-Thom, Amr Mousa, Arturo Argueta, Youssef Oualil*

[A Multimodal Approach to Device-Directed Speech Detection with Large Language Models](https://machinelearning.apple.com/research/llm-device-directed-speech-detection)

*Dominik Wagner (FAU), Alex Churchill, Siddharth Sigtia, Panos Georgiou, Matt Mirsamadi, Aarshee Mishra, Erik Marchi*

[Investigating Salient Representations and Label Varience in Dimensional Speech Emotion Analysis](https://machinelearning.apple.com/research/investigating-salient-representation)

*Vikramjit Mitra, Jingping Nie, Erdrin Azemi*

[Resource-constrained stereo singing voice cancellation](https://machinelearning.apple.com/research/resource-constrained)

*Clara Borrelli, Dogac Basaran, Matthias Mauch , Matthew McVicar, James Rae, Mehrez Souden*

## Workshop Accepted Papers

[Multichannel Voice Trigger Detection based on Transform-average-concatenate](https://machinelearning.apple.com/research/multichannel-voice-trigger)

*Takuya Higuchi, Avamarie Brueggeman (The University of Texas at Dallas), Masood Delfarah, Stephen Shum*



## Acknowledgements

Daniele Giacobello is a member of the ICASSP 2024 Organizing Committee.

Takaaki Hori, Daniele Giacobello, and Yi Su are ICASSP 2024 session chairs.

Vikram Mitra is an Affiliate SLTC Member.

Yi Su, Aswin Sivaraman, Takaaki Hori, Daniele Giacobello, Vineet Garg, Jack Berkowitz, and Vikram Mitra are reviewers for ICASSP 2024.

Apple is sponsoring the International Conference on Acoustics, Speech and Signal Processing (ICASSP), which is taking place in person from April 14 to 19 in Seoul, South Korea. ICASSP is the IEEE Signal Processing Society's flagship conference on signal processing and its applications.

International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024

Apple sponsored the Neural Information Processing Systems (NeurIPS) conference, which took place in person from December 10 to 16 in New Orleans, Louisiana. NeurIPS is a conference that fosters the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects. Below was the schedule of Apple-sponsored workshops and events at NeurIPS 2023.

Neural Information Processing Systems (NeurIPS) 2023 

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

Preference based Reinforcement Learning (PbRL) has shown great promise in learning from human preference binary feedback on agent's trajectory behaviors, where one of the major goals is to reduce the number of queried human feedback. While the binary labels are a direct comment on the goodness of a trajectory behavior, there is still a need for resolving credit assignment especially in limited feedback. We propose our work, PRIor On Rewards (PRIOR) that learns a forward dynamics world model to approximate apriori selective attention over states which serves as a means to perform credit assignment over a given trajectory. Further, we propose an auxiliary objective that redistributes the total predicted return according to these PRIORs as a simple, yet effective means of improving reward learning performance. Our experiments on six robot-manipulation and three locomotion PbRL benchmarks demonstrates PRIOR's significant improvements in feedback-sample efficiency and reward recovery. Finally, we present our extensive ablations that study our design decisions and the ease of using PRIOR with existing PbRL methods.

ICLR

Hindsight PRIORs for Reward Learning from Human Preferences

Long prompts present a significant challenge for practical LLM-based systems that need to operate with low latency and limited resources. We investigate prompt compression for zero-shot dialogue systems that learn to use unseen APIs directly in-context from their documentation, which may take up hundreds of prompt tokens per API. We start from a recently introduced approach (Mu et al., 2023) that learns to compress the prompt into a few “gist token” activations during finetuning. However, this simple idea is ineffective in compressing API documentation, resulting in low accuracy compared to the baseline using an uncompressed prompt. In this work, we introduce two major improvements. First, we specialize gist tokens for different hierarchies within an API: we use one Gist_arg token for compressing an argument and one Gist_value token for compressing an acceptable value of a categorical argument. We then dynamically reveal Gist_value tokens only when they are needed. Second, we add a reconstruction loss to predict the API documentation from the gist tokens. On multiple API-calling tasks, our proposed system keeps the simplicity, efficiency, and large compression factor (20x on SGD) of the gist token approach while achieving significantly better accuracy.

Machine Learning
Research at Apple

Recent research

Hindsight PRIORs for Reward Learning from Human Preferences

Hierarchical and Dynamic Prompt Compression for Efficient Zero-shot API Usage

Research highlights

Enhancing Paragraph Generation with a Latent Language Diffusion Model

Personalizing Health and Fitness with Hybrid Modeling

All events

International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024

Neural Information Processing Systems (NeurIPS) 2023

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

Hindsight PRIORs for Reward Learning from Human Preferences

Hierarchical and Dynamic Prompt Compression for Efficient Zero-shot API Usage

Research highlights

Enhancing Paragraph Generation with a Latent Language Diffusion Model

Personalizing Health and Fitness with Hybrid Modeling

All events

International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024

Neural Information Processing Systems (NeurIPS) 2023

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple