Recent research has explored <a href="https://pubmed.ncbi.nlm.nih.gov/32706685/" target="_blank" aria-label="clinical monitoring, - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">clinical monitoring,</a> <a href="https://arxiv.org/abs/1802.02511" target="_blank" aria-label="cardiovascular events, - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">cardiovascular events,</a> and even clinical lab values from <a href="https://pubmed.ncbi.nlm.nih.gov/34031607/" target="_blank" aria-label="wearables data. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">wearables data.</a> As adoption increases, wearables data may become crucial in public health applications like <a href="https://www.thelancet.com/journals/landig/article/PIIS2589-7500(19)30222-5/fulltext" target="_blank" aria-label="disease monitoring - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">disease monitoring</a> and the <a href="https://pubmed.ncbi.nlm.nih.gov/34711377/" target="_blank" aria-label="design of epidemiological studies. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">design of epidemiological studies.</a>

Personalizing Health and Fitness with Hybrid Modeling

To address these challenges, we wanted to call attention to a [latent text diffusion model](/research/planner) that we introduced in the fall of 2023. The model synergizes non-autoregressive latent semantic diffusion with autoregressive generation to overcome the hurdles faced by its predecessors. Specifically, we hope to conduct research to improve the experience of users who benefit from more diversified and controlled text generation. By adopting a latent diffusion approach (as discussed in [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2212.09462) and [Latent Diffusion for Language Generation](https://arxiv.org/abs/2212.09462), PLANNER mitigates computational expenses typically associated with similar models, while simultaneously delivering superior diversity and cohesiveness, and reduce the repetition level of generated text, particularly in longer blocks of text and paragraphs, which have traditionally posed a challenge for text generation models. 

Our model, PLANNER, extends its benefit to various text generation tasks such as semantic generation, text completion, and summarization, with extensive evaluations of fluency, diversity, and repetition mitigation. 

<Figure id="figure1" 
caption={<>Figure 1: A three-stage model for text generation. We begin with a variational paragraph embedder in stage 1 and evolve the coarse text through our latent diffusion model, PLANNER, for a finer coherent result in stage 3.</>}><Figure1 title="A two-part model for text generation." theme="dark-gray" />
</Figure>

In stage 1 of [Figure 1](#figure1), a variational paragraph embedder encodes paragraphs into a series of latent codes. The encoder E and decoder D construct a bidirectional mapping between the discrete data space and the latent code space. The paragraph embeddings z are extracted by taking the first k hidden state vectors of dimension h from the final layer of E, which are fed into the initial steps of the decoder, which is trained to reconstruct the original text x. BOS and EOS represent “beginning of sentence” and “end of sentence” tokens, respectively.

In stage 2 of [Figure 1](#figure1), these latent codes z are processed by a transformer-based latent diffusion model (as discussed in the work [Scalable Diffusion Models with Transformers](https://arxiv.org/abs/2212.09748)) for training, so that it can generate new latent codes over time during inference time, simulating the evolution of text from coarse to fine. Finally, in stage 3 the decoder D translates these evolving latent codes into coherent text.

Our PLANNER latent diffusion model considers the conditioning signal as raw text, such as preceding context or the document to be summarized. We applied a conditional feature encoder τ to the input and used the hidden states at the last layer as y. We fed y and the time embedding t into the latent diffusion model through two channels, namely cross-attention and adaptive layer normalization. The aim of our research is to use existing text samples, such as an email or a summary of a document, to help generate longer texts that are both cohesive and readable. Examples in the following two figures are taken from a public dataset of text samples related to hotel reviews. 
 
<Figure id="figure2" theme="dark-gray"
caption={<>Figure 2: Compare the fine-tuned GPT-2 large model (the most relevant model at the time of research) results in column at the left with the PLANNER results at the right when generating text from a repetitive prompt (shown as “Prefix" in the figure). On the left, the GPT-2 model, despite using top-p sampling, still yields text with self-reinforced repetition. On the right, data from 512 generation roll-outs illustrate that the new method produces a wider variety of first 1-grams, showcasing its ability to generate more diversified text unaffected by the poorly devised prompt.</>}>
<Figure2 title="compares two language models" theme="dark-gray"/>
</Figure>
 
[Figure 2](#figure2) compares two language models: a fine-tuned GPT-2 large model and our method. It showcases how each model handles a prompt designed to evaluate their ability to generate diversified text from a repetitive cue. We decided to select GPT-2 because it was the most relevant model at the time of conducting research. Starting with the fine-tuned GPT-2 large model, this model has been initialized using GPT-2 large, which has 774 million parameters. As for publicly available versions of GPT-2, OpenAI has released different sizes of GPT-2 models, including a large version that is accessible for researchers and developers. However, the particular fine-tuned version we used in our paper, [PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model](/research/planner), may include proprietary dataset adjustments and may not be directly available.

- _FT_ stands for fine-tuning, which is the process of taking a pre-trained model and training it further on a new dataset to specialize its knowledge.
- _Greedy decoding_ is a method where, at each step in generating text, the model picks the word with the highest probability.
- _Top-p sampling_ is a technique where the model chooses from the top p percent of probable words, allowing for more randomness and potential creativity in its output, as addressed in the work [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751) 
- _512 generation rollouts_ refers to the number of times the model generates text to test its capabilities. In this context, it means the model was used to generate text, starting from the prompt, 512 times for evaluation.
- _N-grams_ are sequences of N tokens.

The percentage numbers in the n-gram columns indicate the frequency of each n-gram’s appearance within the generated text by a specific method. A lower maximum percentage suggests that there is a larger variety of different n-grams, which is typically seen as desirable for the generation of text that is less repetitive and more diverse.

"More diversified" implies that the generated sequences of words (n-grams) are more varied and less repetitive compared to the repetitive n-grams generated by other methods or models. This diversification generally indicates a higher quality of text generation that is more likely to generate useful and novel content for users.

Lastly, we observed accumulative errors in traditional autoregressive models, such as the ones in GPT-2, where the model gets stuck in a loop and produces repetitive or unhelpful output. In the context given, the repeated phrase “awful hotel” in the generated text from GPT-2 is an example of such an accumulative error. 

<Figure id="figure3" caption={<>Figure 3: This hotel review text generated by a diffusion model progresses over 10 steps, from a vague to a more distinct and richly detailed positive sentiment about the hotel experience. This development follows a coarse-to-fine approach, starting from general commendation and culminating in a vibrant and specific final review that praises the bartender and the establishment’s ambiance and amenities. </>}>
<Figure3 />
</Figure>

[Figure 3](#figure3) illustrates the gradual evolution of generated text over a series of 10 steps. The model begins with coarse initial predictions (represented in Figure 3 as step 1, the initial state) and progresses by performing repeated processing steps to denoise and improve the text.

The reader should envision this scenario not as a snapshot of text being entered or prompted by an iPhone user but as a systematic process by which a language model refines an initially vague or broad expression into a more detailed and specific review text. At step 1, the text is a rough suggestion of what the user might want to express — it is terse and lacks detail. As time progresses, the model fine-tunes the text, introducing more specific descriptions, sentiment, and sophisticated language. By step 10, the end state, the generated text resembles a thoughtfully composed review that one might expect from an experienced reviewer who gives particular attention to various aspects of their hotel stay.

Thus, [Figure 3](#figure3) shows how the PLANNER model's generation progresses from coarse to fine, giving readers a step-by-step visualization of how the text is iteratively enhanced to improve readability, specificity, and overall quality. The scenario starts with a minimal outline of positive sentiment and, over time, develops into a fleshed-out testimonial with vivid details emerging at each subsequent step.

## Conclusion 

The PLANNER model represents an advancement in the pursuit of improved natural language. Tackling the challenge of accumulative errors in traditional autoregressive models, our model leverages latent semantic diffusion to generate text that's fluent, controlled, and diversified.

## Acknowledgments 

Many people contributed to this work, including Richard Bai, Ronan Collobert, Zhe Gan, David Grangier, Edouard Grave, Tatiana Likhomanenko, Barry Theobald, Yinfei Yang, and Yizhe Zhang.

## Apple Resources 

Xu, Jin, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. “Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation.” [[link.]](/research/analyzing-mitigating-repetitions)

Zhang, Yizhe, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. 2023. “PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model.” [[link.]](/research/planner)

## External References 

Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." [[link.]](https://arxiv.org/abs/1506.03099)

Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” [[link.]](https://arxiv.org/abs/1904.09751)

Hu, Zhiting, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. “Toward Controlled Generation of Text.” [[link.]](https://arxiv.org/abs/1703.00955)

Keskar, Nitish Shirish, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. “CTRL: A Conditional Transformer Language Model for Controllable Generation.” [[link.]](https://arxiv.org/abs/1909.05858)

Lovelace, Justin, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q. Weinberger. 2023. “Latent Diffusion for Language Generation." [link.]](https://doi.org/10.48550/arXiv.2212.09462)

Peebles, William, and Saining Xie. 2022. “Scalable Diffusion Models with Transformers.” [[link.]](https://arxiv.org/abs/2212.09748) 

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” [[link.]](https://arxiv.org/abs/2112.10752)

In the fast-evolving world of natural language processing (NLP), there is a strong demand for generating coherent and controlled text, as referenced in the work <a href="https://arxiv.org/abs/1703.00955" target="_blank" aria-label="Toward Controlled Generation of Text. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Toward Controlled Generation of Text.</a> Traditional autoregressive models such as GPT, which have long been the industry standard, possess inherent limitations that sometimes manifest as repetitive and low-quality outputs, as seen in the work <a href="https://arxiv.org/abs/1904.09751" target="_blank" aria-label="The Curious Case of Neural Text Degeneration. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">The Curious Case of Neural Text Degeneration.</a> This is primarily due to a phenomenon known as "exposure bias," as seen in the work <a href="https://arxiv.org/abs/1506.03099" target="_blank" aria-label="Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.</a> This imperfection arises due to a mismatch between how these models are trained and their actual use during inference, often leading to error accumulation during text generation.

 Enhancing Paragraph Generation with a Latent Language Diffusion Model

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events at CHI 2024. Stop by the Apple booth in Kamehameha 1 Hall, Booth 100, from 5:30pm - 7:30pm HST on May 13; 10:00am - 6:00pm HST on May 14; 10:00am - 5:30pm HST on May 15; 10:00am - 12:00pm HST on May 16.

### Tuesday, May&nbsp;14
<ul class="links-stacked">
<li>LUNCHEON</li>
<li>CHI 2024 Appreciation Luncheon</a></li>
<li>12:20 PM - 2:00 PM HST, Room 304A</li>
<li>Alex Lim and Erdrin Azemi will be representing Apple at the CHI 2024 Appreciation Luncheon.</li>
</ul>

<ul class="links-stacked">
<li>WORKSHOP</li>
<li>Journey to Apple - Career Panel with Human Factors Engineers</a></li>
<li>12:30 PM - 1:05 PM HST, Room TBD</li>
<li>Spencer Jefferies, Ava Rezvani Lannin, Stacie Lafko and Alex Lim are speakers.</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
<li><a href=https://machinelearning.apple.com/research/model-compression-in-practice>Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences</a></li>
<li>4:00 PM - 4:15 PM HST, 316A</li>
<li>Fred Hohman, Mary Beth Kery, Donghao Ren, Dominik Moritz</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
<li><a href=https://machinelearning.apple.com/research/talaria>Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference</a></li>
<li>4:15 PM - 4:30 PM HST, 316A</li>
<li>Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey Bigham, Zhile Ren, Cecile Foret, Qi Shan, Xiaoyi Zhang</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
<li><a href=https://machinelearning.apple.com/research/axnav>AXNav: Replaying Accessibility Tests from Natural Language</a></li>
<li>4:30 PM - 4:45 PM HST, 314</li>
<li>Maryam Taeb (Florida State University), Amanda Swearngin, Eldon Schoop, Regina Cheng, Yue Jiang, Jeff Nichols</li>
</ul>


</div>

## Accepted Papers

[AXNav: Replaying Accessibility Tests from Natural Language](https://machinelearning.apple.com/research/axnav)

*Maryam Taeb (Florida State University), Amanda Swearngin, Eldon Schoop, Regina Cheng, Yue Jiang, Jeff Nicholsl*
 
[Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://machinelearning.apple.com/research/model-compression-in-practice)

*Fred Hohman, Mary Beth Kery, Donghao Ren, Dominik Moritz*

[Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference](https://machinelearning.apple.com/research/talaria)

*Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey Bigham, Zhile Ren, Cecile Foret, Qi Shan, Xiaoyi Zhang*



## Acknowledgements

Amanda Swearngin, Titus Barik and Eldon Schoop are Associate Chairs of a program or organizing committee for CHI 2024.

Xiaoying Pu, Moses Namara, Amanda Swearngin, Xiaoyi Zhang, Regina Cheng, Mary Beth Kery, Eldon Schoop, Jeremy Warner and Agni Kumar are reviewers for CHI 2024.

Apple is sponsoring the ACM Human-Computer Interaction Conference (CHI), which is taking place in person from May 11 to May 16, 2024 in Honolulu, Hawai'i.

ACM Human-Computer Interaction conference (CHI) 2024

Apple is sponsoring the International Conference on Learning Representations (ICLR), which is taking place in person from May 7 to 11 in Vienna Austria. ICLR brings together professionals dedicated to the advancement of deep learning.

International Conference on Learning Representations (ICLR) 2024

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

This paper has been accepted at the Data Problems for Foundation Models workshop at ICLR 2024.

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased.  Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web.   In this work, we proposeWebRephrase Augmented Pre-training (WRAP) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First,  we show that using WRAP on the C4 dataset, which is naturally noisy,  speeds up pre-training by ~3 times. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data  (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher "quality" than web-scraped data.

Data Problems for Foundation Models Workshop at ICLR

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed by estimating a distribution over the output, but in reality, such approaches can be sensitive to estimation error and yield unstable intervals. Here, we circumvent the challenges by converting regression to a classification problem and then use CP for classification to obtain CP sets for regression. To preserve the ordering of the continuous-output space, we design a new loss function and make necessary modifications to the CP classification techniques. Empirical results on many benchmarks shows that this simple approach gives surprisingly good results on many practical problems.

Machine Learning
Research at Apple

Recent research

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Conformal Prediction via Regression-as-Classification

Research highlights

Personalizing Health and Fitness with Hybrid Modeling

Enhancing Paragraph Generation with a Latent Language Diffusion Model

All events

ACM Human-Computer Interaction conference (CHI) 2024

International Conference on Learning Representations (ICLR) 2024

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Conformal Prediction via Regression-as-Classification

Research highlights

Personalizing Health and Fitness with Hybrid Modeling

Enhancing Paragraph Generation with a Latent Language Diffusion Model

All events

ACM Human-Computer Interaction conference (CHI) 2024

International Conference on Learning Representations (ICLR) 2024

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple