In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with [Private Cloud Compute](https://security.apple.com/blog/private-cloud-compute/) and running on Apple silicon servers — have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly. These two foundation models are part of a larger family of generative models created by Apple to support users and developers; this includes a coding model to build intelligence into Xcode, as well as a diffusion model to help users express themselves visually, for example, in the Messages app. We look forward to sharing more information soon on this broader set of models.


Update - July 29, 2024: The figures in this article have been updated to reflect the model versions and evaluations used in the technical report released today. For more detail, please see the paper: <a href="/research/apple-intelligence-foundation-language-models" className="more">Apple Intelligence Foundation Language Models</a>.


## Our Focus on Responsible AI Development
Apple Intelligence is designed with our core values at every step and built on a foundation of groundbreaking privacy innovations.

Additionally, we have created a set of Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them:

1. Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.
2. Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models.
3. Design with care: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback.
4. Protect privacy: We protect our users' privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users' private personal data or user interactions when training our foundation models.

These principles are reflected throughout the architecture that enables Apple Intelligence, connects features and tools with specialized models, and scans inputs and outputs to provide each feature with the information needed to function responsibly.

In the remainder of this overview, we provide details on decisions such as: how we develop models that are highly capable, fast, and power-efficient; how we approach training these models; how our adapters are fine-tuned for specific user needs; and how we evaluate model performance for both helpfulness and unintended harm.

<Figure alt="A diagram that shows the training steps for the Apple foundation models." id="figure1" caption={<>Figure 1: Modeling overview for the Apple foundation models.</>} animate="fade-up">
 <Image src="https://mlr.cdn-apple.com/media/Fig1_Responsible_AI_8bf7727ab5.png" alt="Modeling overview" />
</Figure>

## Pre-Training
Our foundation models are trained on [Apple's AXLearn framework](https://github.com/apple/axlearn), an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot. Web publishers have [the option to opt out](https://support.apple.com/en-us/119829) of the use of their web content for Apple Intelligence training with a data usage control.

We never use our users’ private personal data or user interactions when training our foundation models, and we apply filters to remove personally identifiable information like social security and credit card numbers that are publicly available on the Internet. We also filter profanity and other low-quality content to prevent its inclusion in the training corpus. In addition to filtering, we perform data extraction, deduplication, and the application of a model-based classifier to identify high quality documents.

## Post-Training
We find that data quality is essential to model success, so we utilize a hybrid data strategy in our training pipeline, incorporating both human-annotated and synthetic data, and conduct thorough data curation and filtering procedures. We have developed two novel algorithms in post-training: (1) a rejection sampling fine-tuning algorithm with teacher committee, and (2) a reinforcement learning from human feedback (RLHF) algorithm with mirror descent policy optimization and a leave-one-out advantage estimator. We find that these two algorithms lead to significant improvement in the model’s instruction-following quality.

## Optimization
In addition to ensuring our generative models are highly capable, we have used a range of innovative techniques to optimize them on-device and on our private cloud for speed and efficiency. We have applied an extensive set of optimizations for both first token and extended token inference performance.

Both the on-device and server models use grouped-query-attention. We use shared input and output vocab embedding tables to reduce memory requirements and inference cost. These shared embedding tensors are mapped without duplications. The on-device model uses a vocab size of 49K, while the server model uses a vocab size of 100K, which includes additional language and technical tokens.

For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. To maintain model quality, we developed a new framework using LoRA adapters that incorporates a mixed 2-bit and 4-bit configuration strategy &mdash; averaging 3.7 bits-per-weight &mdash; to achieve the same accuracy as the uncompressed models. More aggressively, the model can be compressed to 3.5 bits-per-weight without significant quality loss.

Additionally, we use an interactive model latency and power analysis tool, [Talaria](/research/talaria), to better guide the bit rate selection for each operation. We also utilize activation quantization and embedding quantization, and have developed an approach to enable efficient Key-Value (KV) cache update on our neural engines.

With this set of optimizations, on iPhone 15 Pro we are able to reach time-to-first-token latency of about 0.6 millisecond per prompt token, and a generation rate of 30 tokens per second. Notably, this performance is attained before employing token speculation techniques, from which we see further enhancement on the token generation rate.

## Model Adaptation
Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks. For our models we adapt the attention matrices, the attention projection matrix, and the fully connected layers in the point-wise feedforward networks for a suitable set of the decoding layers of the transformer architecture.

By fine-tuning only the adapter layers, the original parameters of the base pre-trained model remain unchanged, preserving the general knowledge of the model while tailoring the adapter layers to support specific tasks.

<Figure alt="A diagram of the Apple Foundation Model and Adapters." id="figure2" caption={<>Figure 2: Adapters are small collections of model weights that are overlaid onto the common base foundation model. They can be dynamically loaded and swapped &mdash; giving the foundation model the ability to specialize itself on-the-fly for the task at hand. Apple Intelligence includes a broad set of adapters, each fine-tuned for a specific feature. It’s an efficient way to scale the capabilities of our foundation model.</>} className="overflow-visible" animate="fade-up">
 <Figure2 src="https://mlr.cdn-apple.com/video/Fig2_Adapters_9f087bc5c5.mp4" />
</Figure>

We represent the values of the adapter parameters using 16 bits, and for the ~3 billion parameter on-device model, the parameters for a rank 16 adapter typically require 10s of megabytes. The adapter models can be dynamically loaded, temporarily cached in memory, and swapped — giving our foundation model the ability to specialize itself on the fly for the task at hand while efficiently managing memory and guaranteeing the operating system's responsiveness.

To facilitate the training of the adapters, we created an efficient infrastructure that allows us to rapidly retrain, test, and deploy adapters when either the base model or the training data gets updated. The adapter parameters are initialized using the accuracy-recovery adapter introduced in the Optimization section.

## Performance and Evaluation
Our focus is on delivering generative models that can enable users to communicate, work, express themselves, and get things done across their Apple products. When benchmarking our models, we focus on human evaluation as we find that these results are highly correlated to user experience in our products. We conducted performance evaluations on both feature-specific adapters and the foundation models.

To illustrate our approach, we look at how we evaluated our adapter for summarization. As product requirements for summaries of emails, messages, and notifications differ in subtle but important ways, we fine-tune accuracy-recovery low-rank (LoRA) adapters on top of the palletized model to meet these specific requirements. Our training data is based on synthetic summaries generated from bigger server models, filtered by a rejection sampling strategy that keeps only the high quality summaries.

To evaluate the product-specific summarization, we use a set of 750 responses carefully sampled for each use case. These evaluation datasets emphasize a diverse set of inputs that our product features are likely to face in production, and include a stratified mixture of single and stacked documents of varying content types and lengths. As product features, it was important to evaluate performance against datasets that are representative of real use cases. We find that overall, our models with adapters generate better summaries than a comparable model.

As part of responsible development, we identified and evaluated specific risks inherent to summarization. For example, summaries occasionally remove important nuance or other details in ways that are undesirable. However, we found that the summarization adapter did not amplify sensitive content in over 99% of targeted adversarial examples. We continue to adversarially probe to identify unknown harms and expand our evaluations to help guide further improvements.

<Figure 
 id="figure3" 
 caption={<>Figure 3: Ratio of "good" and "poor" responses for three summarization use cases relative to all responses. Summaries are classified as "good", "neutral", "poor" given the grader's scores across five dimensions. A result is classified as "good" if all of the dimensions are good (higher is better). A result is classified as "poor" if any of the dimensions are poor (lower is better). Our models with adapters overall generate better summaries than comparable models.</>}
 animate="fade-up">
 <Figure3 title="Human Satisfaction with Summarization Feature" />
</Figure>

In addition to evaluating feature specific performance powered by foundation models and adapters, we evaluate both the on-device and server-based models’ general capabilities. We utilize a comprehensive evaluation set of real-world prompts to test the general model capabilities. These prompts are diverse across different difficulty levels and cover major categories such as brainstorming, classification, closed question answering, coding, extraction, mathematical reasoning, open question answering, rewriting, safety, summarization, and writing.

We compare our models with both open-source models (Phi-3, Gemma, Mistral, DBRX, Llama) and commercial models of comparable size (GPT-3.5, GPT-4)<a href="#footnotes">1</a>. We find that our models are preferred by human graders over most comparable competitor models. On this benchmark, our on-device model, with ~3B parameters, outperforms larger models including Phi-3-mini, Mistral-7B, Gemma-7B, and Llama-3-8B. Our server model compares favorably to DBRX-Instruct, Mixtral-8x22B, GPT-3.5, and Llama-3-70B while being highly efficient.

<Figure 
 id="figure4" 
 caption={<>Figure 4: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models. We find that our models are preferred by human graders.</>}
 animate="fade-up">
 <Figure4 title="Human Evaluation" />
</Figure>

We use a set of diverse adversarial prompts to test the model performance on harmful content, sensitive topics, and factuality. We measure the violation rates of each model as evaluated by human graders on this evaluation set, with a lower number being desirable. Both the on-device and server models are robust when faced with adversarial prompts, achieving violation rates lower than open-source and commercial models.

<Figure 
 id="figure5" 
 caption={<>Figure 5: Fraction of violating responses for harmful content, sensitive topics, and factuality (lower is better). Our models are robust when faced with adversarial prompts.</>} 
 animate="fade-up">
 <Figure5 title="Human Evaluation of Output Harmfulness" />
</Figure>

Our models are preferred by human graders as safe and helpful over competitor models for these prompts. However, considering the broad capabilities of large language models, we understand the limitation of our safety benchmark. We are actively conducting both manual and automatic red-teaming with internal and external teams to continue evaluating our models' safety.

<Figure 
 id="figure6" 
 caption={<>Figure 6: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models on safety prompts. Human graders found our responses safer and more helpful.</>}
 animate="fade-up">
 <Figure6 title="Human Preference Evaluation on Safety Prompts" />
</Figure>

To further evaluate our models, we use the Instruction-Following Eval (IFEval) benchmark to compare their instruction-following capabilities with models of comparable size. The results suggest that both our on-device and server model follow detailed instructions better than the open-source and commercial models of comparable size.

<Figure 
 id="figure7" 
 caption={<>Figure 7: Instruction-following capability (measured with IFEval) for Apple's foundation models and models of comparable size (higher is better).</>}
 animate="fade-up">
 <Figure7 title="IFEval Benchmarks" />
</Figure>

We evaluate our models’ writing ability on our internal summarization and composition benchmarks, consisting of a variety of writing instructions. These results do not refer to our feature-specific adapter for summarization (seen in [Figure 3](#figure3)), nor do we have an adapter focused on composition.

<Figure 
 id="figure8" 
 caption={<>Figure 8: Writing ability on internal summarization and composition benchmarks (higher is better).</>}
 animate="fade-up">
 <Figure8 title="Writing Benchmarks" />
</Figure>

## Conclusion
The Apple foundation models and adapters introduced at WWDC24 underlie Apple Intelligence, the new personal intelligence system that is integrated deeply into iPhone, iPad, and Mac, and enables powerful capabilities across language, images, actions, and personal context. Our models have been created with the purpose of helping users do everyday activities across their Apple products, and developed responsibly at every stage and guided by Apple’s core values. We look forward to sharing more information soon on our broader family of generative models, including language, diffusion, and coding models.

{<h2 id="footnotes">Footnotes</h2>}
[1] We compared against the following model versions: gpt-3.5-turbo-0125, gpt-4-0125-preview, Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.2, Mixtral-8x22B-Instruct-v0.1, Gemma-1.1-2B, Gemma-1.1-7B, Llama-3-8B-Instruct, and Llama-3-70B-Instruct. The open-source and Apple models are evaluated in bfloat16 precision.

At the 2024 <a href="https://developer.apple.com/wwdc24/" class="more">Worldwide Developers Conference</a>, we introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia.
Apple Intelligence is comprised of multiple highly-capable generative models that are specialized for our users’ everyday tasks, and can adapt on the fly for their current activity. The foundation models built into Apple Intelligence have been fine-tuned for user experiences such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across apps.

Introducing Apple’s On-Device and Server Foundation Models

To address these challenges, we wanted to call attention to a [latent text diffusion model](/research/planner) that we introduced in the fall of 2023. The model synergizes non-autoregressive latent semantic diffusion with autoregressive generation to overcome the hurdles faced by its predecessors. Specifically, we hope to conduct research to improve the experience of users who benefit from more diversified and controlled text generation. By adopting a latent diffusion approach (as discussed in [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2212.09462) and [Latent Diffusion for Language Generation](https://arxiv.org/abs/2212.09462), PLANNER mitigates computational expenses typically associated with similar models, while simultaneously delivering superior diversity and cohesiveness, and reduce the repetition level of generated text, particularly in longer blocks of text and paragraphs, which have traditionally posed a challenge for text generation models. 

Our model, PLANNER, extends its benefit to various text generation tasks such as semantic generation, text completion, and summarization, with extensive evaluations of fluency, diversity, and repetition mitigation. 

<Figure id="figure1" 
caption={<>Figure 1: A three-stage model for text generation. We begin with a variational paragraph embedder in stage 1 and evolve the coarse text through our latent diffusion model, PLANNER, for a finer coherent result in stage 3.</>}><Figure1 title="A two-part model for text generation." theme="dark-gray" />
</Figure>

In stage 1 of [Figure 1](#figure1), a variational paragraph embedder encodes paragraphs into a series of latent codes. The encoder E and decoder D construct a bidirectional mapping between the discrete data space and the latent code space. The paragraph embeddings z are extracted by taking the first k hidden state vectors of dimension h from the final layer of E, which are fed into the initial steps of the decoder, which is trained to reconstruct the original text x. BOS and EOS represent “beginning of sentence” and “end of sentence” tokens, respectively.

In stage 2 of [Figure 1](#figure1), these latent codes z are processed by a transformer-based latent diffusion model (as discussed in the work [Scalable Diffusion Models with Transformers](https://arxiv.org/abs/2212.09748)) for training, so that it can generate new latent codes over time during inference time, simulating the evolution of text from coarse to fine. Finally, in stage 3 the decoder D translates these evolving latent codes into coherent text.

Our PLANNER latent diffusion model considers the conditioning signal as raw text, such as preceding context or the document to be summarized. We applied a conditional feature encoder τ to the input and used the hidden states at the last layer as y. We fed y and the time embedding t into the latent diffusion model through two channels, namely cross-attention and adaptive layer normalization. The aim of our research is to use existing text samples, such as an email or a summary of a document, to help generate longer texts that are both cohesive and readable. Examples in the following two figures are taken from a public dataset of text samples related to hotel reviews. 
 
<Figure id="figure2" theme="dark-gray"
caption={<>Figure 2: Compare the fine-tuned GPT-2 large model (the most relevant model at the time of research) results in column at the left with the PLANNER results at the right when generating text from a repetitive prompt (shown as “Prefix" in the figure). On the left, the GPT-2 model, despite using top-p sampling, still yields text with self-reinforced repetition. On the right, data from 512 generation roll-outs illustrate that the new method produces a wider variety of first 1-grams, showcasing its ability to generate more diversified text unaffected by the poorly devised prompt.</>}>
<Figure2 title="compares two language models" theme="dark-gray"/>
</Figure>
 
[Figure 2](#figure2) compares two language models: a fine-tuned GPT-2 large model and our method. It showcases how each model handles a prompt designed to evaluate their ability to generate diversified text from a repetitive cue. We decided to select GPT-2 because it was the most relevant model at the time of conducting research. Starting with the fine-tuned GPT-2 large model, this model has been initialized using GPT-2 large, which has 774 million parameters. As for publicly available versions of GPT-2, OpenAI has released different sizes of GPT-2 models, including a large version that is accessible for researchers and developers. However, the particular fine-tuned version we used in our paper, [PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model](/research/planner), may include proprietary dataset adjustments and may not be directly available.

- _FT_ stands for fine-tuning, which is the process of taking a pre-trained model and training it further on a new dataset to specialize its knowledge.
- _Greedy decoding_ is a method where, at each step in generating text, the model picks the word with the highest probability.
- _Top-p sampling_ is a technique where the model chooses from the top p percent of probable words, allowing for more randomness and potential creativity in its output, as addressed in the work [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751) 
- _512 generation rollouts_ refers to the number of times the model generates text to test its capabilities. In this context, it means the model was used to generate text, starting from the prompt, 512 times for evaluation.
- _N-grams_ are sequences of N tokens.

The percentage numbers in the n-gram columns indicate the frequency of each n-gram’s appearance within the generated text by a specific method. A lower maximum percentage suggests that there is a larger variety of different n-grams, which is typically seen as desirable for the generation of text that is less repetitive and more diverse.

"More diversified" implies that the generated sequences of words (n-grams) are more varied and less repetitive compared to the repetitive n-grams generated by other methods or models. This diversification generally indicates a higher quality of text generation that is more likely to generate useful and novel content for users.

Lastly, we observed accumulative errors in traditional autoregressive models, such as the ones in GPT-2, where the model gets stuck in a loop and produces repetitive or unhelpful output. In the context given, the repeated phrase “awful hotel” in the generated text from GPT-2 is an example of such an accumulative error. 

<Figure id="figure3" caption={<>Figure 3: This hotel review text generated by a diffusion model progresses over 10 steps, from a vague to a more distinct and richly detailed positive sentiment about the hotel experience. This development follows a coarse-to-fine approach, starting from general commendation and culminating in a vibrant and specific final review that praises the bartender and the establishment’s ambiance and amenities. </>}>
<Figure3 />
</Figure>

[Figure 3](#figure3) illustrates the gradual evolution of generated text over a series of 10 steps. The model begins with coarse initial predictions (represented in Figure 3 as step 1, the initial state) and progresses by performing repeated processing steps to denoise and improve the text.

The reader should envision this scenario not as a snapshot of text being entered or prompted by an iPhone user but as a systematic process by which a language model refines an initially vague or broad expression into a more detailed and specific review text. At step 1, the text is a rough suggestion of what the user might want to express — it is terse and lacks detail. As time progresses, the model fine-tunes the text, introducing more specific descriptions, sentiment, and sophisticated language. By step 10, the end state, the generated text resembles a thoughtfully composed review that one might expect from an experienced reviewer who gives particular attention to various aspects of their hotel stay.

Thus, [Figure 3](#figure3) shows how the PLANNER model's generation progresses from coarse to fine, giving readers a step-by-step visualization of how the text is iteratively enhanced to improve readability, specificity, and overall quality. The scenario starts with a minimal outline of positive sentiment and, over time, develops into a fleshed-out testimonial with vivid details emerging at each subsequent step.

## Conclusion 

The PLANNER model represents an advancement in the pursuit of improved natural language. Tackling the challenge of accumulative errors in traditional autoregressive models, our model leverages latent semantic diffusion to generate text that's fluent, controlled, and diversified.

## Acknowledgments 

Many people contributed to this work, including Richard Bai, Ronan Collobert, Zhe Gan, David Grangier, Edouard Grave, Tatiana Likhomanenko, Barry Theobald, Yinfei Yang, and Yizhe Zhang.

## Apple Resources 

Xu, Jin, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. “Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation.” [[link.]](/research/analyzing-mitigating-repetitions)

Zhang, Yizhe, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. 2023. “PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model.” [[link.]](/research/planner)

## External References 

Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." [[link.]](https://arxiv.org/abs/1506.03099)

Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” [[link.]](https://arxiv.org/abs/1904.09751)

Hu, Zhiting, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. “Toward Controlled Generation of Text.” [[link.]](https://arxiv.org/abs/1703.00955)

Keskar, Nitish Shirish, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. “CTRL: A Conditional Transformer Language Model for Controllable Generation.” [[link.]](https://arxiv.org/abs/1909.05858)

Lovelace, Justin, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q. Weinberger. 2023. “Latent Diffusion for Language Generation." [link.]](https://doi.org/10.48550/arXiv.2212.09462)

Peebles, William, and Saining Xie. 2022. “Scalable Diffusion Models with Transformers.” [[link.]](https://arxiv.org/abs/2212.09748) 

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” [[link.]](https://arxiv.org/abs/2112.10752)

In the fast-evolving world of natural language processing (NLP), there is a strong demand for generating coherent and controlled text, as referenced in the work <a href="https://arxiv.org/abs/1703.00955" target="_blank" aria-label="Toward Controlled Generation of Text. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Toward Controlled Generation of Text.</a> Traditional autoregressive models such as GPT, which have long been the industry standard, possess inherent limitations that sometimes manifest as repetitive and low-quality outputs, as seen in the work <a href="https://arxiv.org/abs/1904.09751" target="_blank" aria-label="The Curious Case of Neural Text Degeneration. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">The Curious Case of Neural Text Degeneration.</a> This is primarily due to a phenomenon known as "exposure bias," as seen in the work <a href="https://arxiv.org/abs/1506.03099" target="_blank" aria-label="Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.</a> This imperfection arises due to a mismatch between how these models are trained and their actual use during inference, often leading to error accumulation during text generation.

 Enhancing Paragraph Generation with a Latent Language Diffusion Model

<div class="callout">


## Schedule

Stop by the Apple booth in Centre de Convencions Internacional de Barcelona (CCIB), Booth 28, from 10:30 - 19:00 (CEST) on August 27 and 28, and 10:30 - 14:30 (CEST) on August 29.

### Monday, August&nbsp;26

<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href=https://wamlm-kdd.github.io/wamlm/index.html/>Workshop on Applied Machine Learning Management</a></li>
 <li>14:00 - 18:00 CEST, Rooms 127-128</li> 

<li> Saloni Potdar is an organizer for this workshop.</li>
</ul>

<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href=https://biokdd.org/biokdd24/>BioKDD 2024</a></li>
 <li>9:00 - 13:00 CEST, Rooms 131-132</li> 


<li>ORAL</li>
 <li><a href=/research/pre-trained-foundation-model>Pre-Trained Foundation Model Representations to Uncover Breathing patterns in Speech</a></li>
<li>12:45 - 13:00 CEST, Rooms 131-132</li>
<li>Vikram Mitra, Anirban Chatterjee, Ke Zhai, Helen Weng, Ayuko Hill, Nicole Hay, Christopher Webb, Jamie Cheng, Erdrin Azemi</li>
</ul>
</div>


## Workshop Accepted Papers

[Pre-Trained Foundation Model Representations to Uncover Breathing Patterns in Speech](/research/pre-trained-foundation-model)

*Vikram Mitra, Anirban Chatterjee, Ke Zhai, Helen Weng, Ayuko Hill, Nicole Hay, Christopher Webb, Jamie Cheng, Erdrin Azemi*

## Acknowledgements

Aakash Goel is a reviewer for KDD 2024.

Apple is sponsoring the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), being held in Barcelona, Spain from August 25 to August 29. KDD is an interdisciplinary conference bringing together researchers and practitioners from data science, data mining, knowledge discovery, large-scale data analytics, and big data. Below is the schedule of Apple-sponsored workshops and events at KDD 2024.

ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2024

<div class="callout">

## Schedule

Stop by the Apple booth in Centara Grand and Bangkok Convention Center, Floor 22, Booth #1, from 9:00 - 17:30 (UTC+7) on August 12, 13 and 14.

### Monday, August&nbsp;12
 
<ul class="links-stacked">
<li>ORAL</li>
 <li><a href=/research/efficient-large-language>LLM in a Flash: Efficient Large Language Model Inference with Limited Memory</a></li>
 <li>14:00 - 15:30 UTC+7, Efficient/Low-Resource Methods for NLP II, Lotus Suite 1-4</li>
<li>Keivan Alizadeh Vahid, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
 <li><a href=/research/quality-estimation-data-annotation>On Efficient and Statistical Quality Estimation for Data Annotation</a></li>
 <li>14:00 - 15:30 UTC+7, Resources and Evaluation II, World Ballroom A</li>
<li>Jan-Christoph Klie, Juan Haladjian, Marc Kirchner, Rahul Nair</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/direct-large-language>Direct Large Language Model Alignment Through Self-Rewarding Contrast Prompt Distillation</a></li>
<li>16:00 - 17:30 UTC+7, In-Person Poster Session 3, Convention Center A1</li>
<li>law Liu, Felix Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, Lijie Wen </li>
</ul>

### Tuesday, August&nbsp;13

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/parlor-game-arena>Probing the Multi-Turn Planning Capabilities of LLMs via 20 Question Games</a></li>
<li>10:30 - 12:00 UTC+7, In-Person Poster Session 4, Convention Center A1</li>
<li>Yizhe Zhang, Jiarui Lu, Navdeep Jaitly</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/toad>TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles</a></li>
<li>12:15 - 13:15 UTC+7, Findings In-Person Poster Session 3, Convention Center A1</li>
<li>Yinhong Liu, Yimai Fang, David Vandyke, Nigel Collier</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/quality-estimation-data-annotation>On Efficient and Statistical Quality Estimation for Data Annotation</a></li>
<li>16:00 - 17:30 UTC+7, In-Person Poster Session 5, Convention Center A1</li>
<li>Jan-Christoph Klie, Juan Haladjian, Marc Kirchner, Rahul Nair</li>
</ul>

### Wednesday, August&nbsp;14

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/efficient-large-language>LLM in a Flash: Efficient Large Language Model Inference with Limited Memory</a></li>
<li>10:30 - 12:00 UTC+7, In-Person Poster Session 6, Convention Center A1</li>
<li>Keivan Alizadeh Vahid, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/recipe-for-compute>Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling</a></li>
<li>10:30 - 12:00 UTC+7, In-Person Poster Session 6, Convention Center A1</li>
<li>Pratyush Maini, Skyler Seto, David Grangier, Richard Bai, Yizhe Zhang, Navdeep Jaitly</li>
</ul>

### Thursday, August&nbsp;15

<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href=https://sites.google.com/view/loresmt/>The Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT) 2024</a></li>
 <li>9:00 - 19:40 UTC+7, Lotus Suite 1&2</li> 

<li>ORAL</li>
 <li><a href=/research/contrastive-alignment-instructions>Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages</a></li>
<li>11:00 - 11:18 UTC+7, Lotus Suite 1&2</li>
<li>Zhuoyuan Mao, Yen Yu</li>

<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href=https://nl-reasoning-workshop.github.io>Natural Language Reasoning and Structured Explanations Workshop 2024</a></li>
 <li> 8:55 - 17:30 UTC+7, Lotus Suite 7</li> 

<li>POSTER</li>
 <li><a href=/research/applying-rlaif>Applying RLAIF for Code Generation with API-usage in Lightweight LLMs</a></li>
<li>16:00 - TBA UTC+7, Lotus Suite 7</li>
<li>Sujan Dutta, Sayantan Mahinder, Raviteja Anantha Ramesh, Bortik Bandyopadhyay</li>
</ul>

### Friday, August&nbsp;16

<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href=https://knowledgeable-lm.github.io>Towards Knowledgeable Language Models 2024</a></li>
 <li>9:00am - 17:30 UTC+7, Lotus Suite 10</li> 

<li>POSTER</li>
 <li><a href=/research/spinning-configurable-scalable>ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models</a></li>
<li>16:00 - 1730 UTC+7, Lotus Suite 10</li>
<li>Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeff Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan Khalilabad, Yunyao Li</li>

<li>POSTER</li>
 <li><a href=/research/kglens-towards-efficient>KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphs</a></li>
<li>16:00 - 17:30 UTC+7, Lotus Suite 10</li>
<li>Daniel Zheng, Richard Bai, Yizhe Zhang, Yi Su, Xiaochuan Niu, Navdeep Jaitly</li>
</ul>

<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href=https://genderbiasnlp.talp.cat>5th Workshop on Gender Bias in Natural Language Processing 2024</a></li>
 <li>9:00 - 18:15 UTC+7, World Ballroom B</li> 

<li>POSTER</li>
 <li><a href=/research/generating-gender-alternatives>Generating Gender Alternatives in Machine Translation</a></li>
<li>14:00 - 15:30 UTC+7, World Ballroom B</li>
<li>Sarthak Garg, Mozhdeh Gheini, Clara Emmanuel, Tatiana Likhomanenko, Qin Gao, Matthias Paulik</li>
</ul>

### Thursday, August&nbsp;22

<ul class="links-stacked">
<li>VIRTUAL POSTER</li>
 <li><a href=/research/time-sensitive-finetuning>Time Sensitive Knowledge Editing through Efficient Finetuning</a></li>
<li>10:15 UTC+7, Virtual Poster Session 2</li>
<li>Hugh Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan Khalilabad, Yunyao Li</li>
</ul>

</div>

## Accepted Papers

[Direct Large Language Model Alignment Through Self-Rewarding Contrast Prompt Distillation](/research/direct-large-language)

*law Liu, Felix Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, Lijie Wen*

[LLM in a Flash: Efficient Large Language Model Inference with Limited Memory](/research/efficient-large-language)

*Keivan Alizadeh Vahid, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, Mehrdad Farajtabar*

[On Efficient and Statistical Quality Estimation for Data Annotation](/research/quality-estimation-data-annotation)

*Jan-Christoph Klie, Juan Haladjian, Marc Kirchner, Rahul Nair*

[Probing the Multi-Turn Planning Capabilities of LLMs via 20 Question Games](/research/parlor-game-arena)

*Yizhe Zhang, Jiarui Lu, Navdeep Jaitly*

[Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](/research/recipe-for-compute)

*Pratyush Maini, Skyler Seto, David Grangier, Richard Bai, Yizhe Zhang, Navdeep Jaitly*

[Time Sensitive Knowledge Editing through Efficient Finetuning](/research/time-sensitive-finetuning)

*Hugh Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan Khalilabad, Yunyao Li*

[TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles](/research/toad)

*Yinhong Liu, Yimai Fang, David Vandyke, Nigel Collier*

## Workshop Accepted Papers

[Applying RLAIF for Code Generation with API-usage in Lightweight LLMs](/research/applying-rlaif)

*Sujan Dutta, Sayantan Mahinder, Raviteja Anantha Ramesh, Bortik Bandyopadhyay*

[ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models](/research/spinning-configurable-scalable)

*Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeff Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan Khalilabad, Yunyao Li*

[Generating Gender Alternatives in Machine Translation](/research/generating-gender-alternatives)

*Sarthak Garg, Mozhdeh Gheini, Clara Emmanuel, Tatiana Likhomanenko, Qin Gao, Matthias Paulik*

[KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphs](/research/kglens-towards-efficient)

*Daniel Zheng, Richard Bai, Yizhe Zhang, Yi Su, Xiaochuan Niu, Navdeep Jaitly*

[Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages](/research/contrastive-alignment-instructions)

*Zhuoyuan Mao, Yen Yu*


## Acknowledgements

Yizhe Zhang is an Area Chair for ACL 2024.

Maartje ter Hoeve, Matthias Sperber, Mingbin Xu, Richard Bai, and Yimai Fang are reviewers for ACL 2024.

Apple is sponsoring the annual meeting of the Association for Computational Linguistics (ACL), which takes place in person from August 11 to 16, in Bangkok, Thailand. ACL is a conference in the field of computational linguistics, covering a broad spectrum of diverse research areas that are concerned with computational approaches to natural language. Below is the schedule of Apple-sponsored workshops and events at ACL 2024.

Association for Computational Linguistics (ACL) 2024

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

Always-on machine learning models require a very low memory and compute footprint. Their restricted parameter count limits the model’s capacity to learn, and the effectiveness of the usual training algorithms to find the best parameters. Here we show that a small convolutional model can be better trained by first refactoring its computation into a larger redundant multi-branched architecture. Then, for inference, we algebraically re-parameterize the trained model into the single-branched form with fewer parameters for a lower memory footprint and compute cost. Using this technique, we show that our always-on wake-word detector model, RepCNN, provides a good trade-off between latency and accuracy during inference. RepCNN re-parameterized models are 43% more accurate than a uni-branch convolutional model while having the same runtime. RepCNN also meets the accuracy of complex architectures like BC-ResNet, while having 2x lesser peak memory usage and 10x faster runtime.

Interspeech

RepCNN: Micro-Sized, Mighty Models for Wakeword Detection

Prompt engineering is an iterative procedure that often requires extensive manual efforts to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and efficacious approach to provide LLMs with precise and tangible instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase an interactive tool called APE (Active Prompt Tuning) designed for refining prompts through human feedback. Drawing inspiration from active learning, APE iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt. 

Machine Learning
Research at Apple

Recent research

RepCNN: Micro-Sized, Mighty Models for Wakeword Detection

APE: Active Prompt Engineering - Identifying Informative Few-Shot Examples for LLMs

Research highlights

Introducing Apple’s On-Device and Server Foundation Models

Enhancing Paragraph Generation with a Latent Language Diffusion Model

All events

ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2024

Association for Computational Linguistics (ACL) 2024

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

RepCNN: Micro-Sized, Mighty Models for Wakeword Detection

APE: Active Prompt Engineering - Identifying Informative Few-Shot Examples for LLMs

Research highlights

Introducing Apple’s On-Device and Server Foundation Models

Enhancing Paragraph Generation with a Latent Language Diffusion Model

All events

ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2024

Association for Computational Linguistics (ACL) 2024

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple