Machine Translation (MT) enables people to connect with others and engage with content across language barriers. Grammatical gender presents a difficult challenge for these systems, as some languages require specificity for terms that can be ambiguous or neutral in other languages. For example, when translating the English word "nurse" into Spanish, one must decide whether the feminine "enfermera" or the masculine "enfermero" is appropriate. However, particularly when contextual clues are absent, such as in translating a single sentence, a model cannot determine which would be correct. This challenge is especially prevalent for many European languages, which often require gender specificity not only for professional titles, but also for terms like child, friend, and member, as well as sometimes for animals. Often, machine translation systems will be biased toward the gendered form most prevalent in their training data, but in addition to not necessarily providing an accurate translation for the user, this can inadvertently reinforce harmful societal stereotypes.

To address this problem and give people more control over grammatical gender in machine translations, we presented [Generating Gender Alternatives in Machine Translation](https://machinelearning.apple.com/research/generating-gender-alternatives) at GeBNLP 2024 (Workshop on Gender Bias in NLP). Our approach trains translation models that give users fine-grained control over how gendered entities are translated, without requiring any additional components or inference overhead. With our method, a single pass of translation inference delivers all grammatically correct alternatives for gendered terms, enabling a user to select the most appropriate for their context. In addition to publishing this work, we have [released training and test datasets](https://github.com/apple/ml-gendered-translation) to enable the broader ML community to more easily develop systems that allow such control over translation of gendered entities.

At Apple, we have also leveraged this research advancement to benefit users of the [Translate](https://apps.apple.com/us/app/translate/id1514844618) app, which uses the underlying method to power the "grammatical gender" feature. This feature currently allows users to choose the most appropriate translations from all possible combinations when translating English content with ambiguously gendered entities into Spanish, French, or Portuguese. 

## Addressing the Challenge of Grammatical Gender in Machine Translation

Prior work simplified the problem by producing only "all masculine" or "all feminine" translations, i.e., enforcing that all entities in the translation are of the same gender. However, requiring that both, "doctor" and "nurse" in the sentence "The doctor met the nurse" are either both male or both female obviously does not provide adequate flexibility.

A central challenge to giving users control over the grammatical gender of translated terms is the number of possible combinations. For (n) gendered entities, there are 2n possible translations. For the very brief example sentence above with only two entities, there are four possible Spanish translations, depending on the gender choices for "nurse" and "doctor;" if there were four entities, however, there would be 16 possible translations. Prior approaches have used specialized systems like "ambiguous entity detection" and/or "rewriters" at inference time to provide different gendered translations, but these systems have additional computational overhead at inference time, introduce additional latency, and do not scale well to handle all possible combinations.

Rather than using those systems at inference time, our approach instead uses them to distill training data for our translation models. During the distillation process, these specialized systems add gender structure and alignment information to the training data [(see Figure 1)](#figure1). Each gender-sensitive phrase is translated into a "gender structure" with both masculine and feminine forms. These gender structures are also aligned to the corresponding gender-ambiguous entity in the source sentence that controls its form. 

<Figure id="1" title="Figure 1" className="inline" id="figure1" caption={<>Figure 1: An example of gender structures and alignments added to the training data during distillation.</>} animate="fade-up">
 <Figure1 />
</Figure>

The different forms of gender-sensitive phrases like "El anfitrión / La anfitriona," "el pintor / la pintora" are structured together as gender structures. Each gender structure is aligned to its corresponding ambiguously gendered entity (shown here by color coding). Given a translation containing gender structures and alignments, translations corresponding to any combination of gender choices can be easily derived. 

By training on this distilled data with gender structures and alignments, our translation models learn to provide a single translation with the appropriate grammatical gender alternatives, without requiring any additional components that would add computational overhead and latency at inference time.

## New Resources for the Research Community 

To enable the research community to build translations systems that generate gender alternatives, we have [released](https://github.com/apple/ml-gendered-translation) supervised training datasets for five language pairs (English to Spanish, French, German, Russian, Portuguese) and evaluation benchmarks for six language pairs (those listed above, as well as English to Italian). These datasets contain gender ambiguity annotations in the English source sentences and gender structure and alignment annotations in the translations. 

## Giving Users Control of Grammatical Gender in the Translate App

At Apple, we’ve brought this advancement from research into production to enhance the [Translate](https://apps.apple.com/us/app/translate/id1514844618) app experience for users [(see Figure 2)](#figure2). With iOS 17, we shipped the "[grammatical gender"](https://support.apple.com/guide/iphone/translate-text-voice-and-conversations-iphd74cb450f/ios#:~:text=You%20can%20also%20select%20feminine,tap%20Next%20in%20the%20keyboard) feature, which is powered by the underlying method from this research. The feature currently allows users to select the appropriate grammatical gender of ambiguous entities when translating from English to Spanish, French, or Portuguese. Because our method trains a single translation model and doesn’t require multiple complete translations or other components at inference time, it does not incur additional computational overhead or result in additional latency for users, and was suitable for production application at scale. 

<Figure id="figure1" title="Figure 2" className="plain" animate="fade-up">
<ImageGallery
id="figure2"
items={[
{
src: 'https://mlr.cdn-apple.com/media/Fig_2_translate1_9e48588757.jpg',
caption:
'Figure 2A: How to control grammatical gender in the Translate app - finding the feature.'
},
{
src: 'https://mlr.cdn-apple.com/media/Fig_2_translate2_19cefdb711.jpg',
caption:
'Figure 2B: How to control grammatical gender in the Translate app - turning on the feature.'
},
{
src: 'https://mlr.cdn-apple.com/media/Fig_2_translate3_cead15c67c.jpg',
caption:
'Figure 2C: How to control grammatical gender in the Translate app - selecting gender.'
}
]}
/>
</Figure>

## Conclusion 

Translating content with ambiguously gendered entities into languages with gender-specific forms has been a challenge for machine translation, and it has often resulted in systems providing translations that reinforce societal stereotypes reflected in models' training data. Our research provides a new approach to training translation models, which enables them to provide all appropriate alternative translations for gendered entities, without requiring additional components or computational overhead at inference time. Using this method, we have given users of the [Translate](https://apps.apple.com/us/app/translate/id1514844618) app greater control over their translations, so that they can select the appropriate grammatical gender in cases of ambiguity when translating from English into several European languages. 

While we have made important progress on this problem in machine translation, there remain additional challenges for future research, including expanding to additional language pairs, including gender neutral forms, and appropriately addressing non-binary gender identities. Our hope is that our publication and the new resources we have shared help to accelerate progress within the broader research community to continue to improve translation systems for everyone.

## Acknowledgements

Many people contributed to this project including: Qin Gao, Sarthak Garg, Mozhdeh Gheini, Yi-Hsiu Liao, Tatiana Likhomanenko, Louie Livon-Bemel, Udhay Nallasamy, Alex Ovchinnikov, Matthias Paulik, Telmo Pessoa Pires, and Hendra Setiawan.

Improving How Machine Translations Handle Grammatical Gender Ambiguity

In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with [Private Cloud Compute](https://security.apple.com/blog/private-cloud-compute/) and running on Apple silicon servers — have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly. These two foundation models are part of a larger family of generative models created by Apple to support users and developers; this includes a coding model to build intelligence into Xcode, as well as a diffusion model to help users express themselves visually, for example, in the Messages app. We look forward to sharing more information soon on this broader set of models.


Update - July 29, 2024: The figures in this article have been updated to reflect the model versions and evaluations used in the technical report released today. For more detail, please see the paper: <a href="/research/apple-intelligence-foundation-language-models" className="more">Apple Intelligence Foundation Language Models</a>.


## Our Focus on Responsible AI Development
Apple Intelligence is designed with our core values at every step and built on a foundation of groundbreaking privacy innovations.

Additionally, we have created a set of Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them:

1. Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.
2. Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models.
3. Design with care: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback.
4. Protect privacy: We protect our users' privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users' private personal data or user interactions when training our foundation models.

These principles are reflected throughout the architecture that enables Apple Intelligence, connects features and tools with specialized models, and scans inputs and outputs to provide each feature with the information needed to function responsibly.

In the remainder of this overview, we provide details on decisions such as: how we develop models that are highly capable, fast, and power-efficient; how we approach training these models; how our adapters are fine-tuned for specific user needs; and how we evaluate model performance for both helpfulness and unintended harm.

<Figure alt="A diagram that shows the training steps for the Apple foundation models." id="figure1" caption={<>Figure 1: Modeling overview for the Apple foundation models.</>} animate="fade-up">
 <Image src="https://mlr.cdn-apple.com/media/Fig1_Responsible_AI_8bf7727ab5.png" alt="Modeling overview" />
</Figure>

## Pre-Training
Our foundation models are trained on [Apple's AXLearn framework](https://github.com/apple/axlearn), an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot. Web publishers have [the option to opt out](https://support.apple.com/en-us/119829) of the use of their web content for Apple Intelligence training with a data usage control.

We never use our users’ private personal data or user interactions when training our foundation models, and we apply filters to remove personally identifiable information like social security and credit card numbers that are publicly available on the Internet. We also filter profanity and other low-quality content to prevent its inclusion in the training corpus. In addition to filtering, we perform data extraction, deduplication, and the application of a model-based classifier to identify high quality documents.

## Post-Training
We find that data quality is essential to model success, so we utilize a hybrid data strategy in our training pipeline, incorporating both human-annotated and synthetic data, and conduct thorough data curation and filtering procedures. We have developed two novel algorithms in post-training: (1) a rejection sampling fine-tuning algorithm with teacher committee, and (2) a reinforcement learning from human feedback (RLHF) algorithm with mirror descent policy optimization and a leave-one-out advantage estimator. We find that these two algorithms lead to significant improvement in the model’s instruction-following quality.

## Optimization
In addition to ensuring our generative models are highly capable, we have used a range of innovative techniques to optimize them on-device and on our private cloud for speed and efficiency. We have applied an extensive set of optimizations for both first token and extended token inference performance.

Both the on-device and server models use grouped-query-attention. We use shared input and output vocab embedding tables to reduce memory requirements and inference cost. These shared embedding tensors are mapped without duplications. The on-device model uses a vocab size of 49K, while the server model uses a vocab size of 100K, which includes additional language and technical tokens.

For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. To maintain model quality, we developed a new framework using LoRA adapters that incorporates a mixed 2-bit and 4-bit configuration strategy &mdash; averaging 3.7 bits-per-weight &mdash; to achieve the same accuracy as the uncompressed models. More aggressively, the model can be compressed to 3.5 bits-per-weight without significant quality loss.

Additionally, we use an interactive model latency and power analysis tool, [Talaria](/research/talaria), to better guide the bit rate selection for each operation. We also utilize activation quantization and embedding quantization, and have developed an approach to enable efficient Key-Value (KV) cache update on our neural engines.

With this set of optimizations, on iPhone 15 Pro we are able to reach time-to-first-token latency of about 0.6 millisecond per prompt token, and a generation rate of 30 tokens per second. Notably, this performance is attained before employing token speculation techniques, from which we see further enhancement on the token generation rate.

## Model Adaptation
Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks. For our models we adapt the attention matrices, the attention projection matrix, and the fully connected layers in the point-wise feedforward networks for a suitable set of the decoding layers of the transformer architecture.

By fine-tuning only the adapter layers, the original parameters of the base pre-trained model remain unchanged, preserving the general knowledge of the model while tailoring the adapter layers to support specific tasks.

<Figure alt="A diagram of the Apple Foundation Model and Adapters." id="figure2" caption={<>Figure 2: Adapters are small collections of model weights that are overlaid onto the common base foundation model. They can be dynamically loaded and swapped &mdash; giving the foundation model the ability to specialize itself on-the-fly for the task at hand. Apple Intelligence includes a broad set of adapters, each fine-tuned for a specific feature. It’s an efficient way to scale the capabilities of our foundation model.</>} className="overflow-visible" animate="fade-up">
 <Figure2 src="https://mlr.cdn-apple.com/video/Fig2_Adapters_9f087bc5c5.mp4" />
</Figure>

We represent the values of the adapter parameters using 16 bits, and for the ~3 billion parameter on-device model, the parameters for a rank 16 adapter typically require 10s of megabytes. The adapter models can be dynamically loaded, temporarily cached in memory, and swapped — giving our foundation model the ability to specialize itself on the fly for the task at hand while efficiently managing memory and guaranteeing the operating system's responsiveness.

To facilitate the training of the adapters, we created an efficient infrastructure that allows us to rapidly retrain, test, and deploy adapters when either the base model or the training data gets updated. The adapter parameters are initialized using the accuracy-recovery adapter introduced in the Optimization section.

## Performance and Evaluation
Our focus is on delivering generative models that can enable users to communicate, work, express themselves, and get things done across their Apple products. When benchmarking our models, we focus on human evaluation as we find that these results are highly correlated to user experience in our products. We conducted performance evaluations on both feature-specific adapters and the foundation models.

To illustrate our approach, we look at how we evaluated our adapter for summarization. As product requirements for summaries of emails, messages, and notifications differ in subtle but important ways, we fine-tune accuracy-recovery low-rank (LoRA) adapters on top of the palletized model to meet these specific requirements. Our training data is based on synthetic summaries generated from bigger server models, filtered by a rejection sampling strategy that keeps only the high quality summaries.

To evaluate the product-specific summarization, we use a set of 750 responses carefully sampled for each use case. These evaluation datasets emphasize a diverse set of inputs that our product features are likely to face in production, and include a stratified mixture of single and stacked documents of varying content types and lengths. As product features, it was important to evaluate performance against datasets that are representative of real use cases. We find that overall, our models with adapters generate better summaries than a comparable model.

As part of responsible development, we identified and evaluated specific risks inherent to summarization. For example, summaries occasionally remove important nuance or other details in ways that are undesirable. However, we found that the summarization adapter did not amplify sensitive content in over 99% of targeted adversarial examples. We continue to adversarially probe to identify unknown harms and expand our evaluations to help guide further improvements.

<Figure 
 id="figure3" 
 caption={<>Figure 3: Ratio of "good" and "poor" responses for three summarization use cases relative to all responses. Summaries are classified as "good", "neutral", "poor" given the grader's scores across five dimensions. A result is classified as "good" if all of the dimensions are good (higher is better). A result is classified as "poor" if any of the dimensions are poor (lower is better). Our models with adapters overall generate better summaries than comparable models.</>}
 animate="fade-up">
 <Figure3 title="Human Satisfaction with Summarization Feature" />
</Figure>

In addition to evaluating feature specific performance powered by foundation models and adapters, we evaluate both the on-device and server-based models’ general capabilities. We utilize a comprehensive evaluation set of real-world prompts to test the general model capabilities. These prompts are diverse across different difficulty levels and cover major categories such as brainstorming, classification, closed question answering, coding, extraction, mathematical reasoning, open question answering, rewriting, safety, summarization, and writing.

We compare our models with both open-source models (Phi-3, Gemma, Mistral, DBRX, Llama) and commercial models of comparable size (GPT-3.5, GPT-4)<a href="#footnotes">1</a>. We find that our models are preferred by human graders over most comparable competitor models. On this benchmark, our on-device model, with ~3B parameters, outperforms larger models including Phi-3-mini, Mistral-7B, Gemma-7B, and Llama-3-8B. Our server model compares favorably to DBRX-Instruct, Mixtral-8x22B, GPT-3.5, and Llama-3-70B while being highly efficient.

<Figure 
 id="figure4" 
 caption={<>Figure 4: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models. We find that our models are preferred by human graders.</>}
 animate="fade-up">
 <Figure4 title="Human Evaluation" />
</Figure>

We use a set of diverse adversarial prompts to test the model performance on harmful content, sensitive topics, and factuality. We measure the violation rates of each model as evaluated by human graders on this evaluation set, with a lower number being desirable. Both the on-device and server models are robust when faced with adversarial prompts, achieving violation rates lower than open-source and commercial models.

<Figure 
 id="figure5" 
 caption={<>Figure 5: Fraction of violating responses for harmful content, sensitive topics, and factuality (lower is better). Our models are robust when faced with adversarial prompts.</>} 
 animate="fade-up">
 <Figure5 title="Human Evaluation of Output Harmfulness" />
</Figure>

Our models are preferred by human graders as safe and helpful over competitor models for these prompts. However, considering the broad capabilities of large language models, we understand the limitation of our safety benchmark. We are actively conducting both manual and automatic red-teaming with internal and external teams to continue evaluating our models' safety.

<Figure 
 id="figure6" 
 caption={<>Figure 6: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models on safety prompts. Human graders found our responses safer and more helpful.</>}
 animate="fade-up">
 <Figure6 title="Human Preference Evaluation on Safety Prompts" />
</Figure>

To further evaluate our models, we use the Instruction-Following Eval (IFEval) benchmark to compare their instruction-following capabilities with models of comparable size. The results suggest that both our on-device and server model follow detailed instructions better than the open-source and commercial models of comparable size.

<Figure 
 id="figure7" 
 caption={<>Figure 7: Instruction-following capability (measured with IFEval) for Apple's foundation models and models of comparable size (higher is better).</>}
 animate="fade-up">
 <Figure7 title="IFEval Benchmarks" />
</Figure>

We evaluate our models’ writing ability on our internal summarization and composition benchmarks, consisting of a variety of writing instructions. These results do not refer to our feature-specific adapter for summarization (seen in [Figure 3](#figure3)), nor do we have an adapter focused on composition.

<Figure 
 id="figure8" 
 caption={<>Figure 8: Writing ability on internal summarization and composition benchmarks (higher is better).</>}
 animate="fade-up">
 <Figure8 title="Writing Benchmarks" />
</Figure>

## Conclusion
The Apple foundation models and adapters introduced at WWDC24 underlie Apple Intelligence, the new personal intelligence system that is integrated deeply into iPhone, iPad, and Mac, and enables powerful capabilities across language, images, actions, and personal context. Our models have been created with the purpose of helping users do everyday activities across their Apple products, and developed responsibly at every stage and guided by Apple’s core values. We look forward to sharing more information soon on our broader family of generative models, including language, diffusion, and coding models.

{<h2 id="footnotes">Footnotes</h2>}
[1] We compared against the following model versions: gpt-3.5-turbo-0125, gpt-4-0125-preview, Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.2, Mixtral-8x22B-Instruct-v0.1, Gemma-1.1-2B, Gemma-1.1-7B, Llama-3-8B-Instruct, and Llama-3-70B-Instruct. The open-source and Apple models are evaluated in bfloat16 precision.

At the 2024 <a href="https://developer.apple.com/wwdc24/" class="more">Worldwide Developers Conference</a>, we introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia.
Apple Intelligence is comprised of multiple highly-capable generative models that are specialized for our users’ everyday tasks, and can adapt on the fly for their current activity. The foundation models built into Apple Intelligence have been fine-tuned for user experiences such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across apps.

Introducing Apple’s On-Device and Server Foundation Models

<div class="callout">

## Schedule

Stop by the Apple booth #34, in the Allianz MiCo Convention Center during exhibition hours (all times GMT+2):
- Tuesday, October 1 — Thursday, October 3: 09:00-18:30
- Friday, October 4: 09:00-12:30



### Sunday, September&nbsp;29

<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://sites.google.com/view/avgenl">AVGenL: Audio-Visual Generation and Learning 2024</a></li>
 <li>14:00 - 18:00, Suite 9</li>
 <li>
 <ul>
 <li>POSTER</li>
 <li><a href="/research/acl-pseudo-labeling">AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition</a></li>
 <li>15:40 - 16:20</li>
 <li>Andrew Rouditchenko, Ronan Collobert, Tatiana Likhomanenko</li>
 </ul>
 </li>
</ul>

### Monday, September&nbsp;30

<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://vision-based-industrial-inspection.github.io/eccv-24/">2nd workshop on Vision-based InduStrial InspectiON (VISION)</a></li>
 <li>09:00 - 13:00, Tower Lounge</li>
 <li>
 <ul>
 <li>POSTER</li>
 <li>Synth4Seg - Learning Defect Data Synthesis for Defect Segmentation using Bi-level Optimization</li>
 <li>10:40 - 11:40</li>
 <li>Shancong Mou, Raviteja Vemulapalli, Shiyu Li, Andy Liu, C Thomas, Meng Cao, Felix Bai, Oncel Tuzel, Ping Huang, Jiulong Shan, Jianjun Shi</li>
 </ul>
 </li>
</ul>

### Tuesday, October&nbsp;1

<ul class="links-stacked">
 <li>POSTER</li>
 <li><a href="https://machinelearning.apple.com/research/ferretui-mobile">Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs</a></li>
 <li>16:30 - 18:30, Poster Session 2</li>
 <li>Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeff Nichols, Yinfei Yang, Zhe Gan</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
<li><a href="https://machinelearning.apple.com/research/conditional-loradapter">CTRLorALTer: Conditional LoRAdapter for Efficient Zero-Shot Control & Altering of T2I Models</a></li>
<li>16:30 - 18:30, Poster Session 2</li>
<li>Nick Stracke, Stefan Andreas Baumann, Josh Susskind, Miguel Angel Bautista Martin, Björn Ommer</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
<li><a href="https://machinelearning.apple.com/research/mm1-methods-analysis-insights">MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training</a></li>
<li>10:30 - 12:30, Poster Session 1</li>
<li>Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier Biard, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang</li>
</ul>

### Wednesday, October&nbsp;2

<ul class="links-stacked">
 <li>POSTER</li>
 <li><a href="https://machinelearning.apple.com/research/veclip">VeCLIP: Improving CLIP Training via Visual-enriched Captions</a></li>
 <li>16:30 - 18:30, Poster Session 4</li>
 <li>Jeff Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Felix Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao</li>
</ul>
</div>

## Accepted Papers

[AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition](https://machinelearning.apple.com/research/acl-pseudo-labeling)

*Andrew Rouditchenko, Ronan Collobert, Tatiana Likhomanenko*

[CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models](https://machinelearning.apple.com/research/conditional-loradapter)

*Nick Stracke, Stefan Andreas Baumann, Josh Susskind, Miguel Angel Bautista Martin, Björn Ommer*

[Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://machinelearning.apple.com/research/ferretui-mobile)

*Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeff Nichols, Yinfei Yang, Zhe Gan*

[MM1: Methods, Analysis & Insights from Multimodal LLM Pre-Training](https://machinelearning.apple.com/research/mm1-methods-analysis-insights)

*Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier Biard, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang*

Synth4Seg - Learning Defect Data Synthesis for Defect Segmentation Using Bi-level Optimization

*Shancong Mou, Raviteja Vemulapalli, Shiyu Li, Andy Liu, C Thomas, Meng Cao, Felix Bai, Oncel Tuzel, Ping Huang, Jiulong Shan, Jianjun Shi*

[VeCLIP: Improving CLIP Training via Visual-Enriched Captions](https://machinelearning.apple.com/research/veclip)

*Jeff Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Felix Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao*

## Acknowledgements

Stephan Richter is a main conference area chair.

Alaa El-Nouby, Hadi Pour Ansari, Pavan Kumar Anasosalu Vasu, Raviteja Vemulapalli, and Yusu Qian are main conference reviewers.

For the [2nd workshop on Vision-based InduStrial InspectiON (VISION):](https://vision-based-industrial-inspection.github.io/eccv-24/)

Alexander Wong, C Thomas, Carrie Yu, Javad Shafiee, Jeff Lai, and Tatiana Likhomanenko are co-organizers.

Shiyu Li and Yuxuan Liu are workshop chairs.

Raviteja Vemulapalli is a keynote speaker.

Vimal Thilak is a workshop reviewer.

Apple is presenting new research at the <a href="https://eccv.ecva.net" target="_blank" aria-label="European Conference on Computer Vision (ECCV) - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">European Conference on Computer Vision (ECCV)</a>, which takes place in person in Milan, Italy, from September 29 - October 4. We are proud to again sponsor the biennial conference, which brings together the scientific and industrial research communities around ML and computer vision. Below is an overview of Apple’s participation at ECCV 2024.

European Conference on Computer Vision (ECCV) 2024

Apple Workshop on Privacy-Preserving Machine Learning 2024 

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

While server-side Large Language Models (LLMs) demonstrate proficiency in tool integration and complex reasoning, deploying Small Language Models (SLMs) directly on devices brings opportunities to improve latency and privacy but also introduces unique challenges for accuracy and memory. We introduce CAMPHOR, an innovative on-device SLM multi-agent framework designed to handle multiple user inputs and reason over personal context locally, ensuring privacy is maintained. CAMPHOR employs a hierarchical architecture where a high-order reasoning agent decomposes complex tasks and coordinates expert agents responsible for personal context retrieval, tool interaction, and dynamic plan generation. By implementing parameter sharing across agents and leveraging prompt compression, we significantly reduce model size, latency, and memory usage. To validate our approach, we present a novel dataset capturing multi-agent task trajectories centered on personalized mobile assistant use cases. Our experiments reveal that fine-tuned SLM agents not only surpass closed-source LLMs in task completion F1 by 35\% but also eliminate the need for server device communication, all while enhancing privacy.

CAMPHOR: Collaborative Agents for Multi-Input Planning and High-Order Reasoning On Device

Optimal transport (OT) has profoundly impacted machine learning by providing theoretical and computational tools to realign datasets. In this context, given two large point clouds of sizes $n$ and $m$ in $\mathbb{R}^d$, entropic OT (EOT) solvers have emerged as the most reliable tool to either solve the Kantorovich problem and output a $n\times m$ coupling matrix, or to solve the Monge problem and learn a vector-valued push-forward map. While the robustness of EOT couplings/maps makes them a go-to choice in practical applications, EOT solvers remain difficult to tune because of a small but influential set of hyperparameters, notably the omnipresent entropic regularization strength $\varepsilon$. Setting $\varepsilon$ can be difficult, as it simultaneously impacts various performance metrics, such as compute speed, statistical performance, generalization, and bias. In this work, we propose a new class of EOT solvers (ProgOT),
that can estimate both plans and transport maps.
We take advantage of several opportunities to optimize the computation of EOT solutions by dividing mass displacement using a time discretization, borrowing inspiration from dynamic OT formulations (McCann 1997), and conquering each of these steps using EOT with properly scheduled parameters. We provide experimental evidence demonstrating that ProgOT is a faster and more robust alternative to EOT solvers when computing couplings and maps at large scales, even outperforming neural network-based approaches. We also prove the statistical consistency of ProgOT when estimating OT maps. 


Machine Learning
Research at Apple

Recent research

CAMPHOR: Collaborative Agents for Multi-Input Planning and High-Order Reasoning On Device

Progressive Entropic Optimal Transport Solvers

Research highlights

Improving How Machine Translations Handle Grammatical Gender Ambiguity

Introducing Apple’s On-Device and Server Foundation Models

All events

European Conference on Computer Vision (ECCV) 2024

Apple Workshop on Privacy-Preserving Machine Learning 2024

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

CAMPHOR: Collaborative Agents for Multi-Input Planning and High-Order Reasoning On Device

Progressive Entropic Optimal Transport Solvers

Research highlights

Improving How Machine Translations Handle Grammatical Gender Ambiguity

Introducing Apple’s On-Device and Server Foundation Models

All events

European Conference on Computer Vision (ECCV) 2024

Apple Workshop on Privacy-Preserving Machine Learning 2024

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple