3D Scene understanding has been an active area of machine learning (ML) research for more than a decade. More recently the release of <a href="https://developer.apple.com/augmented-reality/roomplan/" class="more">LiDAR sensor functionality</a> in Apple iPhone and iPad has begun a new era in scene understanding for the computer vision and developer communities. Fundamental research in scene understanding combined with the advances in ML can now impact everyday experiences. A variety of methods are addressing different parts of the challenge, like depth estimation, 3D reconstruction, instance segmentation, object detection, and more. Among these problems, creating a 3D floor plan is becoming key for many applications in augmented reality, robotics, e-commerce, games, and real estate.

3D Parametric Room Representation with RoomPlan

Scene analysis is an integral core technology that powers many features and experiences in the Apple ecosystem. From visual content search to powerful memories marking special occasions in one’s life, outputs (or "signals") produced by scene analysis are critical to how users interface with the photos on their devices. Deploying dedicated models for each of these individual features is inefficient as many of these models can benefit from sharing resources. We present how we developed Apple Neural Scene Analyzer (ANSA), a unified backbone to build and maintain scene analysis workflows in production. This was an important step towards enabling Apple to be among the first in the industry to deploy fully client-side scene analysis in 2016.

A Multi-Task Neural Architecture for On-Device Scene Analysis

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events. Check back for our accepted paper presentation times and locations. Visit the [NeurIPS 2022 website](https://neurips.cc/virtual/2022/calendar) for the full conference schedule.

### Monday November&nbsp;28
<ul class="links-stacked">
 <li>EXPO TALK</li>
 <li><a href="https://nips.cc/Conferences/2022/Schedule?showEvent=56490">Generative Understanding of 3D Scenes</a></li>
 <li>3:00 - 4:00 PM CST in Theater A</li>
 <li>Miguel Angel Bautista Martin</li>
<li><a href=https://nips.cc/Conferences/2022/Schedule?q=expo&type=Expo+Talk+Panel>See the full schedule of talks for Expo Talk day</a></li>
<li>WORKSHOP</li>
<li><a href=“https://blackinai.github.io/#/workshop/bai2022">Black in AI</a></li>
 <li>10:00 AM - 6:00 PM CST in Room 288 - 289</li>
<li>WORKSHOP</li>
<li><a href="https://www.latinxinai.org/neurips-2022">LatinX in AI</a></li>
 <li>8:00 AM - 6:00 PM CST in Room 298 - 299</li>
 <li>Samy Bengio will be participating in the mentoring hour as a mentor.</li>
<li>WORKSHOP</li>
<li><a href="https://www.queerinai.com/neurips-2022">Queer in AI</a></li>
 <li>9:00 AM - 6:00 PM CST in Room 283</li>
<li>WORKSHOP</li>
<li><a href="https://sites.google.com/view/wiml2022/">Women in Machine Learning</a></li>
 <li>7:30 AM - 6:00 PM CST in Hall I-2</li>
 <li>Lauren Araujo and Sefani Tadesse will be hosting a roundtable on Internships at Apple.</li>
</ul>

### Tuesday November&nbsp;29
<ul class="links-stacked">
 <li>POSTER SESSION 1</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/54631">Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation</a></li>
 <li>From 4:30 - 6:00 PM CST in Hall J #707</li>
 <li>Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, Jian Li</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/54286">Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures</a></li>
 <li>11:00 AM - 1:00 PM CST, Hall J #406</li> 
 <li><a href="https://neurips.cc/virtual/2022/poster/54640">Low-Rank Optimal Transport: Approximation, Statistics and Debiasing</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #507</li>
<li><a href="https://neurips.cc/virtual/2022/poster/54462">Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss</a></li>
 <li>From 11:00 AM - 1:00 PM CST in Hall J #331</li>
 <li><a href=" https://neurips.cc/virtual/2022/poster/53984">Subspace Recovery from Heterogeneous Data with Non-isotropic Noise</a></li>
 <li>11:00 AM - 1:00 PM CST, Hall J #531</li> 
</ul>

### Wednesday November&nbsp;30
<ul class="links-stacked">
 <li>POSTER SESSION 4</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/55624">FLAIR: Federated Learning Annotated Image Repository</a></li>
 <li>4:00 - 6:00 PM CST, Hall J #1028</li> 
</ul>

### Thursday December&nbsp;1
<ul class="links-stacked">
<li>POSTER SESSION 5</li>
<li><a href="https://neurips.cc/virtual/2022/poster/54862">GAUDI: A Neural Architect for Immersive 3D Scene Generation</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #427</li> 
<li><a href="https://neurips.cc/virtual/2022/poster/55742">MBW: Multi-view Bootstrapping in the Wild</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #1020</li> 
<li><a href="https://neurips.cc/virtual/2022/poster/52931">Mean Estimation with User-level Privacy</a></li>
 <li>4:00 - 6:00 PM CST in Hall J #730</li> 
<li><a href="/research/supervised-training">Supervised Training of Conditional Monge Maps</a></li>
 <li>4:00 - 6:00 PM CST, Hall J #1035</li> 
</ul>

### Friday December&nbsp;2
<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href="https://timeseriesforhealth.github.io/index">Learning from Time Series for Health</a></li>
 <li>9:00 AM - 5:10 PM CST, Room 392</li>
<li><a href="/research/masked-auto-encoder">Accepted Paper: MAEEG: Masked Autoencoder for EEG Representation Learning</a></li>
<li><a href="/research/modeling-heart-rate">Accepted Paper: Modeling Heart Rate Dynamics and Wearable Workout Data</a></li>
<li>Sana Tonekaboni, one of our Apple Scholars in AI/ML, co-organized this workshop.</li>
<li>WORKSHOP</li>
 <li><a href="https://www.cml-4-impact.vanderschaar-lab.com">Causal Machine Learning for Real-World Impact</a></li>
<li>9:00 AM - 5:45 PM CST, Room 296</li>
<li><a href="/research/large-scale-observational">Accepted Paper: Causal Analysis of a Large Scale Behavioral Intervention Nudge</a></li>
<li>WORKSHOP</li>
 <li><a href="https://hity-workshop.github.io/NeurIPS2022/">Has it Trained Yet? A Workshop for Algorithmic Efficiency in Practical Neural Network Training</a></li>
<li>8:30 AM - 5:00 PM CST, Theater B</li>
<li><a href=/research/slingshot-mechanism>Accepted Paper: The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon</a></li>
<li>WORKSHOP</li>
 <li><a href="https://neurips-hill.github.io/">Human in the Loop Learning</a></li>
<li>4:30 - 7:00 PM CST, Room 396</li>
<li>Accepted Paper: Symbol Guided Hindsight Priors for Reward Learning from Human Preferences</li>
<li><a href=/research/rewards-encoding>Accepted Paper: Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning</a></li>
</ul>

### Saturday December&nbsp;3
<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href="https://nips.cc/virtual/2022/workshop/50006">Transfer Learning for Natural Language Processing</a></li>
<li>8:50 AM - 5:00 PM CST, Theatre A</li>
<li><a href="/research/impact-of-language">Accepted Paper: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer</a></li>
<li>WORKSHOP</li>
 <li><a href="https://sites.google.com/view/RL4RealLife">Reinforcement Learning for Real Life</a></li>
<li>8:25 AM - 5:00 PM CST, Theatre A</li>
<li>Accepted Paper: CAGEd Wisdom: Applying Reinforcement Learning to Autonomous Cyber Defense</li>
<li>WORKSHOP</li>
<li><a href="https://sslneurips22.github.io/">Self-Supervised Learning: Theory and Practice</a></li>
<li>Workshop time TBC, Room 291 - 292</li>
<li><a href="/research/elastic-weight-consolidation">Accepted Paper: Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer</a></li>
<li>Accepted Paper: Contrastive Self-Supervised Learning for Skeleton Representations</li>
<li><a href="/research/homomorphic-self-supervised-learning">Accepted Paper: Homomorphic Self-Supervised Learning with Equivariant Networks</a></li>
<li>WORKSHOP</li>
<li><a href="https://sites.google.com/view/icbinb-2022/home">I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification</a></li>
<li>6:15 AM - 3:00 PM CST, Room 286</li>
<li><a href="/research/soft-pseudo-labeling">Accepted Paper: Continuous Soft Pseudo-Labeling in ASR</a></li>
<li>Arno Blaas co-organized this workshop, and Luca Zappella is a member of the coordination team.</li>
<li>WORKSHOP</li>
<li><a href="https://ml4physicalsciences.github.io/2022/">Machine Learning and the Physical Sciences</a></li>
<li>7:50 AM - 5:00 PM CST, Room 275 - 277</li>
<li>Accepted Paper: Robust Hybrid Learning with Expert Augmentations</li>
</ul>

### Monday December&nbsp;5
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="https://neurips.cc/virtual/2022/affinity-workshop/65366">Women in Machine Learning</a></li>
<li>9:30 AM - 4:30 PM CST, Held Virtually</li>
<li>Katherine Metcalf will give a talk on machine learning research at Apple during the workshop.</li>
</ul>
</div>

## Accepted Papers

### Conference Accepted Papers

[FLAIR: Federated Learning Annotated Image Repository](/research/flair)

*Congzheng Song, Filip Granqvist, Kunal Talwar*

[GAUDI: A Neural Architect for Immersive 3D Scene Generation](/research/gaudi)

*Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Joshua Susskind*

[Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](/research/analyzing-mitigating-repetitions)

*Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, Jian Li*

[Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures](/research/generalization-unseen-data-boolean)

*Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia, Jon Kleinberg, Aryo Lotfi, Maithra Raghu, Chiyuan Zhang*

[Low-rank Optimal Transport: Approximation, Statistics and Debiasing](/research/low-rank-optimal-transport)

*Meyer Scetbon, Marco Cuturi*

[MBW: Multi-view Bootstrapping in the Wild](/research/multi-view-bootstrapping)

*Mosam Dabhi, Chaoyang Wang, Tim Clifford, László Jeni, Ian Fasel, Simon Lucey*

[Mean Estimation with User-level Privacy under Data Heterogeneity](/research/mean-estimation)

*Rachel Cummings, Vitaly Feldman, Audra McMillan, Kunal Talwar*

[Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss](/research/privacy-noisy-stochastic)

*Jason Altschuler, Kunal Talwar*

[Subspace Recovery from Heterogeneous Data with Non-isotropic Noise](/research/subspace-recovery)

*John Duchi, Vitaly Feldman, Lunjia Hu, Kunal Talwar*

[Supervised Training of Conditional Monge Maps](/research/supervised-training)

*Charlotte Bunne, Andreas Krause, Marco Cuturi*

### Workshop Accepted Papers

[Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies](/research/beyond-cage)

*Melody Wolk, Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, and Adrian Webster*

[Large-Scale Observational Study of the Causal Effects of a Behavioral Health Nudge](/research/large-scale-observational)

*Achille Nazaret, Guillermo Sapiro*

[Modeling Heart Rate Response to Exercise with Wearable Data](/research/modeling-heart-rate)

*Achille Nazaret, Sana Tonekaboni, Greg Darnell, Shirley Ren, Guillermo Sapiro, Andrew C. Miller*

[How Soft Labels Cause Divergence in Speech Recognition for Continuous Pseudo-Labeling](/research/soft-pseudo-labeling)

*Tatiana Likhomanenko, Ronan Collobert, Navdeep Jaitly, Samy Bengio*

[Improving Generalization with Physical Equations](/research/improving-generalization)

*Antoine Wehenkel, Jens Behrmann, Hsiang Hsu, Guillermo Sapiro, Gilles Louppe, Joern-Henrik Jacobsen*

[Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning](/research/rewards-encoding)

*Katherine Metcalf, Miguel Sarabia and Barry-John Theobald*

[Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer](/research/elastic-weight-consolidation)

*Andrius Ovsianas, Jason Ramapuram, Dan Busbridge, Eeshan Gunesh Dhekane, Russ Webb*

[MAEEG: Masked Autoencoder for EEG Representation Learning](/research/masked-auto-encoder)

*Sherry Chien, Hanlin Goh, Chris Sandino, Joseph Yitan Cheng*

[Homomorphic Self-Supervised Learning](/research/homomorphic-self-supervised-learning)

*Thomas Anderson Keller, Xavier Suau, Luca Zappella*

[Improving Generalization with Physical Equations](/research/improving-generalization)

*Antoine Wehenkel, Jens Behrmann, Guillermo Sapiro, Joern Jacobsen, Gilles Louppe, Hsiang (Shawn) Hsu*

[The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](/research/slingshot-mechanism)

*Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Josh Susskind*

## Demos

**RoomPlan**

[RoomPlan](https://developer.apple.com/augmented-reality/roomplan/) technology allows the user to captures a room and it's defining objects in a parametric format within minutes. Capture progress is automatically displayed and easy to understand at a glance. The resulting room capture is provided as a parametric representation of the room and can optionally be exported to USD, USDA or USDZ formats. The technology is supported on any of the Apple devices (iPad and iPhone) that are equipped with LiDAR sensor.

**Information Intelligence**

This demo aims to illustrate the rich information extraction and understanding capabilities embedded in select Apple products that use our state of the art search and intelligence technologies to cater to user’s information needs. These features leverage deep text and image understanding capabilities coupled with powerful search technologies to surface accurate and relevant information for user’s queries.

All NeurIPS attendees are invited to stop by the Apple booth (booth number 603, located in Hall G of the Ernest N. Morial Convention Center) to experience these demos in person.

## Acknowledgements

Audra McMillan, Chong Wang, Congzheng Song, Devon Hjelm, Felix Bai, Hilal Asi, Jacob Yao, Jerome Bellegarda, Joe Futoma, Luca Zappella, Miguel Angel Bautista Martin, Sachin Agarwal, Sachin Mehta, and Tatiana Likhomanenko are reviewers for this conference. 

Marian Stewart Bartlett is the treasurer for this conference. 

Samy Bengio is a senior area chair and board member for this conference. 

Kunal Talwar and Laurent Dinh are area chairs for this conference. 

Arno Blaas co-organized the I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification workshop.

Luca Zappella is a member of the coordination team for the I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification workshop.

<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple is sponsoring the Neural Information Processing Systems Conference (NeurIPS), which will be held in New Orleans, Louisiana from November 28 to December 9. NeurIPS is a global conference focused on fostering the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.

NeurIPS 2022

All EMNLP attendees are invited to stop by the Apple booth (booth number 4, located near Conference Room B of the Abu Dhabi National Exhibition Centre) to chat with available recruiters and booth staff.

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events. Visit the [EMNLP 2022 website](https://2022.emnlp.org/program/) for the full conference schedule.

### Wednesday December&nbsp;7
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="http://www.winlp.org/winlp-2022-workshop">Widening Natural Language Processing</a></li>
 <li>8:45 AM - 4:50 PM GST, Capital Suite 21B</li>
 <li>Yunyao Li will be participating in the sponsor lunch during the workshop.</li>
</ul>

### Thursday December&nbsp;8
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="https://www.dashworkshops.org/emnlp-2022">Data Science with Human in the Loop</a></li>
 <li>8:55 AM - 5:30 PM GST, Capital Suite 10</li>
 <li>Accepted Paper: Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers</li>
<li>Yunyao Li co-organized this workshop, and Jeff Bigham, Katherine Luna, Shen Ma, Kun Qian, and Saptarshi Guha are members of the workshop program committee.</li>
</ul>


### Sunday December&nbsp;11
<ul class="links-stacked">
<li>POSTER SESSION</li>
<li><a href="/research/non-autoagressive-neural-machine">Non-Autoregressive Neural Machine Translation: A Call for Clarity</a></li>
 <li>11:00 AM - 12:30 PM GST, Atrium</li> 
</ul>
</div>

## Accepted Papers

### Conference Accepted Paper

[Non-Autoregressive Neural Machine Translation: A Call for Clarity](/research/non-autoagressive-neural-machine)

*Robin M. Schmidt, Telmo Pires, Stephan Peitz, Jonas Lööf*

## Demo

**Dictation Auto-Punctuation**

Dictation is one of the most heavily used input features on Apple devices. In our latest release, we have added auto-punctuation and emoji capabilities to dictation in 5 languages covering 13 locales. We support periods, commas, and question marks in the first version. These were identified as the top used punctuations that were explicitly dictated and also the punctuations that can be inferred from text alone. We have also added the ability to dictate frequently used emojis.

All EMNLP attendees are invited to stop by the Apple booth (booth number 4, located near Conference Room B of the Abu Dhabi National Exhibition Centre) to experience this demo in person.

## Acknowledgements

Yunyao Li co-organized the [Data Science with Human in the Loop workshop.](https://www.dashworkshops.org/emnlp-2022)

Jeff Bigham, Katherine Luna, Shen Ma, Kun Qian, and Saptarshi Guha are members of the [Data Science with Human in the Loop](https://www.dashworkshops.org/emnlp-2022) workshop program committee.

Rajen Chatterjee is co-organizing [Automatic Post Editing](https://www.statmt.org/wmt22/ape-task.html) shared task at the seventh conference on Machine Translation collocated with EMNLP 2022. 
 
<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple is sponsoring Empirical Methods in Natural Language Processing (EMNLP), which will be held in Abu Dhabi, United Arab Emirates from December 7 to 11. EMNLP is a leading conference focused on natural language processing.

EMNLP 2022

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

State-of-the-art automatic augmentation methods (e.g., AutoAugment and RandAugment) for visual recognition tasks diversify training data using a large set of augmentation operations. The range of magnitudes of many augmentation operations (e.g., brightness and contrast) is continuous. Therefore, to make search computationally tractable, these methods use fixed and manually-defined magnitude ranges for each operation, which may lead to sub-optimal policies. To answer the open question on the importance of magnitude ranges for each augmentation operation, we introduce RangeAugment that allows us to efficiently learn the range of magnitudes for individual as well as composite augmentation operations. RangeAugment uses an auxiliary loss based on image similarity as a measure to control the range of magnitudes of augmentation operations. As a result, RangeAugment has a single scalar parameter for search, image similarity, which we simply optimize via linear search. RangeAugment integrates seamlessly with any model and learns model- and task-specific augmentation policies. With extensive experiments on the ImageNet dataset across different networks, we show that RangeAugment achieves competitive performance to state-of-the-art automatic augmentation methods with 4-5 times fewer augmentation operations. Experimental results on semantic segmentation, object detection, foundation models, and knowledge distillation further shows RangeAugment's effectiveness.

RangeAugment: Efficient Online Augmentation with Range Learning

This paper was accepted at the "Human in the Loop Learning Workshop" at NeurIPS 2022. 

Specification of reward functions for Reinforcement Learning is a challenging task which is bypassed by the framework of Preference Based Learning methods which instead learn from preference labels on trajectory queries. These methods, however, still suffer from high requirements of preference labels and often would still achieve low reward recovery. We present the PRIOR framework that alleviates the issues of impractical number of queries to humans as well as poor reward recovery through computing priors about the reward function based on the environment dynamics and a surrogate preference classification model. We find that imposing these priors as soft constraints significantly reduces the queries made to the human in the loop and improves the overall reward recovery. Additionally, we investigate the use of an abstract state space for the computation of these priors to further improve the agent's performance.

Machine Learning
Research at Apple

Recent research

RangeAugment: Efficient Online Augmentation with Range Learning

Symbol Guided Hindsight Priors for Reward Learning from Human Preferences

Research articles

3D Parametric Room Representation with RoomPlan

A Multi-Task Neural Architecture for On-Device Scene Analysis

All events

NeurIPS 2022

EMNLP 2022

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

RangeAugment: Efficient Online Augmentation with Range Learning

Symbol Guided Hindsight Priors for Reward Learning from Human Preferences

Research articles

3D Parametric Room Representation with RoomPlan

A Multi-Task Neural Architecture for On-Device Scene Analysis

All events

NeurIPS 2022

EMNLP 2022

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple