3D Scene understanding has been an active area of machine learning (ML) research for more than a decade. More recently the release of <a href="https://developer.apple.com/augmented-reality/roomplan/" class="more">LiDAR sensor functionality</a> in Apple iPhone and iPad has begun a new era in scene understanding for the computer vision and developer communities. Fundamental research in scene understanding combined with the advances in ML can now impact everyday experiences. A variety of methods are addressing different parts of the challenge, like depth estimation, 3D reconstruction, instance segmentation, object detection, and more. Among these problems, creating a 3D floor plan is becoming key for many applications in augmented reality, robotics, e-commerce, games, and real estate.

3D Parametric Room Representation with RoomPlan

Scene analysis is an integral core technology that powers many features and experiences in the Apple ecosystem. From visual content search to powerful memories marking special occasions in one’s life, outputs (or "signals") produced by scene analysis are critical to how users interface with the photos on their devices. Deploying dedicated models for each of these individual features is inefficient as many of these models can benefit from sharing resources. We present how we developed Apple Neural Scene Analyzer (ANSA), a unified backbone to build and maintain scene analysis workflows in production. This was an important step towards enabling Apple to be among the first in the industry to deploy fully client-side scene analysis in 2016.

A Multi-Task Neural Architecture for On-Device Scene Analysis

<div class="callout">

## Schedule

Below was the schedule of Apple sponsored workshops and events at NeurIPS 2022. 

### Monday November&nbsp;28
<ul class="links-stacked">
 <li>EXPO TALK</li>
 <li><a href="https://nips.cc/Conferences/2022/Schedule?showEvent=56490">Generative Understanding of 3D Scenes</a></li>
 <li>3:00 - 4:00 PM CST in Theater A</li>
 <li>Miguel Angel Bautista Martin</li>
<li><a href=https://nips.cc/Conferences/2022/Schedule?q=expo&type=Expo+Talk+Panel>See the full schedule of talks for Expo Talk day</a></li>
<li>WORKSHOP</li>
<li><a href=“https://blackinai.github.io/#/workshop/bai2022">Black in AI</a></li>
 <li>10:00 AM - 6:00 PM CST in Room 288 - 289</li>
<li>WORKSHOP</li>
<li><a href="https://www.latinxinai.org/neurips-2022">LatinX in AI</a></li>
 <li>8:00 AM - 6:00 PM CST in Room 298 - 299</li>
 <li>Samy Bengio will be participating in the mentoring hour as a mentor.</li>
<li>WORKSHOP</li>
<li><a href="https://www.queerinai.com/neurips-2022">Queer in AI</a></li>
 <li>9:00 AM - 6:00 PM CST in Room 283</li>
<li>WORKSHOP</li>
<li><a href="https://sites.google.com/view/wiml2022/">Women in Machine Learning</a></li>
 <li>7:30 AM - 6:00 PM CST in Hall I-2</li>
 <li>Lauren Araujo and Sefani Tadesse will be hosting a roundtable on Internships at Apple.</li>
</ul>

### Tuesday November&nbsp;29
<ul class="links-stacked">
 <li>POSTER SESSION 1</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/54631">Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation</a></li>
 <li>From 4:30 - 6:00 PM CST in Hall J #707</li>
 <li>Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, Jian Li</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/54286">Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures</a></li>
 <li>11:00 AM - 1:00 PM CST, Hall J #406</li> 
 <li><a href="https://neurips.cc/virtual/2022/poster/54640">Low-Rank Optimal Transport: Approximation, Statistics and Debiasing</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #507</li>
<li><a href="https://neurips.cc/virtual/2022/poster/54462">Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss</a></li>
 <li>From 11:00 AM - 1:00 PM CST in Hall J #331</li>
 <li><a href=" https://neurips.cc/virtual/2022/poster/53984">Subspace Recovery from Heterogeneous Data with Non-isotropic Noise</a></li>
 <li>11:00 AM - 1:00 PM CST, Hall J #531</li> 
</ul>

### Wednesday November&nbsp;30
<ul class="links-stacked">
 <li>POSTER SESSION 4</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/55624">FLAIR: Federated Learning Annotated Image Repository</a></li>
 <li>4:00 - 6:00 PM CST, Hall J #1028</li> 
</ul>

### Thursday December&nbsp;1
<ul class="links-stacked">
<li>POSTER SESSION 5</li>
<li><a href="https://neurips.cc/virtual/2022/poster/54862">GAUDI: A Neural Architect for Immersive 3D Scene Generation</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #427</li> 
<li><a href="https://neurips.cc/virtual/2022/poster/55742">MBW: Multi-view Bootstrapping in the Wild</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #1020</li> 
<li><a href="https://neurips.cc/virtual/2022/poster/52931">Mean Estimation with User-level Privacy</a></li>
 <li>4:00 - 6:00 PM CST in Hall J #730</li> 
<li><a href="/research/supervised-training">Supervised Training of Conditional Monge Maps</a></li>
 <li>4:00 - 6:00 PM CST, Hall J #1035</li> 
</ul>

### Friday December&nbsp;2
<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href="https://timeseriesforhealth.github.io/index">Learning from Time Series for Health</a></li>
 <li>9:00 AM - 5:10 PM CST, Room 392</li>
<li><a href="/research/masked-auto-encoder">Accepted Paper: MAEEG: Masked Autoencoder for EEG Representation Learning</a></li>
<li><a href="/research/modeling-heart-rate">Accepted Paper: Modeling Heart Rate Dynamics and Wearable Workout Data</a></li>
<li>Sana Tonekaboni, one of our Apple Scholars in AI/ML, co-organized this workshop.</li>
<li>WORKSHOP</li>
 <li><a href="https://www.cml-4-impact.vanderschaar-lab.com">Causal Machine Learning for Real-World Impact</a></li>
<li>9:00 AM - 5:45 PM CST, Room 296</li>
<li><a href="/research/large-scale-observational">Accepted Paper: Causal Analysis of a Large Scale Behavioral Intervention Nudge</a></li>
<li>WORKSHOP</li>
 <li><a href="https://hity-workshop.github.io/NeurIPS2022/">Has it Trained Yet? A Workshop for Algorithmic Efficiency in Practical Neural Network Training</a></li>
<li>8:30 AM - 5:00 PM CST, Theater B</li>
<li><a href=/research/slingshot-mechanism>Accepted Paper: The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon</a></li>
<li>WORKSHOP</li>
 <li><a href="https://neurips-hill.github.io/">Human in the Loop Learning</a></li>
<li>4:30 - 7:00 PM CST, Room 396</li>
<li>Accepted Paper: Symbol Guided Hindsight Priors for Reward Learning from Human Preferences</li>
<li><a href=/research/rewards-encoding>Accepted Paper: Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning</a></li>
</ul>

### Saturday December&nbsp;3
<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href="https://nips.cc/virtual/2022/workshop/50006">Transfer Learning for Natural Language Processing</a></li>
<li>8:50 AM - 5:00 PM CST, Theatre A</li>
<li><a href="/research/impact-of-language">Accepted Paper: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer</a></li>
<li>WORKSHOP</li>
 <li><a href="https://sites.google.com/view/RL4RealLife">Reinforcement Learning for Real Life</a></li>
<li>8:25 AM - 5:00 PM CST, Theatre A</li>
<li>Accepted Paper: CAGEd Wisdom: Applying Reinforcement Learning to Autonomous Cyber Defense</li>
<li>WORKSHOP</li>
<li><a href="https://sslneurips22.github.io/">Self-Supervised Learning: Theory and Practice</a></li>
<li>Workshop time TBC, Room 291 - 292</li>
<li><a href="/research/elastic-weight-consolidation">Accepted Paper: Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer</a></li>
<li>Accepted Paper: Contrastive Self-Supervised Learning for Skeleton Representations</li>
<li><a href="/research/homomorphic-self-supervised-learning">Accepted Paper: Homomorphic Self-Supervised Learning with Equivariant Networks</a></li>
<li>WORKSHOP</li>
<li><a href="https://sites.google.com/view/icbinb-2022/home">I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification</a></li>
<li>6:15 AM - 3:00 PM CST, Room 286</li>
<li><a href="/research/soft-pseudo-labeling">Accepted Paper: Continuous Soft Pseudo-Labeling in ASR</a></li>
<li>Arno Blaas co-organized this workshop, and Luca Zappella is a member of the coordination team.</li>
<li>WORKSHOP</li>
<li><a href="https://ml4physicalsciences.github.io/2022/">Machine Learning and the Physical Sciences</a></li>
<li>7:50 AM - 5:00 PM CST, Room 275 - 277</li>
<li>Accepted Paper: Robust Hybrid Learning with Expert Augmentations</li>
</ul>

### Monday December&nbsp;5
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="https://neurips.cc/virtual/2022/affinity-workshop/65366">Women in Machine Learning</a></li>
<li>9:30 AM - 4:30 PM CST, Held Virtually</li>
<li>Katherine Metcalf will give a talk on machine learning research at Apple during the workshop.</li>
</ul>
</div>

## Accepted Papers

### Conference Accepted Papers

[FLAIR: Federated Learning Annotated Image Repository](/research/flair)

*Congzheng Song, Filip Granqvist, Kunal Talwar*

[GAUDI: A Neural Architect for Immersive 3D Scene Generation](/research/gaudi)

*Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Joshua Susskind*

[Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](/research/analyzing-mitigating-repetitions)

*Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, Jian Li*

[Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures](/research/generalization-unseen-data-boolean)

*Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia, Jon Kleinberg, Aryo Lotfi, Maithra Raghu, Chiyuan Zhang*

[Low-rank Optimal Transport: Approximation, Statistics and Debiasing](/research/low-rank-optimal-transport)

*Meyer Scetbon, Marco Cuturi*

[MBW: Multi-view Bootstrapping in the Wild](/research/multi-view-bootstrapping)

*Mosam Dabhi, Chaoyang Wang, Tim Clifford, László Jeni, Ian Fasel, Simon Lucey*

[Mean Estimation with User-level Privacy under Data Heterogeneity](/research/mean-estimation)

*Rachel Cummings, Vitaly Feldman, Audra McMillan, Kunal Talwar*

[Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss](/research/privacy-noisy-stochastic)

*Jason Altschuler, Kunal Talwar*

[Subspace Recovery from Heterogeneous Data with Non-isotropic Noise](/research/subspace-recovery)

*John Duchi, Vitaly Feldman, Lunjia Hu, Kunal Talwar*

[Supervised Training of Conditional Monge Maps](/research/supervised-training)

*Charlotte Bunne, Andreas Krause, Marco Cuturi*

### Workshop Accepted Papers

[Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies](/research/beyond-cage)

*Melody Wolk, Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, and Adrian Webster*

[Large-Scale Observational Study of the Causal Effects of a Behavioral Health Nudge](/research/large-scale-observational)

*Achille Nazaret, Guillermo Sapiro*

[Modeling Heart Rate Response to Exercise with Wearable Data](/research/modeling-heart-rate)

*Achille Nazaret, Sana Tonekaboni, Greg Darnell, Shirley Ren, Guillermo Sapiro, Andrew C. Miller*

[How Soft Labels Cause Divergence in Speech Recognition for Continuous Pseudo-Labeling](/research/soft-pseudo-labeling)

*Tatiana Likhomanenko, Ronan Collobert, Navdeep Jaitly, Samy Bengio*

[Improving Generalization with Physical Equations](/research/improving-generalization)

*Antoine Wehenkel, Jens Behrmann, Hsiang Hsu, Guillermo Sapiro, Gilles Louppe, Joern-Henrik Jacobsen*

[Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning](/research/rewards-encoding)

*Katherine Metcalf, Miguel Sarabia and Barry-John Theobald*

[Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer](/research/elastic-weight-consolidation)

*Andrius Ovsianas, Jason Ramapuram, Dan Busbridge, Eeshan Gunesh Dhekane, Russ Webb*

[MAEEG: Masked Autoencoder for EEG Representation Learning](/research/masked-auto-encoder)

*Sherry Chien, Hanlin Goh, Chris Sandino, Joseph Yitan Cheng*

[Homomorphic Self-Supervised Learning](/research/homomorphic-self-supervised-learning)

*Thomas Anderson Keller, Xavier Suau, Luca Zappella*

[Improving Generalization with Physical Equations](/research/improving-generalization)

*Antoine Wehenkel, Jens Behrmann, Guillermo Sapiro, Joern Jacobsen, Gilles Louppe, Hsiang (Shawn) Hsu*

[The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](/research/slingshot-mechanism)

*Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Josh Susskind*

## Demos

**RoomPlan**

[RoomPlan](https://developer.apple.com/augmented-reality/roomplan/) technology allows the user to captures a room and it's defining objects in a parametric format within minutes. Capture progress is automatically displayed and easy to understand at a glance. The resulting room capture is provided as a parametric representation of the room and can optionally be exported to USD, USDA or USDZ formats. The technology is supported on any of the Apple devices (iPad and iPhone) that are equipped with LiDAR sensor.

**Information Intelligence**

This demo aims to illustrate the rich information extraction and understanding capabilities embedded in select Apple products that use our state of the art search and intelligence technologies to cater to user’s information needs. These features leverage deep text and image understanding capabilities coupled with powerful search technologies to surface accurate and relevant information for user’s queries.

All NeurIPS attendees are invited to stop by the Apple booth (booth number 603, located in Hall G of the Ernest N. Morial Convention Center) to experience these demos in person.

## Acknowledgements

Audra McMillan, Chong Wang, Congzheng Song, Devon Hjelm, Felix Bai, Hilal Asi, Jacob Yao, Jerome Bellegarda, Joe Futoma, Luca Zappella, Miguel Angel Bautista Martin, Sachin Agarwal, Sachin Mehta, and Tatiana Likhomanenko are reviewers for this conference. 

Marian Stewart Bartlett is the treasurer for this conference. 

Samy Bengio is a senior area chair and board member for this conference. 

Kunal Talwar and Laurent Dinh are area chairs for this conference. 

Arno Blaas co-organized the I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification workshop.

Luca Zappella is a member of the coordination team for the I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification workshop.

<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple sponsored the Neural Information Processing Systems Conference (NeurIPS), which was held in New Orleans, Louisiana from November 28 to December 9. NeurIPS is a global conference focused on fostering the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.

NeurIPS 2022

All EMNLP attendees were invited to stop by the Apple booth (booth number 4, located near Conference Room B of the Abu Dhabi National Exhibition Centre) to chat with available recruiters and booth staff.

<div class="callout">

## Schedule

Below was the schedule of Apple sponsored workshops and events. 

### Wednesday December&nbsp;7
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="http://www.winlp.org/winlp-2022-workshop">Widening Natural Language Processing</a></li>
 <li>8:45 AM - 4:50 PM GST, Capital Suite 21B</li>
 <li>Yunyao Li will be participating in the sponsor lunch during the workshop.</li>
</ul>

### Thursday December&nbsp;8
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="https://www.dashworkshops.org/emnlp-2022">Data Science with Human in the Loop</a></li>
 <li>8:55 AM - 5:30 PM GST, Capital Suite 10</li>
 <li><a href="/research/improving-human-annotation">Accepted Paper: Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers</a></li>
<li>Yunyao Li co-organized this workshop, and Jeff Bigham, Katherine Luna, Shen Ma, Kun Qian, and Saptarshi Guha are members of the workshop program committee.</li>
</ul>


### Sunday December&nbsp;11
<ul class="links-stacked">
<li>POSTER SESSION</li>
<li><a href="/research/non-autoagressive-neural-machine">Non-Autoregressive Neural Machine Translation: A Call for Clarity</a></li>
 <li>11:00 AM - 12:30 PM GST, Atrium</li> 
</ul>
</div>

## Accepted Papers

### Conference Accepted Paper

[Non-Autoregressive Neural Machine Translation: A Call for Clarity](/research/non-autoagressive-neural-machine)

*Robin M. Schmidt, Telmo Pires, Stephan Peitz, Jonas Lööf*

## Demo

**Dictation Auto-Punctuation**

Dictation is one of the most heavily used input features on Apple devices. In our latest release, we have added auto-punctuation and emoji capabilities to dictation in 5 languages covering 13 locales. We support periods, commas, and question marks in the first version. These were identified as the top used punctuations that were explicitly dictated and also the punctuations that can be inferred from text alone. We have also added the ability to dictate frequently used emojis.

All EMNLP attendees are invited to stop by the Apple booth (booth number 4, located near Conference Room B of the Abu Dhabi National Exhibition Centre) to experience this demo in person.

## Acknowledgements

Yunyao Li co-organized the [Data Science with Human in the Loop workshop.](https://www.dashworkshops.org/emnlp-2022)

Jeff Bigham, Katherine Luna, Shen Ma, Kun Qian, and Saptarshi Guha are members of the [Data Science with Human in the Loop](https://www.dashworkshops.org/emnlp-2022) workshop program committee.

Rajen Chatterjee is co-organizing [Automatic Post Editing](https://www.statmt.org/wmt22/ape-task.html) shared task at the seventh conference on Machine Translation collocated with EMNLP 2022. 
 
<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple sponsored Empirical Methods in Natural Language Processing (EMNLP), which was held in Abu Dhabi, United Arab Emirates from December 7 to 11. EMNLP is a leading conference focused on natural language processing.

EMNLP 2022

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

Consumer speech recognition systems do not work as well for many people with speech differences, such as stuttering, relative to the rest of the general population. However, what is not clear is the degree to which these systems do not work, how they can be improved, or how much people want to use them. In this paper, we first address these questions using results from a 61-person survey from people who stutter and find participants want to use speech recognition but are frequently cut off, misunderstood, or speech predictions do not represent intent. In a second study, where 91 people who stutter recorded voice assistant commands and dictation, we quantify how dysfluencies impede performance in a consumer-grade speech recognition system. Through three technical investigations, we demonstrate how many common errors can be prevented, resulting in a system that cuts utterances off 79.1% less often and improves word error rate from 25.4% to 9.9%. 

From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

This paper explores the possibility of using visual object detection techniques for word localization in speech data. Object detection has been thoroughly studied in the contemporary literature for visual data. Noting that an audio can be interpreted as a 1-dimensional image, object localization techniques can be fundamentally useful for word localization. Building upon this idea, we propose a lightweight solution for word detection and localization. We use bounding box regression for word localization, which enables our model to detect the occurrence, offset, and duration of keywords in a given audio stream. We experiment with LibriSpeech and train a model to localize 1000 words. Compared to existing work (SpeechYolo), our method reduces model size by 94%, and improves the F1 score by 6.5%

Machine Learning
Research at Apple

Recent research

From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

I See What You Hear: A Vision-inspired Method to Localize Words

Research articles

3D Parametric Room Representation with RoomPlan

A Multi-Task Neural Architecture for On-Device Scene Analysis

All events

NeurIPS 2022

EMNLP 2022

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

I See What You Hear: A Vision-inspired Method to Localize Words

Research articles

3D Parametric Room Representation with RoomPlan

A Multi-Task Neural Architecture for On-Device Scene Analysis

All events

NeurIPS 2022

EMNLP 2022

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple