This paper was accepted at the workshop at "Human-in-the-Loop Learning Workshop" at NeurIPS 2022. 

Preference-based reinforcement learning (RL) algorithms help avoid the pitfalls of hand-crafted reward functions by distilling them from human preference feedback, but they remain impractical due to the burdensome number of labels required from the human, even for relatively simple tasks.  In this work, we demonstrate that encoding environment dynamics in the reward function (REED) dramatically reduces the number of preference labels required in state-of-the-art preference-based RL frameworks. We hypothesize that REED-based methods better partition the state-action space and facilitate generalization to state-action pairs not included in the preference dataset. REED iterates between encoding environment dynamics in a state-action representation via a self-supervised temporal consistency task, and bootstrapping the preference-based reward function from the state-action representation. Whereas prior approaches train only on the preference-labelled trajectory pairs, REED exposes the state-action representation to all transitions experienced during policy training. We explore the benefits of REED within the PrefPPO [1] and PEBBLE [2] preference learning frameworks and demonstrate improvements across experimental conditions to both the speed of policy learning and the final policy performance. For example, on quadruped-walk and walker-walk with 50 preference labels, REED-based reward functions recover 83% and 66% of ground truth reward policy performance and without REED only 38\% and 21% are recovered. For some domains, REED-based reward functions result in policies that outperform policies trained on the ground truth reward.


NeurIPS Workshop

Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning

This paper was accepted at the workshop "Self-Supervised Learning - Theory and Practice" at NeurIPS 2022.

Many state of the art self-supervised learning approaches fundamentally rely on transformations applied to the input in order to selectively extract task-relevant information. Recently, the field of equivariant deep learning has developed to introduce structure into the feature space of deep neural networks, specifically with respect to such input transformations. In this work, we observe both theoretically and empirically, that through the lens of equivariant representations, many existing self-supervised learning algorithms can be both unified and generalized. Specifically, we introduce a general framework we call Structured Self-Supervised Learning (S-SSL), and theoretically show how it may subsume the concept of input-augmentations provided a sufficiently structured representation. We validate this theory experimentally for simple augmentations, demonstrate how the framework fails when representational structure is removed, and further empirically explore how the parameters of this framework relate to those traditional augmentation-based self-supervised learning. We conclude with a discussion of the potential benefits of this framework as a new perspective on self-supervised learning.

NeurIPS

Homomorphic Self-Supervised Learning 

This paper was accepted at the workshop "I Can’t Believe It’s Not Better: Understanding Deep Learning Through Empirical Falsification" 

Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently emerged as a powerful strategy for semi-supervised learning in speech recognition. In contrast with earlier strategies that alternated between training a model and generating pseudo-labels (PLs) with it, here PLs are generated in end-to-end manner as training proceeds, improving training speed and the accuracy of the final model. PL shares a common theme with teacher-student models such as distillation in that a teacher model generates targets that need to be mimicked by the student model being trained. However, interestingly, PL strategies in general use hard-labels, whereas distillation uses the distribution over labels as the target to mimic. Inspired by distillation we expect that specifying the whole distribution (aka soft-labels) over sequences as the target for unlabeled data, instead of a single best pass pseudo-labeled transcript (hard-labels) should improve PL performance and convergence. Surprisingly and unexpectedly, we find that soft-labels targets can lead to training divergence, with the model collapsing to a degenerate token distribution per frame. We hypothesize that the reason this does not happen with hard-labels is that training loss on hard-labels imposes sequence-level consistency that keeps the model from collapsing to the degenerate solution. In this paper, we show several experiments that support this hypothesis, and experiment with several regularization approaches that can ameliorate the degenerate collapse when using soft-labels. These approaches can bring the accuracy of soft-labels closer to that of hard-labels, and while they are unable to outperform them yet, they serve as a useful framework for further improvements.

Continuous Soft Pseudo-Labeling in ASR

This paper was accepted at the workshop "Self-Supervised Learning - Theory and Practice" at NeurIPS 2022. 

Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. However, in numerous realistic scenarios, the downstream task might be biased with respect to the target label distribution. This in turn moves the learned fine-tuned model posterior away from the initial (label) bias-free self-supervised model posterior. In this work, we re-interpret SSL fine-tuning under the lens of Bayesian continual learning and consider regularization through the Elastic Weight Consolidation (EWC) framework. We demonstrate that self-regularization against an initial SSL backbone improves worst sub-group performance in Waterbirds by 5% and Celeb-A by 2% when using the ViT-B/16 architecture. Furthermore, to help simplify the use of EWC with SSL, we pre-compute and publicly release the Fisher Information Matrix (FIM), evaluated with 10,000 ImageNet-1K variates evaluated on large modern SSL architectures including ViT-B/16 and ResNet50 trained with DINO.

Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events. Check back for our accepted paper presentation times and locations. Visit the [NeurIPS 2022 website](https://neurips.cc/virtual/2022/calendar) for the full conference schedule.

### Monday November&nbsp;28
<ul class="links-stacked">
 <li>EXPO TALK</li>
 <li><a href="https://nips.cc/Conferences/2022/Schedule?showEvent=56490">Generative Understanding of 3D Scenes</a></li>
 <li>3:00 - 4:00 PM CST in Theater A</li>
 <li>Miguel Angel Bautista Martin</li>
<li><a href=https://nips.cc/Conferences/2022/Schedule?q=expo&type=Expo+Talk+Panel>See the full schedule of talks for Expo Talk day</a></li>
<li>WORKSHOP</li>
<li><a href=“https://blackinai.github.io/#/workshop/bai2022">Black in AI</a></li>
 <li>10:00 AM - 6:00 PM CST in Room 288 - 289</li>
<li>WORKSHOP</li>
<li><a href="https://www.latinxinai.org/neurips-2022">LatinX in AI</a></li>
 <li>8:00 AM - 6:00 PM CST in Room 298 - 299</li>
 <li>Samy Bengio will be participating in the mentoring hour as a mentor.</li>
<li>WORKSHOP</li>
<li><a href="https://www.queerinai.com/neurips-2022">Queer in AI</a></li>
 <li>9:00 AM - 6:00 PM CST in Room 283</li>
<li>WORKSHOP</li>
<li><a href="https://sites.google.com/view/wiml2022/">Women in Machine Learning</a></li>
 <li>7:30 AM - 6:00 PM CST in Hall I-2</li>
 <li>Lauren Araujo and Sefani Tadesse will be hosting a roundtable on Internships at Apple.</li>
</ul>

### Tuesday November&nbsp;29
<ul class="links-stacked">
 <li>POSTER SESSION 1</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/54286">Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures</a></li>
 <li>11:00 AM - 1:00 PM CST, Hall J #406</li> 
 <li><a href="https://neurips.cc/virtual/2022/poster/54640">Low-Rank Optimal Transport: Approximation, Statistics and Debiasing</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #507</li>
<li><a href="https://neurips.cc/virtual/2022/poster/54462">Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss</a></li>
 <li>From 11:00 AM - 1:00 PM CST in Hall J #331</li>
 <li><a href=" https://neurips.cc/virtual/2022/poster/53984">Subspace Recovery from Heterogeneous Data with Non-isotropic Noise</a></li>
 <li>11:00 AM - 1:00 PM CST, Hall J #531</li> 
</ul>

### Wednesday November&nbsp;30
<ul class="links-stacked">
 <li>POSTER SESSION 4</li>
 <li><a href="https://neurips.cc/virtual/2022/poster/55624">FLAIR: Federated Learning Annotated Image Repository</a></li>
 <li>4:00 - 6:00 PM CST, Hall J #1028</li> 
</ul>

### Thursday December&nbsp;1
<ul class="links-stacked">
<li>POSTER SESSION 5</li>
<li><a href="https://neurips.cc/virtual/2022/poster/54862">GAUDI: A Neural Architect for Immersive 3D Scene Generation</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #427</li> 
<li><a href="https://neurips.cc/virtual/2022/poster/55742">MBW: Multi-view Bootstrapping in the Wild</a></li>
 <li>11:00 AM - 1:00 PM CST in Hall J #1020</li> 
<li><a href="https://neurips.cc/virtual/2022/poster/52931">Mean Estimation with User-level Privacy</a></li>
 <li>4:00 - 6:00 PM CST in Hall J #730</li> 
</ul>

### Friday December&nbsp;2
<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href="https://timeseriesforhealth.github.io/index">Learning from Time Series for Health</a></li>
 <li>9:00 AM - 5:10 PM CST, Room 392</li>
<li><a href="/research/masked-auto-encoder">Accepted Paper: MAEEG: Masked Autoencoder for EEG Representation Learning</a></li>
<li><a href="/research/masked-auto-encoder">Accepted Paper: Modeling Heart Rate Dynamics and Wearable Workout Data</a></li>
<li>Sana Tonekaboni, one of our Apple Scholars in AI/ML, co-organized this workshop.</li>
<li>WORKSHOP</li>
 <li><a href="https://www.cml-4-impact.vanderschaar-lab.com">Causal Machine Learning for Real-World Impact</a></li>
<li>9:00 AM - 5:45 PM CST, Room 296</li>
<li><a href="/research/large-scale-observational">Accepted Paper: Causal Analysis of a Large Scale Behavioral Intervention Nudge</a></li>
<li>WORKSHOP</li>
 <li><a href="https://hity-workshop.github.io/NeurIPS2022/">Has it Trained Yet? A Workshop for Algorithmic Efficiency in Practical Neural Network Training</a></li>
<li>8:30 AM - 5:00 PM CST, Theater B</li>
<li><a href=/research/slingshot-mechanism>Accepted Paper: The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon</a></li>
<li>WORKSHOP</li>
 <li><a href="https://neurips-hill.github.io/">Human in the Loop Learning</a></li>
<li>10:00 AM - 7:00 PM CST, Room 396</li>
<li>Accepted Paper: Symbol Guided Hindsight Priors for Reward Learning from Human Preferences</li>
<li><a href=/research/rewards-encoding>Accepted Paper: Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning</a></li>
</ul>

### Saturday December&nbsp;3
<ul class="links-stacked">
<li>WORKSHOP</li>
 <li><a href="https://sites.google.com/view/RL4RealLife">Reinforcement Learning for Real Life</a></li>
<li>8:25 AM - 5:00 PM CST, Theatre A</li>
<li>Accepted Paper: CAGEd Wisdom: Applying Reinforcement Learning to Autonomous Cyber Defense</li>
<li>WORKSHOP</li>
<li><a href="https://sslneurips22.github.io/">Self-Supervised Learning: Theory and Practice</a></li>
<li>Workshop time TBC, Room 291 - 292</li>

<li><a href="/research/elastic-weight-consolidation">Accepted Paper: Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer</a></li>
<li>Accepted Paper: Contrastive Self-Supervised Learning for Skeleton Representations</li>
<li><a href="/research/homomorphic-self-supervised-learning">Accepted Paper: Homomorphic Self-Supervised Learning with Equivariant Networks</a></li>
<li>WORKSHOP</li>
<li><a href="https://sites.google.com/view/icbinb-2022/home">I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification</a></li>
<li>6:15 AM - 3:00 PM CST, Room 286</li>
<li><a href="/research/soft-pseudo-labeling">Accepted Paper: Continuous Soft Pseudo-Labeling in ASR</a></li>
<li>Arno Blaas co-organized this workshop, and Luca Zappella is a member of the coordination team.</li>
<li>WORKSHOP</li>
<li><a href="https://ml4physicalsciences.github.io/2022/">Machine Learning and the Physical Sciences</a></li>
<li>7:50 AM - 5:00 PM CST, Room 275 - 277</li>
<li>Accepted Paper: Robust Hybrid Learning with Expert Augmentations</li>
</ul>

### Monday December&nbsp;5
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="https://neurips.cc/virtual/2022/affinity-workshop/65366">Women in Machine Learning</a></li>
<li>9:30 AM - 4:30 PM CST, Held Virtually</li>
<li>Katherine Metcalf will give a talk on machine learning research at Apple during the workshop.</li>
</ul>
</div>

## Accepted Papers

### Conference Accepted Papers

[FLAIR: Federated Learning Annotated Image Repository](/research/flair)

*Congzheng Song, Filip Granqvist, Kunal Talwar*

[GAUDI: A Neural Architect for Immersive 3D Scene Generation](/research/gaudi)

*Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Joshua Susskind*

[Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](/research/analyzing-mitigating-repetitions)

*Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, Jian Li*

[Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures](/research/generalization-unseen-data-boolean)

*Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia, Jon Kleinberg, Aryo Lotfi, Maithra Raghu, Chiyuan Zhang*

[Low-rank Optimal Transport: Approximation, Statistics and Debiasing](/research/low-rank-optimal-transport)

*Meyer Scetbon, Marco Cuturi*

[MBW: Multi-view Bootstrapping in the Wild](/research/multi-view-bootstrapping)

*Mosam Dabhi, Chaoyang Wang, Tim Clifford, László Jeni, Ian Fasel, Simon Lucey*

[Mean Estimation with User-level Privacy under Data Heterogeneity](/research/mean-estimation)

*Rachel Cummings, Vitaly Feldman, Audra McMillan, Kunal Talwar*

[Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss](/research/privacy-noisy-stochastic)

*Jason Altschuler, Kunal Talwar*

[Subspace Recovery from Heterogeneous Data with Non-isotropic Noise](/research/subspace-recovery)

*John Duchi, Vitaly Feldman, Lunjia Hu, Kunal Talwar*

### Workshop Accepted Papers

[Large-Scale Observational Study of the Causal Effects of a Behavioral Health Nudge](/research/large-scale-observational)

*Achille Nazaret, Guillermo Sapiro*

[How Soft Labels Cause Divergence in Speech Recognition for Continuous Pseudo-Labeling](/research/soft-pseudo-labeling)

*Tatiana Likhomanenko, Ronan Collobert, Navdeep Jaitly, Samy Bengio*

[Improving Generalization with Physical Equations](/research/improving-generalization)

*Antoine Wehenkel, Jens Behrmann, Hsiang Hsu, Guillermo Sapiro, Gilles Louppe, Joern-Henrik Jacobsen*

[Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning](/research/rewards-encoding)

*Katherine Metcalf, Miguel Sarabia and Barry-John Theobald*

[Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer](/research/elastic-weight-consolidation)

*Andrius Ovsianas, Jason Ramapuram, Dan Busbridge, Eeshan Gunesh Dhekane, Russ Webb*

[MAEEG: Masked Autoencoder for EEG Representation Learning](/research/masked-auto-encoder)

*Sherry Chien, Hanlin Goh, Chris Sandino, Joseph Yitan Cheng*

[Homomorphic Self-Supervised Learning](/research/homomorphic-self-supervised-learning)

*Thomas Anderson Keller, Xavier Suau, Luca Zappella*

[Improving Generalization with Physical Equations](/research/improving-generalization)

*Antoine Wehenkel, Jens Behrmann, Guillermo Sapiro, Joern Jacobsen, Gilles Louppe, Hsiang (Shawn) Hsu*

[The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](/research/slingshot-mechanism)

*Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Josh Susskind*

## Demos

**RoomPlan**

[RoomPlan](https://developer.apple.com/augmented-reality/roomplan/) technology allows the user to captures a room and it's defining objects in a parametric format within minutes. Capture progress is automatically displayed and easy to understand at a glance. The resulting room capture is provided as a parametric representation of the room and can optionally be exported to USD, USDA or USDZ formats. The technology is supported on any of the Apple devices (iPad and iPhone) that are equipped with LiDAR sensor.

**Information Intelligence**

This demo aims to illustrate the rich information extraction and understanding capabilities embedded in select Apple products that use our state of the art search and intelligence technologies to cater to user’s information needs. These features leverage deep text and image understanding capabilities coupled with powerful search technologies to surface accurate and relevant information for user’s queries.

All NeurIPS attendees are invited to stop by the Apple booth (booth number 603, located in Hall G of the Ernest N. Morial Convention Center) to experience these demos in person.

## Acknowledgements

Audra McMillan, Chong Wang, Congzheng Song, Devon Hjelm, Felix Bai, Hilal Asi, Jacob Yao, Jerome Bellegarda, Joe Futoma, Luca Zappella, Miguel Angel Bautista Martin, Sachin Agarwal, Sachin Mehta, and Tatiana Likhomanenko are reviewers for this conference. 

Marian Stewart Bartlett is the treasurer for this conference. 

Samy Bengio is a senior area chair and board member for this conference. 

Kunal Talwar and Laurent Dinh are area chairs for this conference. 

Arno Blaas co-organized the I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification workshop.

Luca Zappella is a member of the coordination team for the I Can't Believe It's Not Better: Understanding Deep Learning Through Empirical Falsification workshop.

<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple is sponsoring the Neural Information Processing Systems Conference (NeurIPS), which will be held in New Orleans, Louisiana from November 28 to December 9. NeurIPS is a global conference focused on fostering the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.

NeurIPS 2022

All EMNLP attendees are invited to stop by the Apple booth (booth number 4, located near Conference Room B of the Abu Dhabi National Exhibition Centre) to chat with available recruiters and booth staff.

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored workshops and events. Visit the [EMNLP 2022 website](https://2022.emnlp.org/program/) for the full conference schedule.

### Wednesday December&nbsp;7
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href=“http://www.winlp.org/winlp-2022-workshop/">Widening Natural Language Processing</a></li>
 <li>From 8:45 AM - 4:50 PM GST</li>
 <li>Yunyao Li will be participating in the sponsor lunch during the workshop.</li>
</ul>

### Thursday December&nbsp;8
<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href="https://www.dashworkshops.org/emnlp-2022">Data Science with Human in the Loop</a></li>
 <li>Accepted Paper: Improving Human Annotation Effectiveness for Fact Collection by Identifying the Most Relevant Answers</li>
<li>Yunyao Li co-organized this workshop, and Jeff Bigham, Katherine Luna, Shen Ma, Kun Qian, and Saptarshi Guha are members of the workshop program committee.</li>
</ul>
</div>

## Accepted Papers

### Conference Accepted Paper

[Non-Autoregressive Neural Machine Translation: A Call for Clarity](/research/non-autoagressive-neural-machine)

*Robin M. Schmidt, Telmo Pires, Stephan Peitz, Jonas Lööf*

## Demo

**Dictation Auto-Punctuation**

Dictation is one of the most heavily used input features on Apple devices. In our latest release, we have added auto-punctuation and emoji capabilities to dictation in 5 languages covering 13 locales. We support periods, commas, and question marks in the first version. These were identified as the top used punctuations that were explicitly dictated and also the punctuations that can be inferred from text alone. We have also added the ability to dictate frequently used emojis.

All EMNLP attendees are invited to stop by the Apple booth (booth number 4, located near Conference Room B of the Abu Dhabi National Exhibition Centre) to experience this demo in person.

## Acknowledgements

Yunyao Li co-organized the <a href=https://www.dashworkshops.org/emnlp-2022>Data Science with Human in the Loop</a> workshop.

Jeff Bigham, Katherine Luna, Shen Ma, Kun Qian, and Saptarshi Guha are members of the [Data Science with Human in the Loop](https://www.dashworkshops.org/emnlp-2022) workshop program committee.

<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple is sponsoring Empirical Methods in Natural Language Processing (EMNLP), which will be held in Abu Dhabi, United Arab Emirates from December 7 to 11. EMNLP is a leading conference focused on natural language processing.

Machine Learning
Research at Apple.

Recent research.

Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning

Homomorphic Self-Supervised Learning

Continuous Soft Pseudo-Labeling in ASR

Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer

Updates and events.

NeurIPS 2022

EMNLP 2022

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple.

Recent research.

Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning

Homomorphic Self-Supervised Learning

Continuous Soft Pseudo-Labeling in ASR

Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer

Updates and events.

NeurIPS 2022

EMNLP 2022

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple.