Self-supervised representation learning techniques utilize large datasets without semantic annotations to learn meaningful, universal features that can be conveniently transferred to solve a wide variety of downstream supervised tasks. In this paper, we propose a self-supervised method for learning representations of geographic locations from unlabeled GPS trajectories to solve downstream geospatial computer vision tasks. Tiles resulting from a raster representation of the earth's surface are modeled as nodes on a graph or pixels of an image. GPS trajectories are modeled as allowed Markovian paths on these nodes. A scalable and distributed algorithm is presented to compute image-like tensor representations, called reachability summaries, of the spatial connectivity patterns between tiles and their neighbors implied by the observed Markovian paths. A convolutional, contractive autoencoder is trained to learn compressed representations, called reachability embeddings, of reachability summaries for every tile. Reachability embeddings serve as task-agnostic, feature representations of geographic locations. Using reachability embeddings as pixel representations for five different downstream geospatial tasks, cast as supervised semantic segmentation problems, we quantitatively demonstrate that reachability embeddings are semantically meaningful representations and result in 4-23% gain in performance, as measured using area under the precision-recall curve (AUPRC) metric, when compared to baseline models that use pixel representations that do not account for the spatial connectivity between tiles. The design of reachability embeddings as pixel representations helps address the challenge of alignment and fusion in multimodal learning. Multimodal modeling of 3 different downstream geospatial tasks combining satellite imagery, mobility trajectories, and road network graph data yields 2-4% gain in performance as measured using the AUPRC metric compared to unimodal models for the same tasks. Reachability embeddings transform sequential, spatiotemporal motion trajectory data into semantically meaningful, image-like tensor representations that can be combined with other data modalities that are (e.g., satellite imagery) or can be transformed (e.g., road network graph, SAR imagery) into image-like tensor representations and are designed to facilitate multimodal learning in geospatial computer vision.


IEEE MDM

Reachability Embeddings: Self-Supervised Representation Learning from Spatiotemporal Motion Trajectories for Multimodal Geospatial Computer Vision

Real-world applications of reinforcement learning for recommendation and experimentation faces a practical challenge: the relative reward of different bandit arms can evolve over the lifetime of the learning agent. To deal with these non-stationary cases, the agent must forget some historical knowledge, as it may no longer be relevant to minimise regret. We present a solution to handling non-stationarity that is suitable for deployment at scale, to provide business operators with automated adaptive optimisation. Our solution aims to provide interpretable learning that can be trusted by humans, whilst responding to non-stationarity to minimise regret. To this end, we develop an adaptive Bayesian learning agent that employs a novel form of dynamic memory. It enables interpretability through statistical hypothesis testing, by targeting a set point of statistical power when comparing rewards and adjusting its memory dynamically to achieve this power. By design, the agent is agnostic to different kinds of non-stationarity. Using numerical simulations, we compare its performance against an existing proposal and show that, under multiple non-stationary scenarios, our agent correctly adapts to real changes in the true rewards. In all bandit solutions, there is an explicit trade-off between learning and achieving maximal performance. Our solution sits on a different point on this trade-off when compared to another similarly robust approach: we prioritise interpretability, which relies on more learning, at the cost of some regret. We describe the architecture of a large-scale deployment of automatic optimisation-as-a-service where our agent achieves interpretability whilst adapting to changing circumstances.

*=Equal Contribution

KDD Workshop

Dynamic Memory for Interpretable Sequential Optimization

Typical educational robotics approaches rely on imperative programming for robot navigation. However, with the increasing presence of AI in everyday life, these approaches miss an opportunity to introduce machine learning (ML) techniques grounded in an authentic and engaging learning context. Furthermore, the needs for costly specialized equipment and ample physical space are barriers that limit access to robotics experiences for all learners. We propose ARtonomous, a relatively low-cost, virtual alternative to physical, programming-only robotics kits. With ARtonomous, students employ reinforcement learning (RL) alongside code to train and customize virtual autonomous robotic vehicles. Through a study evaluating ARtonomous, we found that middle-school students developed an understanding of RL, reported high levels of engagement, and demonstrated curiosity for learning more about ML. This research demonstrates the feasibility of an approach like ARtonomous for 1) eliminating barriers to robotics education and 2) promoting student learning and interest in RL and ML.

ACM Interaction Design and Children

ARtonomous: Introducing Middle School Students to Reinforcement Learning Through Virtual Robotics

Estimating dimensional emotions, such as activation, valence and dominance, from acoustic speech signals has been widely explored over the past few years. While accurate estimation of activation and dominance from speech seem to be possible, the same for valence remains challenging. Previous research has shown that the use of lexical information can improve valence estimation performance.

Lexical information can be obtained from pre-trained acoustic models, where the learned representations can improve valence estimation from speech. We investigate the use of pre-trained model representations to improve valence estimation from acoustic speech signal. We also explore fusion of representations to improve emotion estimation across all three emotion dimensions: activation, valence and dominance. Additionally, we investigate if representations from pre-trained models can be distilled into models trained with low-level features, resulting in models with a less number of parameters. We show that fusion of pre-trained model embeddings result in a 79% relative improvement in concordance correlation coefficient CCC on valence estimation compared to standard acoustic feature baseline (mel-filterbank energies), while distillation from pre-trained model embeddings to lower-dimensional representations yielded a relative 12% improvement. Such performance gains were observed over two evaluation sets, indicating that our proposed architecture generalizes across those evaluation sets. We report new state-of-the-art "text-free" acoustic-only dimensional emotion estimation CCC values on two MSP-Podcast evaluation sets.


Interspeech

Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

All KDD attendees are invited to stop by the Apple booth (booth number 144) to chat with available recruiters and booth staff. 

<div class="callout">

## Schedule

Below is the schedule of Apple events. Visit the [KDD Conference website](https://kdd.org/kdd2022/) for the full up-to-date conference schedule.

### Sunday August&nbsp;14
<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://oars-workshop.github.io/">2nd International Workshop on Online and Adaptive Recommender Systems</a></li>
<li>Srivas Chennu, Andrew Maher, Jamie Martin and Subash Prabanantham will present their accepted paper, <a href="/research/dynamic-memory" class="more">Dynamic Memory for Interpretable Sequential Optimization</a> at this workshop at 11:05 - 11:20 AM ET in Room 209C at the Washington DC Convention Center.</li>
</ul>
</div>

## Workshop Accepted Paper 

[Dynamic Memory for Interpretable Sequential Optimization](/research/dynamic-memory)

*Srivas Chennu, Andrew Maher, Jamie Martin, Subash Prabhanantham*

<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)

</div>

Apple is sponsoring the 28th annual ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). KDD will be held in Washington D.C. from August 14 to 18. It is an interdisciplinary conference bringing together researchers and practitioners from data science, data mining, knowledge discovery, large-scale data analytics, and big data.

KDD 2022

All ICML attendees are invited to stop by the Apple booth (booth number 1111, located in Hall B of the Baltimore Convention Center) to check out our demos and chat with available recruiters and booth staff. 

<div class="callout">

## Schedule

Below is the schedule of Apple sponsored talks, workshops and events. Visit the [ICML 2022 website](https://icml.cc/Conferences/2022/Schedule?) for the full up-to-date conference schedule.

### Sunday July&nbsp;17 
<ul class="links-stacked">
 <li>
 EXPO DAY
 </li>
 <li>
 <a href="https://icml.cc/Conferences/2022/Schedule?showEvent=20450"
 >Enabling Hand Gesture Customization on Wrist-Worn Devices</a
 >
 </li>
 <li>
 <a href="https://icml.cc/Conferences/2022/Schedule?q=expo"
 >See the full schedule for Expo day</a
 >
 </li>
</ul>

### Monday July&nbsp;18 
<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://www.latinxinai.org/events/icml-2022">LatinX in AI</a></li>
<li><a href="https://sites.google.com/wimlworkshop.org/wiml-unworkshop2022/home">Women in Machine Learning</a></li>
<li>Tatiana Likhomanenko will give a talk on machine learning research at Apple during the workshop. Agni Kumar served as a panelist for the workshop.</li>
</ul>

### Tuesday July&nbsp;19
<ul class="links-stacked">
 <li>SPOTLIGHT TALK</li>
 <li><a href="https://icml.cc/virtual/2022/session/20055">DL Algorithms: Self-Conditioning Pre-trained Language Models</a>
<li>From 2:25 to 2:30 PM ET at Ballroom 1 & 2</li>
<li>Xavier Suau, Luca Capella, Nicholas Apostoloff</li>
</ul>

### Wednesday July&nbsp;20
<ul class="links-stacked">
 <li>SPOTLIGHT TALK</li>
 <li><a href="https://icml.cc/virtual/2022/session/20100">SA: Private Frequency Estimation via Projective Geometry</a>
<li>From 5:00 to 5:05 PM ET at Room 307</li>
<li>Vitaly Feldman, Jelani Nelson, Huy Nguyen, Kunal Talwar</li>
 <li>SOCIAL</li>
 <li><a href="https://docs.google.com/forms/d/e/1FAIpQLSdy4Y8R-nHGWLpehh7AQx_1VpuDyb9OfjlQyOOrF8mccp-vnA/viewform">Queer and Black in AI</a></li>
</ul>

### Thursday July&nbsp;21
<ul class="links-stacked">
 <li>ORAL PRESENTATION</li>
 <li><a href="https://icml.cc/virtual/2022/session/20132">Optimal Algorithms for Mean Estimation under Local Differential Privacy in Social Aspects/Optimization</a>
<li>From 4:10 to 4:30 PM ET at Room 318 - 320</li>
<li>Hilal Asi, Vitaly Feldman, Kunal Talwar</li>
 <li>SPOTLIGHT TALK</li>
 <li><a href="https://icml.cc/virtual/2022/session/20109">Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models</a>
<li>From 10:35 - 10:40 AM ET in Room 301 - 303 at Hall F</li>
<li>Jen-Hao Rick Chang, Ashish Shrivastava, Hema Koppula, Xiaoshuai Zhang, and Oncel Tuzel</li>
</ul>

### Friday July&nbsp;22
<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://icml.cc/virtual/2022/workshop/13448">Theory and Practice of Differential Privacy: Low-Communication Algorithms for Private Federated Data Analysis</a></li>
<li>Audra McMillan is a co-organizer of this workshop. Kunal Talwar will give a keynote talk at this workshop at 9:05 - 9:45 AM ET.</li>
</ul>
<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://icml.cc/virtual/2022/workshop/13458">Machine Learning for Cybersecurity (ICML-ML4Cyber)</a></li>
<li>Nicole Nichols is a co-organizer of this workshop and will be giving a talk at this workshop.</li>
</ul>

### Saturday July&nbsp;23
<ul class="links-stacked">
 <li>WORKSHOP</li>
 <li><a href="https://www.queerinai.com/icml">Queer in AI</a></li>
</ul>
</div>

## Accepted Papers

### Conference Accepted Papers

[Efficient Representation Learning via Adaptive Context Pooling](/research/efficient-representation)

*Chen Huang, Walter Talbott, Navdeep Jaitly, Josh Susskind*

This paper was accepted to the main session at ICML as a spotlight paper.

[Optimal Algorithms for Mean Estimation under Local Differential Privacy](/research/optimal-algorithms)

*Hilal Asi, Vitaly Feldman, Kunal Talwar*

This paper was accepted to the main session at ICML as an oral paper. 

[Position Prediction as an Effective Pre-training Strategy](/research/position-prediction)

*Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Y Cheng, Walter Talbott, Chen Huang, Hanlin Goh, Joshua M Susskind* 

This paper was accepted to the main session at ICML as a spotlight paper.

[Private Frequency Estimation via Projective Geometry](/research/private-frequency-estimation)

*Vitaly Feldman, Jelani Nelson, Huy Nguyen, Kunal Talwar*

This paper was accepted to the main session at ICML as an spotlight paper. 

[Self-Conditioning Pre-Trained Language Models](/research/self-conditioning-pre-trained) 

*Xavier Suau, Luca Zappella, Nicholas Apostoloff*

This paper was accepted to the main session at ICML as a spotlight paper.

[Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](/research/unsupervised-learning)

*Jen-Hao Rick Chang, Ashish Shrivastava, Hema Swetha Koppula, Xiaoshuai Zhang, Oncel Tuzel*

This paper was accepted to the main session at ICML as a spotlight paper.

All ICML attendees are invited to stop by the Apple booth (booth number 1111, located in Hall B of the Baltimore Convention Center) to experience these demos in person.

## Demos

*RoomPlan*

[RoomPlan](https://developer.apple.com/augmented-reality/roomplan/) technology allows the user to captures a room and its defining objects in a parametric format within minutes. Capture progress is automatically displayed and easy to understand at a glance. The resulting room capture is provided as a parametric representation of the room and can optionally be exported to USD, USDA or USDZ formats. The technology is supported on any of the Apple devices (iPad and iPhone) that are equipped with LiDAR sensor.

*LiveText*

This demo aims to illustrate one of the user facing Visual Intelligence capabilities in our operating systems. A distinguishing factor of the workflows are that they are deeply integrated into the operating system, and run efficiently on device. In this demo, we will interact with visual content via text in static images and streaming camera through LiveText.

Live Text was introduced in iOS 15 through static image and camera based workflows.

*ARKitScenes* 

Scene understanding is an active research area. Commercial depth sensors, such as Kinect, have enabled the release of several RGB-D datasets over the past few years which spawned novel methods in 3D scene understanding. More recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high quality RGB-D data is accessible to millions of people on a device they commonly use. This opens a whole new era in scene understanding for the Computer Vision community as well as app developers. The fundamental research in scene understanding together with the advances in machine learning can now impact people's everyday experiences. However, transforming these scene understanding methods to real-world experiences requires additional innovation and development. In this paper we introduce [ARKitScenes](/research/arkitscenes). It is not only the first RGB-D dataset that is captured with a now widely available depth sensor, but to our best knowledge, it also is the largest indoor scene understanding data released. In addition to the raw and processed data from the mobile device, ARKitScenes includes high resolution depth maps captured using a stationary laser scanner, as well as manually labeled 3D oriented bounding boxes for a large taxonomy of furniture. We further analyze the usefulness of the data for two downstream tasks: 3D object detection and color-guided depth upsampling. We demonstrate that our dataset can help push the boundaries of existing state-of-the-art methods and it introduces new challenges that better represent real-world scenarios. 

## Acknowledgements

Samy Bengio, Marco Cuturi and Ronan Collobert are senior area chairs for ICML 2022.

Navdeep Jaitly is an area chair for ICML 2022. 

Luca Zappella, Shuangfei Zhai, Walter Talbott, Barry Theobald, Tatiana Likhomanenko, Preetum Nakkiran, Xavier Suau and Chen Huang are reviewers for ICML 2022. 

<div class="callout">

Let's innovate together. Build amazing machine-learned experiences with Apple. Discover opportunities for researchers, students, and developers by visiting our [Work with us page.](/work-with-us)
 
</div>

Apple is sponsoring the International Conference on Machine Learning (ICML) which will be held in Baltimore, Maryland from July 17 to 23. ICML is a conference dedicated to the advancement of machine learning.

Machine Learning
Research at Apple.

Recent research.

Reachability Embeddings: Self-Supervised Representation Learning from Spatiotemporal Motion Trajectories for Multimodal Geospatial Computer Vision

Dynamic Memory for Interpretable Sequential Optimization

ARtonomous: Introducing Middle School Students to Reinforcement Learning Through Virtual Robotics

Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation

Updates and events.

KDD 2022

ICML 2022

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple.

Recent research.

Reachability Embeddings: Self-Supervised Representation Learning from Spatiotemporal Motion Trajectories for Multimodal Geospatial Computer Vision

Dynamic Memory for Interpretable Sequential Optimization

ARtonomous: Introducing Middle School Students to Reinforcement Learning Through Virtual Robotics

Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation

Updates and events.

KDD 2022

ICML 2022

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple.