Using end-to-end models for speech translation (ST) has increasingly been the focus of the ST community. These models condense the previously cascaded systems by directly converting sound waves into translated text. However, cascaded models have the advantage of including automatic speech recognition output, useful for a variety of practical ST systems that often display transcripts to the user alongside the translations. To bridge this gap, recent work has shown initial progress into the feasibility for end-to-end models to produce both of these outputs. However, all previous work has only looked at this problem from the consecutive perspective, leaving uncertainty on whether these approaches are effective in the more challenging streaming setting. We develop an end-to-end streaming ST model based on a re-translation approach and compare against standard cascading approaches. We also introduce a novel inference method for the joint case, interleaving both transcript and translation in generation and removing the need to use separate decoders. Our evaluation across a range of metrics capturing accuracy, latency, and consistency shows that our end-to-end models are statistically similar to cascading models, while having half the number of parameters. We also find that both systems provide strong translation quality at low latency, keeping 99% of consecutive quality at a lag of just under a second.

Streaming Models for Joint Speech Recognition and Translation

We tackle the challenge of learning a distribution over complex, realistic, indoor scenes. In this paper, we introduce Generative Scene Networks (GSN), which learns to decompose scenes into a collection of many local radiance fields that can be rendered from a free moving camera. Our model can be used as a prior to generate new scenes, or to complete a scene given only sparse 2D observations. Recent work has shown that generative models of radiance fields can capture properties such as multi-view consistency and view-dependent lighting. However, these models are specialized for constrained viewing of single objects, such as cars or faces. Due to the size and complexity of realistic indoor environments, existing models lack the representational capacity to adequately capture them. Our decomposition scheme scales to larger and more complex scenes while preserving details and diversity, and the learned prior enables high-quality rendering from viewpoints that are significantly different from observed viewpoints. When compared to existing models, GSN produces quantitatively higher-quality scene renderings across several different scene datasets.

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

LEarning TO Rank (LETOR) is a research area in the field of Information Retrieval (IR) where machine learning models are employed to rank a set of items. In the past few years, neural LETOR approaches have become a competitive alternative to traditional ones like LambdaMART. However, neural architectures performance grew proportionally to their complexity and size. This can be an obstacle for their adoption in large-scale search systems where a model size impacts latency and update time. For this reason, we propose a model agnostic approach based on a neural LETOR architecture to reduce the input size to a LETOR model by up to 60% without affecting the system performance. This approach also allows to reduce a LETOR model complexity and, therefore, its training and inference time up to 50%.

Neural Feature Selection for Learning to Rank  

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app's full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

The Apple Scholars program was created to recognize the contributions of emerging leaders in computer science and engineering at the graduate and postgraduate level.

Apple is committed to amplifying cutting-edge machine learning research worldwide, covering a wide range of topics including on-device and private machine learning, accessibility, and health. The selected Apple Scholars are advancing the field of machine learning and AI to push the boundaries of what’s possible, and Apple is committed to supporting the academic research community and their invaluable contributions to the world.

The 2021 Apple Scholars in AI/ML:

<br /><br />


<div class="opportunitiesFlex">
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Alireza_Fallah_a1fd082574.png" alt="Photo of Alireza Fallah" title="Photo of Alireza Fallah"/>
        <h3 class="typography-eyebrow-reduced">Alireza Fallah</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Massachusetts Institute of Technology</div>
        <div class="typography-body-reduced-tight">Fundamentals of Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Andreea_Bobu_b1ef843994.png" alt="Photo of Andreea Bobu" title="Photo of Andreea Bobu"/>
        <h3 class="typography-eyebrow-reduced">Andreea Bobu</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">University of California, Berkeley</div>
        <div class="typography-body-reduced-tight">Human-Centered Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Andrew_Silva_48df382c61.png" alt="Photo of Andrew Silva" title="Photo of Andrew Silva"/>
        <h3 class="typography-eyebrow-reduced">Andrew Silva</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Georgia Institute of Technology</div>
        <div class="typography-body-reduced-tight">Human-Centered Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Biwei_Huang_ed2891d328.png" alt="Photo of Biwei Huang" title="Photo of Biwei Huang"/>
        <h3 class="typography-eyebrow-reduced">Biwei Huang</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Carnegie Mellon University</div>
        <div class="typography-body-reduced-tight">Fundamentals of Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Enric_Boix_Adsera_123fa885dd.png" alt="Photo of Enric Boix-Adserà" title="Photo of Enric Boix-Adserà"/>
        <h3 class="typography-eyebrow-reduced">Enric Boix-Adserà</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Massachusetts Institute of Technology</div>
        <div class="typography-body-reduced-tight">Fundamentals of Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Eugene_Bagdasaryan_de2b98d8b3.png" alt="Photo of Eugene Bagdasaryan" title="Photo of Eugene Bagdasaryan"/>
        <h3 class="typography-eyebrow-reduced">Eugene Bagdasaryan</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Cornell University</div>
        <div class="typography-body-reduced-tight">Privacy Preserving Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Hongyu_Ren_5213141e3d.png" alt="Photo of Hongyu Ren" title="Photo of Hongyu Ren"/>
        <h3 class="typography-eyebrow-reduced">Hongyu Ren</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Stanford University</div>
        <div class="typography-body-reduced-tight">Graph Representation Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Joanne_Truong_edc22ec8dc.png" alt="Photo of Joanne Truong" title="Photo of Joanne Truong"/>
        <h3 class="typography-eyebrow-reduced">Joanne Truong</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Georgia Institute of Technology</div>
        <div class="typography-body-reduced-tight">AI for Autonomous Systems</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Jonas_Rothfuss_f69b50dff6.png" alt="Photo of Jonas Rothfuss" title="Photo of Jonas Rothfuss"/>
        <h3 class="typography-eyebrow-reduced">Jonas Rothfuss</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">ETH Zurich</div>
        <div class="typography-body-reduced-tight">Fundamentals of Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Rachel_Franz_f3210e8929.png" alt="Photo of Rachel Franz" title="Photo of Rachel Franz"/>
        <h3 class="typography-eyebrow-reduced">Rachel Franz</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">University of Washington</div>
        <div class="typography-body-reduced-tight">AI for Accessibility</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Sana_Tonekaboni_529b1a1d18.png" alt="Photo of Sana Tonekaboni" title="Photo of Sana Tonekaboni"/>
        <h3 class="typography-eyebrow-reduced">Sana Tonekaboni</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">University of Toronto</div>
        <div class="typography-body-reduced-tight">AI for Health and Wellness</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Shiori_Sagawa_2fd77ea332.png" alt="Photo of Shiori Sagawa" title="Photo of Shiori Sagawa"/>
        <h3 class="typography-eyebrow-reduced">Shiori Sagawa</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Stanford University</div>
        <div class="typography-body-reduced-tight">Fundamentals of Machine Learning</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Victor_Zhong_f0992ddb8b.png" alt="Photo of Victor Zhong" title="Photo of Victor Zhong"/>
        <h3 class="typography-eyebrow-reduced">Victor Zhong</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">University of Washington</div>
        <div class="typography-body-reduced-tight">Speech and Natural Language</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Wentao_Zhang_d253369866.png" alt="Photo of Wentao Zhang" title="Photo of Wentao Zhang"/>
        <h3 class="typography-eyebrow-reduced">Wentao Zhang</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">Peking University</div>
        <div class="typography-body-reduced-tight">AI for Manufacturing</div>
    </div>
    <div>
        <img class="circleImg" src="https://mlr.cdn-apple.com/media/Zhenpei_Yang_f1eaf44fda.png" alt="Photo of Zhenpei Yang" title="Photo of Zhenpei Yang"/>
        <h3 class="typography-eyebrow-reduced">Zhenpei Yang</h3>
        <div class="muted clear-muted-hovers typography-body-reduced-tight">University of Texas at Austin</div>
        <div class="typography-body-reduced-tight">Computer Vision and Augmented Reality</div>
    </div>
</div>


The PhD fellowship in AI/ML is currently open to invited institutions. For more information, please email [aiml_scholars@apple.com](mailto:aiml_scholars@apple.com).

<a href="/updates/introducing-apple-scholars-aiml" class="more">Learn more about the inaugural class of Apple Scholars</a>

<p>Apple is proud to announce the 2021 recipients of the Apple Scholars in AI/ML PhD fellowship. Each Scholar receives two years of funding as they pursue their PhD, internship opportunities, and a two-year mentorship with an Apple researcher in their field. Nominated students were selected based on their innovative research, their record as thought leaders and collaborators in their fields, and their unique commitment to take risks and push the envelope in machine learning and AI.</p>

Announcing the 2021 Apple Scholars in AI/ML

[Learn more about NeurIPS](https://neurips.cc/Conferences/2020)

Visit our NeurIPS 2020 [virtual booth](https://neurips.cc/ExpoConferences/2020/Sponsorpage?id=654)



## Conference Accepted Papers

### [Collegial Ensembles](/research/collegial-ensembles)

*Etai Littwin, Ben Myara, Sima Sabah, Joshua Susskind, Shuangfei Zhai, Oren Golan*
 
Modern neural network performance typically improves as model size increases. A recent line of research on the Neural Tangent Kernel (NTK) of over-parameterized networks indicates that the improvement with size increase is a product of a better conditioned loss landscape. In this work, we investigate a form of over-parameterization achieved through ensembling, where we define collegial ensembles (CE) as the aggregation of multiple independent models with identical architectures, trained as a single model. We show that the optimization dynamics of CE simplify dramatically when the number of models in the ensemble is large, resembling the dynamics of wide models, yet scale much more favorably. We use recent theoretical results on the finite width corrections of the NTK to perform efficient architecture search in a space of finite width CE that aims to either minimize capacity, or maximize trainability under a set of constraints. The resulting ensembles can be efficiently implemented in practical architectures using group convolutions and block diagonal layers. Finally, we show how our framework can be used to analytically derive optimal group convolution modules originally found using expensive grid searches, without having to train a single model.

### [Faster Differentially Private Samplers via Rényi Divergence Analysis of Discretized Langevin MCMC](/research/differentially-private-samplers)

*Arun Ganesh, Kunal Talwar*
 
Various differentially private algorithms instantiate the exponential mechanism, and require sampling from the distribution exp(−f) for a suitable function f. When the domain of the distribution is high-dimensional, this sampling can be computationally challenging. Using heuristic sampling schemes such as Gibbs sampling does not necessarily lead to provable privacy. When f is convex, techniques from log-concave sampling lead to polynomial-time algorithms, albeit with large polynomials. Langevin dynamics-based algorithms offer much faster alternatives under some distance measures such as statistical distance. In this work, we establish rapid convergence for these algorithms under distance measures more suitable for differential privacy. For smooth, strongly-convex f, we give the first results proving convergence in Rényi divergence. This gives us fast differentially private algorithms for such f. Our techniques and simple and generic and apply also to underdamped Langevin dynamics.


### [On the Error Resistance of Hinge-Loss Minimization](/research/error-resistance)

*Kunal Talwar*

Commonly used classification algorithms in machine learning, such as support vector machines, minimize a convex surrogate loss on training examples. In practice, these algorithms are surprisingly robust to errors in the training data. In this work, we identify a set of conditions on the data under which such surrogate loss minimization algorithms provably learn the correct classifier. This allows us to establish, in a unified framework, the robustness of these algorithms under various models on data as well as error. In particular, we show that if the data is linearly classifiable with a slightly non-trivial margin (i.e. a margin at least C/d‾‾√ for d-dimensional unit vectors), and the class-conditional distributions are near isotropic and logconcave, then surrogate loss minimization has negligible error on the uncorrupted data even when a constant fraction of examples are adversarially mislabeled.

### [Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses](/research/stochastic-gradient-descent)

*Raef Bassily, Vitaly Feldman, Criztobal Guzman, Kunal Talwar*

Uniform stability is a notion of algorithmic stability that bounds the worst case change in the model output by the algorithm when a single data point in the dataset is replaced. An influential work of Hardt et al. (2016) provides strong upper bounds on the uniform stability of the stochastic gradient descent (SGD) algorithm on sufficiently smooth convex losses. These results led to important progress in understanding of the generalization properties of SGD and several applications to differentially private convex optimization for smooth losses.  

### [Stochastic Optimization with Laggard Data Pipelines](/research/stochastic-optimization)

*Naman Agarwal, Rohan Anil, Tomer Koren, Kunal Talwar, Cyril Zhang*

State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes. As a consequence, CPU-bound preprocessing and disk/memory/network operations have emerged as new performance bottlenecks, as opposed to hardware-accelerated gradient computations. In this regime, a recently proposed approach is data echoing (Choi et al., 2019), which takes repeated gradient steps on the same batch while waiting for fresh data to arrive from upstream. We provide the first convergence analyses of "data-echoed" extensions of common optimization methods, showing that they exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with stochastic minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.

### [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](/research/what-neural-networks-memorize)

*Vitaly Feldman, Chiyuan Zhang*

Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. 

In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).

## Talks and Workshops

### [Machine Learning for Mobile Health Workshop](https://sites.google.com/view/ml4mobilehealth-neurips-2020/home)
This workshop, held on December 12, aimed to assemble researchers from the key areas in mobile health to better address the challenges currently facing the widespread use of mobile health technologies. We presented our workshop accepted paper which discusses synthetic data generated using a realistic ECG simulator and a structured noise model. 

**Machine Learning for Mobile Health Workshop Accepted Paper:** <br>
[Representing and Denoising Wearable ECG Recordings](/research/wearable-ecg-recordings) <br>*Jeffrey Chan, Andrew C. Miller, Emily Fox*

### [Privacy Preserving Machine Learning (PPML) Workshop](https://ppml-workshop.github.io/#page-top)
This workshop, held on December 11, focused on privacy preserving techniques for machine learning and disclosure in large scale data analysis both in the distributed and centralized settings. We will be presenting our workshop accepted papers which discusses how considering sequential setting in which a single dataset of individuals is used to perform adaptively-chosen analyses, and how random shuffling of input data amplifies differential privacy guarantees.

**PPML Workshop Accepted Papers:** <br>
[Individual Privacy Accounting via a Renyi Filter](/research/individual-privacy-accounting) <br>
*Vitaly Feldman, Tijana Zrnic* 

[A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling](/research/privacy-amplification-by-shuffling) <br>
*Vitaly Feldman, Audra McMillan, Kunal Talwar*

### [Offline Reinforcement Learning Workshop](https://offline-rl-neurips.github.io/)
This workshop, held on December 12, brought attention to offline Reinforcement Learning. This workshop facilitated discussions surrounding algorithmic challenges, solutions and real-world applications. 

**Offline Reinforcement Learning Workshop Accepted Paper:** <br>
[Uncertainty Weighted Offline Reinforcement Learning](/research/offline-reinforcement-learning) <br>
*Yue Wu, Shuangfei Zhai, Nitish Srivastava, Josh Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh*  

## Expo Day
**Accelerated Training with ML Compute on M1-Powered Mac** <br>
Apple presented a talk on Accelerated Training with ML Compute on M1-Powered Mac during NeurIPS Expo Day on December 6.

In November, Apple announced Mac powered by the M1 chip, featuring a powerful machine learning accelerator and high-performance GPU. ML Compute, a new framework available in macOS Big Sur, enables developers to accelerate the training of neural networks using the CPU and GPU.

In this talk, we discussed how we use ML Compute to speed up the training of ML models on M1-powered Mac with popular deep learning frameworks such as TensorFlow. We showed how to replace the TensorFlow ops in graph and eager mode with an ML Compute graph. We also present the performance and watt improvements when training neural networks on Mac with M1. Finally, we examined how unified memory and other memory optimizations on M1-powered Mac allow us to minimize the memory footprint when training neural networks. Learn more about how we [leverage ML Compute for Accelerated Training on Mac.](/updates/ml-compute-training-on-mac)

<div class="callout">

## Affinity Group Workshops

Apple sponsored the [Black in AI](http://blackinai2020.vercel.app/), [LatinX in AI](https://www.latinxinai.org/neurips-2020), [Queer in AI](https://sites.google.com/view/queer-in-ai/), and [Women in Machine Learning](https://wimlworkshop.org/neurips2020/) workshops throughout the week.

At the Women in Machine Learning workshop on December 9, we gave a talk on how our on-device machine learning powers intelligent experiences on Apple products.

[Learn more about Apple’s company-wide inclusion and diversity efforts](https://www.apple.com/diversity/)

</div>


<p>Apple sponsored the Neural Information Processing Systems (NeurIPS) conference, which was held virtually from December 6 to 12. NeurIPS is a global conference focused on fostering the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.</p>

Machine Learning
Research at Apple.

Recent papers.

Streaming Models for Joint Speech Recognition and Translation

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

Neural Feature Selection for Learning to Rank

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Updates and events.

Announcing the 2021 Apple Scholars in AI/ML

Apple at NeurIPS 2020

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple.

Recent papers.

Streaming Models for Joint Speech Recognition and Translation

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

Neural Feature Selection for Learning to Rank

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Updates and events.

Announcing the 2021 Apple Scholars in AI/ML

Apple at NeurIPS 2020

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple.