We propose a frame-level SpecAugment method (f-SpecAugment) to improve the performance of deep convolutional neural networks (CNN) for hybrid HMM based ASR systems. Similar to the utterance level SpecAugment, f-SpecAugment performs three transformations: time warping, frequency masking, and time masking. Instead of applying the transformations at the utterance level, f-SpecAugment applies them to each convolution window independently during training. We demonstrate that f-SpecAugment is more effective than the utterance level SpecAugment for deep CNN based hybrid models. We evaluate the proposed f-SpecAugment on 50-layer Self-Normalizing Deep CNN (SNDCNN) acoustic models trained with up to 25000 hours of training data. We observe f-SpecAugment reduces WER by 0.5-4.5% relatively across different ASR tasks for four languages. As the benefits of augmentation techniques tend to diminish as training data size increases, the large scale training reported is important in understanding the effectiveness of f-SpecAugment. Our experiments demonstrate that even with 25k training data, f-SpecAugment is still effective. We also demonstrate that f-SpecAugment has benefits approximately equivalent to doubling the amount of training data for deep CNNs. 

Frame-level SpecAugment for Deep Convolutional Neural Networks in Hybrid ASR Systems

Recent work of Erlingsson, Feldman, Mironov, Raghunathan, Talwar, and Thakurta [EFMRTT19] demonstrates that random shuffling amplifies differential privacy guarantees of locally randomized data. Such amplification implies substantially stronger privacy guarantees for systems in which data is contributed anonymously [BEMMRLRKTS17] and has lead to significant interest in the shuffle model of privacy [CSUZZ19,EFMRTT19]. 

We show that random shuffling of $n$ data records that are input to $\varepsilon_0$-differentially private local randomizers results in an $(O((1 − e^{-\varepsilon_0})\sqrt\frac{e^{\varepsilon_0} \log (1/\delta)}{n}), \delta)$-differentially private algorithm. This significantly improves over previous work and achieves the asymptotically optimal dependence in $\varepsilon_0$. Our result is based on a new approach that is simpler than previous work and extends to approximate differential privacy with nearly the same guarantees. Our work also yields an empirical method to derive tighter bounds the resulting ε and we show that it gets to within a small constant factor of the optimal bound. As a direct corollary of our analysis, we derive a simple and asymptotically optimal algorithm for discrete distribution estimation in the shuffle model of privacy. We also observe that our result implies the first asymptotically optimal privacy analysis of noisy stochastic gradient descent that applies to sampling without replacement.

A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling

For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge with Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. Together, these features make our dataset well-suited for geometric learning problems that require direct 3D supervision, multi-task learning problems that require reasoning jointly over multiple input and output modalities, and inverse rendering problems. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, annotation effort, and computation time. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a state-of-the-art natural language processing model. All the code we used to generate our dataset is available online.

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

State-of-the-art learning-based monocular 3D reconstruction methods learn priors over object categories on the training set, and as a result struggle to achieve reasonable generalization to object categories unseen during training. In this paper we study the inductive biases encoded in the model architecture that impact the generalization of learning-based 3D reconstruction methods. We find that 3 inductive biases impact performance: the spatial extent of the encoder, the use of the underlying geometry of the scene to describe point features, and the mechanism to aggregate information from multiple views. Additionally, we propose mechanisms to enforce those inductive biases: a point representation that is aware of camera position, and a variance cost to aggregate information across views. Our model achieves state-of-the-art results on the standard ShapeNet 3D reconstruction benchmark in various settings.

On the Generalization of Learning-based 3D Reconstruction

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

[Learn more about NeurIPS](https://neurips.cc/Conferences/2020)

Visit our NeurIPS 2020 [virtual booth](https://neurips.cc/ExpoConferences/2020/Sponsorpage?id=654)



## Conference Accepted Papers

### [Collegial Ensembles](/research/collegial-ensembles)

*Etai Littwin, Ben Myara, Sima Sabah, Joshua Susskind, Shuangfei Zhai, Oren Golan*
 
Modern neural network performance typically improves as model size increases. A recent line of research on the Neural Tangent Kernel (NTK) of over-parameterized networks indicates that the improvement with size increase is a product of a better conditioned loss landscape. In this work, we investigate a form of over-parameterization achieved through ensembling, where we define collegial ensembles (CE) as the aggregation of multiple independent models with identical architectures, trained as a single model. We show that the optimization dynamics of CE simplify dramatically when the number of models in the ensemble is large, resembling the dynamics of wide models, yet scale much more favorably. We use recent theoretical results on the finite width corrections of the NTK to perform efficient architecture search in a space of finite width CE that aims to either minimize capacity, or maximize trainability under a set of constraints. The resulting ensembles can be efficiently implemented in practical architectures using group convolutions and block diagonal layers. Finally, we show how our framework can be used to analytically derive optimal group convolution modules originally found using expensive grid searches, without having to train a single model.

### [Faster Differentially Private Samplers via Rényi Divergence Analysis of Discretized Langevin MCMC](/research/differentially-private-samplers)

*Arun Ganesh, Kunal Talwar*
 
Various differentially private algorithms instantiate the exponential mechanism, and require sampling from the distribution exp(−f) for a suitable function f. When the domain of the distribution is high-dimensional, this sampling can be computationally challenging. Using heuristic sampling schemes such as Gibbs sampling does not necessarily lead to provable privacy. When f is convex, techniques from log-concave sampling lead to polynomial-time algorithms, albeit with large polynomials. Langevin dynamics-based algorithms offer much faster alternatives under some distance measures such as statistical distance. In this work, we establish rapid convergence for these algorithms under distance measures more suitable for differential privacy. For smooth, strongly-convex f, we give the first results proving convergence in Rényi divergence. This gives us fast differentially private algorithms for such f. Our techniques and simple and generic and apply also to underdamped Langevin dynamics.


### [On the Error Resistance of Hinge-Loss Minimization](/research/error-resistance)

*Kunal Talwar*

Commonly used classification algorithms in machine learning, such as support vector machines, minimize a convex surrogate loss on training examples. In practice, these algorithms are surprisingly robust to errors in the training data. In this work, we identify a set of conditions on the data under which such surrogate loss minimization algorithms provably learn the correct classifier. This allows us to establish, in a unified framework, the robustness of these algorithms under various models on data as well as error. In particular, we show that if the data is linearly classifiable with a slightly non-trivial margin (i.e. a margin at least C/d‾‾√ for d-dimensional unit vectors), and the class-conditional distributions are near isotropic and logconcave, then surrogate loss minimization has negligible error on the uncorrupted data even when a constant fraction of examples are adversarially mislabeled.

### [Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses](/research/stochastic-gradient-descent)

*Raef Bassily, Vitaly Feldman, Criztobal Guzman, Kunal Talwar*

Uniform stability is a notion of algorithmic stability that bounds the worst case change in the model output by the algorithm when a single data point in the dataset is replaced. An influential work of Hardt et al. (2016) provides strong upper bounds on the uniform stability of the stochastic gradient descent (SGD) algorithm on sufficiently smooth convex losses. These results led to important progress in understanding of the generalization properties of SGD and several applications to differentially private convex optimization for smooth losses. 

### [Stochastic Optimization with Laggard Data Pipelines](/research/stochastic-optimization)

*Naman Agarwal, Rohan Anil, Tomer Koren, Kunal Talwar, Cyril Zhang*

State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes. As a consequence, CPU-bound preprocessing and disk/memory/network operations have emerged as new performance bottlenecks, as opposed to hardware-accelerated gradient computations. In this regime, a recently proposed approach is data echoing (Choi et al., 2019), which takes repeated gradient steps on the same batch while waiting for fresh data to arrive from upstream. We provide the first convergence analyses of "data-echoed" extensions of common optimization methods, showing that they exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with stochastic minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.

### [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](/research/what-neural-networks-memorize)

*Vitaly Feldman, Chiyuan Zhang*

Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. 

In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).

## Talks and Workshops

### [Machine Learning for Mobile Health Workshop](https://sites.google.com/view/ml4mobilehealth-neurips-2020/home)
This workshop, held on December 12, aimed to assemble researchers from the key areas in mobile health to better address the challenges currently facing the widespread use of mobile health technologies. We presented our workshop accepted paper which discusses synthetic data generated using a realistic ECG simulator and a structured noise model. 

**Machine Learning for Mobile Health Workshop Accepted Paper:** 
[Representing and Denoising Wearable ECG Recordings](/research/wearable-ecg-recordings) *Jeffrey Chan, Andrew C. Miller, Emily Fox*

### [Privacy Preserving Machine Learning (PPML) Workshop](https://ppml-workshop.github.io/#page-top)
This workshop, held on December 11, focused on privacy preserving techniques for machine learning and disclosure in large scale data analysis both in the distributed and centralized settings. We will be presenting our workshop accepted papers which discusses how considering sequential setting in which a single dataset of individuals is used to perform adaptively-chosen analyses, and how random shuffling of input data amplifies differential privacy guarantees.

**PPML Workshop Accepted Papers:** 
[Individual Privacy Accounting via a Renyi Filter](/research/individual-privacy-accounting) 
*Vitaly Feldman, Tijana Zrnic* 

[A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling](/research/privacy-amplification-by-shuffling) 
*Vitaly Feldman, Audra McMillan, Kunal Talwar*

### [Offline Reinforcement Learning Workshop](https://offline-rl-neurips.github.io/)
This workshop, held on December 12, brought attention to offline Reinforcement Learning. This workshop facilitated discussions surrounding algorithmic challenges, solutions and real-world applications. 

**Offline Reinforcement Learning Workshop Accepted Paper:** 
[Uncertainty Weighted Offline Reinforcement Learning](/research/offline-reinforcement-learning) 
*Yue Wu, Shuangfei Zhai, Nitish Srivastava, Josh Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh* 

## Expo Day
**Accelerated Training with ML Compute on M1-Powered Mac** 
Apple presented a talk on Accelerated Training with ML Compute on M1-Powered Mac during NeurIPS Expo Day on December 6.

In November, Apple announced Mac powered by the M1 chip, featuring a powerful machine learning accelerator and high-performance GPU. ML Compute, a new framework available in macOS Big Sur, enables developers to accelerate the training of neural networks using the CPU and GPU.

In this talk, we discussed how we use ML Compute to speed up the training of ML models on M1-powered Mac with popular deep learning frameworks such as TensorFlow. We showed how to replace the TensorFlow ops in graph and eager mode with an ML Compute graph. We also present the performance and watt improvements when training neural networks on Mac with M1. Finally, we examined how unified memory and other memory optimizations on M1-powered Mac allow us to minimize the memory footprint when training neural networks. Learn more about how we [leverage ML Compute for Accelerated Training on Mac.](/updates/ml-compute-training-on-mac)

<div class="callout">

## Affinity Group Workshops

Apple sponsored the [Black in AI](http://blackinai2020.vercel.app/), [LatinX in AI](https://www.latinxinai.org/neurips-2020), [Queer in AI](https://sites.google.com/view/queer-in-ai/), and [Women in Machine Learning](https://wimlworkshop.org/neurips2020/) workshops throughout the week.

At the Women in Machine Learning workshop on December 9, we gave a talk on how our on-device machine learning powers intelligent experiences on Apple products.

[Learn more about Apple’s company-wide inclusion and diversity efforts](https://www.apple.com/diversity/)

</div>

Apple sponsored the Neural Information Processing Systems (NeurIPS) conference, which was held virtually from December 6 to 12. NeurIPS is a global conference focused on fostering the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.

Apple at NeurIPS 2020

## ML Compute

Until now, TensorFlow has only utilized the CPU for training on Mac. The new tensorflow_macos fork of TensorFlow 2.4 leverages ML Compute to enable machine learning libraries to take full advantage of not only the CPU, but also the GPU in both M1- and Intel-powered Macs for dramatically faster training performance. This starts by applying higher-level optimizations such as fusing layers, selecting the appropriate device type and compiling and executing the graph as primitives that are accelerated by BNNS on the CPU and Metal Performance Shaders on the GPU. 

## Training Performance with Mac-optimized TensorFlow

Performance benchmarks for Mac-optimized TensorFlow training show significant speedups for common models across M1- and Intel-powered Macs when leveraging the GPU for training. For example, TensorFlow users can now get up to 7x faster training on the new 13-inch MacBook Pro with M1:

<figure role="presentation"><a href="/images/posts/accelerated-tensorflow-on-mac/macbook-pro-tensorflow-graph.png" tabindex="-1" target="_blank"><img src="/images/posts/accelerated-tensorflow-on-mac/macbook-pro-tensorflow-graph.png" width="571px" alt="Shows a chart that compares three performance benchmarks. One running TensorFlow 2.3 on 2020 13” MacBook Pro with Intel, another running Accelerated TensorFlow 2.4 on 2020 13” MacBook Pro with Intel, and a third running Accelerated TensorFlow 2.4 on 2020 13” MacBook Pro with M1. Footnote 1 provides more details."></a><figcaption style="font-weight:normal;color:#86868b;">Training impact on common models using ML Compute on M1- and Intel-powered 13-inch MacBook Pro are shown in seconds per batch, with lower numbers indicating faster training time.</figcaption></figure>

<figure role="presentation"><a href="/images/posts/accelerated-tensorflow-on-mac/mac-pro-tensorflow-graph.png" tabindex="-1" target="_blank"><img src="/images/posts/accelerated-tensorflow-on-mac/mac-pro-tensorflow-graph.png" width="571px" alt="Shows a chart that compares two performance benchmarks. One running TensorFlow 2.3 on 2019 Mac Pro and another running Accelerated TensorFlow 2.4 on 2019 Mac Pro. Footnote 2 provides more details."></a><figcaption style="font-weight:normal;color:#86868b;">Training impact on common models using ML Compute on the Intel-powered 2019 Mac Pro are shown in seconds per batch, with lower numbers indicating faster training time.</figcaption></figure>

## Getting started with Mac-optimized TensorFlow

To start using Mac-optimized TensorFlow, visit the [tensorflow_macos](https://github.com/apple/tensorflow_macos) GitHub repository. You can also visit [TensorFlow’s blog post](https://blog.tensorflow.org/2020/11/accelerating-tensorflow-performance-on-mac.html) to learn more.





 

<ol role="contentinfo" aria-label="Footnotes">
<li class="footnote" style="font-size:12px;color:#86868b;padding-bottom:0.41667em">
Testing conducted by Apple in October and November 2020 using a preproduction 13-inch MacBook Pro system with Apple M1 chip, 16GB of RAM, and 256GB SSD, as well as a production 1.7GHz quad-core Intel Core i7-based 13-inch MacBook Pro system with Intel Iris Plus Graphics 645, 16GB of RAM, and 2TB SSD. Tested with prerelease macOS Big Sur, TensorFlow 2.3, prerelease TensorFlow 2.4, ResNet50V2 with fine-tuning, CycleGAN, Style Transfer, MobileNetV3, and DenseNet121. Performance tests are conducted using specific computer systems and reflect the approximate performance of MacBook Pro.</li>

<li class="footnote" style="font-size:12px;color:#86868b;padding-bottom:0.41667em;">Testing conducted by Apple in October and November 2020 using a production 3.2GHz 16-core Intel Xeon W-based Mac Pro system with 32GB of RAM, AMD Radeon Pro Vega II Duo graphics with 64GB of HBM2, and 256GB SSD. Tested with prerelease macOS Big Sur, TensorFlow 2.3, prerelease TensorFlow 2.4, ResNet50V2 with fine-tuning, CycleGAN, Style Transfer, MobileNetV3, and DenseNet121. Performance tests are conducted using specific computer systems and reflect the approximate performance of Mac Pro.</li>
</ol>

The Mac has long been a popular platform for developers, engineers, and researchers. Now, with Macs powered by the all new M1 chip, and the ML Compute framework available in macOS Big Sur, neural networks can be trained right on the Mac with a huge leap in performance.

Machine Learning
Research at Apple.

Recent papers.

Frame-level SpecAugment for Deep Convolutional Neural Networks in Hybrid ASR Systems

A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

On the Generalization of Learning-based 3D Reconstruction

Updates and events.

Apple at NeurIPS 2020

Leveraging ML Compute for Accelerated Training on Mac

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple.

Recent papers.

Frame-level SpecAugment for Deep Convolutional Neural Networks in Hybrid ASR Systems

A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

On the Generalization of Learning-based 3D Reconstruction

Updates and events.

Apple at NeurIPS 2020

Leveraging ML Compute for Accelerated Training on Mac

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple.