Images with shared characteristics naturally form sets. For example, in a face verification benchmark, images of the same identity form sets. For generative models, the standard way of dealing with sets is to represent each as a one hot vector, and learn a conditional generative model p(x|y). This representation assumes that the number of sets is limited and known, such that the distribution over sets reduces to a simple multinomial distribution. In contrast, we study a more generic problem where the number of sets is large and unknown. We introduce Set Distribution Networks (SDNs), a novel framework that learns to autoencode and freely generate sets. We achieve this by jointly learning a set encoder, set discriminator, set generator, and set prior. We show that SDNs are able to reconstruct image sets that preserve salient attributes of the inputs in our benchmark datasets, and are also able to generate novel objects/identities. We examine the sets generated by SDN with a pre-trained 3D reconstruction network and a face verification network, respectively, as a novel way to evaluate the quality of generated sets of images.

Shuangfei Zhai, Walter Talbott, Miguel Angel Bautista, Carlos Guestrin, Josh M. Susskind

Set Distribution Networks: a Generative Model for Sets of Images   

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.

Learning Lines, Planes, and Simplexes of Neural Networks

The ability to automatically detect stuttering events in speech could help speech pathologists track an individual’s fluency over time or help improve speech recognition systems for people with atypical speech patterns. Despite increasing interest in this area, existing public datasets are too small to build generalizable dysfluency detection systems and lack sufficient annotations. In this work, we introduce Stuttering Events in Podcasts (SEP-28k), a dataset containing over 28k clips labeled with five event types including blocks, prolongations, sound repetitions, word repetitions, and interjections. Audio comes from public podcasts largely consisting of people who stutter interviewing other people who stutter. We benchmark a set of acoustic models on SEP-28k and the public FluencyBank dataset and highlight how simply increasing the amount of training data improves relative detection performance by 28% and 24% F1 on each. Annotations from over 32k clips across both datasets will be publicly released.

Sep-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter

Conversational question answering (QA) requires the ability to correctly interpret a question in the context of previous conversation turns. We address the conversational QA task by decomposing it into question rewriting and question answering subtasks. The question rewriting (QR) subtask is specifically designed to reformulate ambiguous questions, which depend on the conversational context, into unambiguous questions that can be correctly interpreted outside of the conversational context. We introduce a conversational QA architecture that sets the new state of the art on the TREC CAsT 2019 passage retrieval dataset. Moreover, we show that the same QR model improves QA performance on the QuAC dataset with respect to answer span extraction, which is the next step in QA after passage retrieval. Our evaluation results indicate that the QR model we proposed achieves near human-level performance on both datasets and the gap in performance on the end-to-end conversational QA task is attributed mostly to the errors in QA.


Question Rewriting for End to End Conversational Question Answering

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

[Learn more about NeurIPS](https://neurips.cc/Conferences/2020)

Visit our NeurIPS 2020 [virtual booth](https://neurips.cc/ExpoConferences/2020/Sponsorpage?id=654)



## Conference Accepted Papers

### [Collegial Ensembles](/research/collegial-ensembles)

*Etai Littwin, Ben Myara, Sima Sabah, Joshua Susskind, Shuangfei Zhai, Oren Golan*
 
Modern neural network performance typically improves as model size increases. A recent line of research on the Neural Tangent Kernel (NTK) of over-parameterized networks indicates that the improvement with size increase is a product of a better conditioned loss landscape. In this work, we investigate a form of over-parameterization achieved through ensembling, where we define collegial ensembles (CE) as the aggregation of multiple independent models with identical architectures, trained as a single model. We show that the optimization dynamics of CE simplify dramatically when the number of models in the ensemble is large, resembling the dynamics of wide models, yet scale much more favorably. We use recent theoretical results on the finite width corrections of the NTK to perform efficient architecture search in a space of finite width CE that aims to either minimize capacity, or maximize trainability under a set of constraints. The resulting ensembles can be efficiently implemented in practical architectures using group convolutions and block diagonal layers. Finally, we show how our framework can be used to analytically derive optimal group convolution modules originally found using expensive grid searches, without having to train a single model.

### [Faster Differentially Private Samplers via Rényi Divergence Analysis of Discretized Langevin MCMC](/research/differentially-private-samplers)

*Arun Ganesh, Kunal Talwar*
 
Various differentially private algorithms instantiate the exponential mechanism, and require sampling from the distribution exp(−f) for a suitable function f. When the domain of the distribution is high-dimensional, this sampling can be computationally challenging. Using heuristic sampling schemes such as Gibbs sampling does not necessarily lead to provable privacy. When f is convex, techniques from log-concave sampling lead to polynomial-time algorithms, albeit with large polynomials. Langevin dynamics-based algorithms offer much faster alternatives under some distance measures such as statistical distance. In this work, we establish rapid convergence for these algorithms under distance measures more suitable for differential privacy. For smooth, strongly-convex f, we give the first results proving convergence in Rényi divergence. This gives us fast differentially private algorithms for such f. Our techniques and simple and generic and apply also to underdamped Langevin dynamics.


### [On the Error Resistance of Hinge-Loss Minimization](/research/error-resistance)

*Kunal Talwar*

Commonly used classification algorithms in machine learning, such as support vector machines, minimize a convex surrogate loss on training examples. In practice, these algorithms are surprisingly robust to errors in the training data. In this work, we identify a set of conditions on the data under which such surrogate loss minimization algorithms provably learn the correct classifier. This allows us to establish, in a unified framework, the robustness of these algorithms under various models on data as well as error. In particular, we show that if the data is linearly classifiable with a slightly non-trivial margin (i.e. a margin at least C/d‾‾√ for d-dimensional unit vectors), and the class-conditional distributions are near isotropic and logconcave, then surrogate loss minimization has negligible error on the uncorrupted data even when a constant fraction of examples are adversarially mislabeled.

### [Stability of Stochastic Gradient Descent on Nonsmooth Convex Losses](/research/stochastic-gradient-descent)

*Raef Bassily, Vitaly Feldman, Criztobal Guzman, Kunal Talwar*

Uniform stability is a notion of algorithmic stability that bounds the worst case change in the model output by the algorithm when a single data point in the dataset is replaced. An influential work of Hardt et al. (2016) provides strong upper bounds on the uniform stability of the stochastic gradient descent (SGD) algorithm on sufficiently smooth convex losses. These results led to important progress in understanding of the generalization properties of SGD and several applications to differentially private convex optimization for smooth losses. 

### [Stochastic Optimization with Laggard Data Pipelines](/research/stochastic-optimization)

*Naman Agarwal, Rohan Anil, Tomer Koren, Kunal Talwar, Cyril Zhang*

State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes. As a consequence, CPU-bound preprocessing and disk/memory/network operations have emerged as new performance bottlenecks, as opposed to hardware-accelerated gradient computations. In this regime, a recently proposed approach is data echoing (Choi et al., 2019), which takes repeated gradient steps on the same batch while waiting for fresh data to arrive from upstream. We provide the first convergence analyses of "data-echoed" extensions of common optimization methods, showing that they exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with stochastic minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.

### [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](/research/what-neural-networks-memorize)

*Vitaly Feldman, Chiyuan Zhang*

Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. 

In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).

## Talks and Workshops

### [Machine Learning for Mobile Health Workshop](https://sites.google.com/view/ml4mobilehealth-neurips-2020/home)
This workshop, held on December 12, aimed to assemble researchers from the key areas in mobile health to better address the challenges currently facing the widespread use of mobile health technologies. We presented our workshop accepted paper which discusses synthetic data generated using a realistic ECG simulator and a structured noise model. 

**Machine Learning for Mobile Health Workshop Accepted Paper:** 
[Representing and Denoising Wearable ECG Recordings](/research/wearable-ecg-recordings) *Jeffrey Chan, Andrew C. Miller, Emily Fox*

### [Privacy Preserving Machine Learning (PPML) Workshop](https://ppml-workshop.github.io/#page-top)
This workshop, held on December 11, focused on privacy preserving techniques for machine learning and disclosure in large scale data analysis both in the distributed and centralized settings. We will be presenting our workshop accepted papers which discusses how considering sequential setting in which a single dataset of individuals is used to perform adaptively-chosen analyses, and how random shuffling of input data amplifies differential privacy guarantees.

**PPML Workshop Accepted Papers:** 
[Individual Privacy Accounting via a Renyi Filter](/research/individual-privacy-accounting) 
*Vitaly Feldman, Tijana Zrnic* 

[A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling](/research/privacy-amplification-by-shuffling) 
*Vitaly Feldman, Audra McMillan, Kunal Talwar*

### [Offline Reinforcement Learning Workshop](https://offline-rl-neurips.github.io/)
This workshop, held on December 12, brought attention to offline Reinforcement Learning. This workshop facilitated discussions surrounding algorithmic challenges, solutions and real-world applications. 

**Offline Reinforcement Learning Workshop Accepted Paper:** 
[Uncertainty Weighted Offline Reinforcement Learning](/research/offline-reinforcement-learning) 
*Yue Wu, Shuangfei Zhai, Nitish Srivastava, Josh Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh* 

## Expo Day
**Accelerated Training with ML Compute on M1-Powered Mac** 
Apple presented a talk on Accelerated Training with ML Compute on M1-Powered Mac during NeurIPS Expo Day on December 6.

In November, Apple announced Mac powered by the M1 chip, featuring a powerful machine learning accelerator and high-performance GPU. ML Compute, a new framework available in macOS Big Sur, enables developers to accelerate the training of neural networks using the CPU and GPU.

In this talk, we discussed how we use ML Compute to speed up the training of ML models on M1-powered Mac with popular deep learning frameworks such as TensorFlow. We showed how to replace the TensorFlow ops in graph and eager mode with an ML Compute graph. We also present the performance and watt improvements when training neural networks on Mac with M1. Finally, we examined how unified memory and other memory optimizations on M1-powered Mac allow us to minimize the memory footprint when training neural networks. Learn more about how we [leverage ML Compute for Accelerated Training on Mac.](/updates/ml-compute-training-on-mac)

<div class="callout">

## Affinity Group Workshops

Apple sponsored the [Black in AI](http://blackinai2020.vercel.app/), [LatinX in AI](https://www.latinxinai.org/neurips-2020), [Queer in AI](https://sites.google.com/view/queer-in-ai/), and [Women in Machine Learning](https://wimlworkshop.org/neurips2020/) workshops throughout the week.

At the Women in Machine Learning workshop on December 9, we gave a talk on how our on-device machine learning powers intelligent experiences on Apple products.

[Learn more about Apple’s company-wide inclusion and diversity efforts](https://www.apple.com/diversity/)

</div>

Apple sponsored the Neural Information Processing Systems (NeurIPS) conference, which was held virtually from December 6 to 12. NeurIPS is a global conference focused on fostering the exchange of research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects.

Apple at NeurIPS 2020

## ML Compute

Until now, TensorFlow has only utilized the CPU for training on Mac. The new tensorflow_macos fork of TensorFlow 2.4 leverages ML Compute to enable machine learning libraries to take full advantage of not only the CPU, but also the GPU in both M1- and Intel-powered Macs for dramatically faster training performance. This starts by applying higher-level optimizations such as fusing layers, selecting the appropriate device type and compiling and executing the graph as primitives that are accelerated by BNNS on the CPU and Metal Performance Shaders on the GPU. 

## Training Performance with Mac-optimized TensorFlow

Performance benchmarks for Mac-optimized TensorFlow training show significant speedups for common models across M1- and Intel-powered Macs when leveraging the GPU for training. For example, TensorFlow users can now get up to 7x faster training on the new 13-inch MacBook Pro with M1:

<figure role="presentation"><a href="/images/posts/accelerated-tensorflow-on-mac/macbook-pro-tensorflow-graph.png" tabindex="-1" target="_blank"><img src="/images/posts/accelerated-tensorflow-on-mac/macbook-pro-tensorflow-graph.png" width="571px" alt="Chart comparing three performance benchmarks: one running TensorFlow 2.3 on 2020 13” MacBook Pro with Intel, another running Accelerated TensorFlow 2.4 on 2020 13” MacBook Pro with Intel, and a third running Accelerated TensorFlow 2.4 on 2020 13” MacBook Pro with M1, showing up to 7x faster training. Footnote 1 provides more details."></a><figcaption style="font-weight:normal;color:#86868b;">Training impact on common models using ML Compute on M1- and Intel-powered 13-inch MacBook Pro are shown in seconds per batch, with lower numbers indicating faster training time.</figcaption></figure>

<figure role="presentation"><a href="/images/posts/accelerated-tensorflow-on-mac/mac-pro-tensorflow-graph.png" tabindex="-1" target="_blank"><img src="/images/posts/accelerated-tensorflow-on-mac/mac-pro-tensorflow-graph.png" width="571px" alt="Chart comparing two performance benchmarks: one running TensorFlow 2.3 on 2019 Mac Pro and another running Accelerated TensorFlow 2.4 on 2019 Mac Pro, showing up to 7x faster training for common models. Footnote 2 provides more details."></a><figcaption style="font-weight:normal;color:#86868b;">Training impact on common models using ML Compute on the Intel-powered 2019 Mac Pro are shown in seconds per batch, with lower numbers indicating faster training time.</figcaption></figure>

## Getting started with Mac-optimized TensorFlow

To start using Mac-optimized TensorFlow, visit the [tensorflow_macos](https://github.com/apple/tensorflow_macos) GitHub repository. You can also visit [TensorFlow’s blog post](https://blog.tensorflow.org/2020/11/accelerating-tensorflow-performance-on-mac.html) to learn more.





 

<ol role="contentinfo" aria-label="Footnotes">
<li class="footnote" style="font-size:12px;color:#86868b;padding-bottom:0.41667em">
Testing conducted by Apple in October and November 2020 using a preproduction 13-inch MacBook Pro system with Apple M1 chip, 16GB of RAM, and 256GB SSD, as well as a production 1.7GHz quad-core Intel Core i7-based 13-inch MacBook Pro system with Intel Iris Plus Graphics 645, 16GB of RAM, and 2TB SSD. Tested with prerelease macOS Big Sur, TensorFlow 2.3, prerelease TensorFlow 2.4, ResNet50V2 with fine-tuning, CycleGAN, Style Transfer, MobileNetV3, and DenseNet121. Performance tests are conducted using specific computer systems and reflect the approximate performance of MacBook Pro.</li>

<li class="footnote" style="font-size:12px;color:#86868b;padding-bottom:0.41667em;">Testing conducted by Apple in October and November 2020 using a production 3.2GHz 16-core Intel Xeon W-based Mac Pro system with 32GB of RAM, AMD Radeon Pro Vega II Duo graphics with 64GB of HBM2, and 256GB SSD. Tested with prerelease macOS Big Sur, TensorFlow 2.3, prerelease TensorFlow 2.4, ResNet50V2 with fine-tuning, CycleGAN, Style Transfer, MobileNetV3, and DenseNet121. Performance tests are conducted using specific computer systems and reflect the approximate performance of Mac Pro.</li>
</ol>

The Mac has long been a popular platform for developers, engineers, and researchers. Now, with Macs powered by the all new M1 chip, and the ML Compute framework available in macOS Big Sur, neural networks can be trained right on the Mac with a huge leap in performance.

Machine Learning
Research at Apple.

Recent papers.

Set Distribution Networks: a Generative Model for Sets of Images

Learning Lines, Planes, and Simplexes of Neural Networks

Sep-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter

Question Rewriting for End to End Conversational Question Answering

Updates and events.

Apple at NeurIPS 2020

Leveraging ML Compute for Accelerated Training on Mac

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple.

Recent papers.

Set Distribution Networks: a Generative Model for Sets of Images

Learning Lines, Planes, and Simplexes of Neural Networks

Sep-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter

Question Rewriting for End to End Conversational Question Answering

Updates and events.

Apple at NeurIPS 2020

Leveraging ML Compute for Accelerated Training on Mac

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple.