We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces sequential iterative routing with concurrent iterative routing. When compared to previously proposed routing algorithms, our method improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100, and it performs at-par with a powerful CNN (ResNet-18) with 4x fewer parameters. On a different task of recognizing digits from overlayed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.

Related readings and updates.

Stabilizing Transformer Training by Preventing Attention Entropy Collapse

m*= Equal Contributors Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is…
See paper details

ICLR 2020

Apple sponsored the 8th International Conference on Learning Representations (ICLR) in April 2020, which took place virtually from April 26 - May 1. ICLR focuses on the advancement of representation learning, and this year’s conference included presentations on cutting-edge research on deep learning areas including computer vision, text understanding, data science, and more.

See event details