International Conference on Machine Learning (ICML) 2024

Apple is sponsoring the International Conference on Machine Learning (ICML) 2024, which is taking place in person from July 21 to 27 in the Messe Wien Exhibition and Congress Center, Vienna Austria. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. Below is the schedule of our sponsored workshops and events at ICML 2024.

Accepted Papers

Aligning Text-to-Image Diffusion as GFlowNets Dinghuai Zhang (University of Montreal), Yizhe Zhang, Jiatao Gu, Ruixiang Zhang, Josh Susskind, Navdeep Jaitly, Shuangfei Zhai

Careful with that Scalpel: Improving Gradient Surgery with an EMA Pierre Ablin, James Thornton, Eugene Ndiaye, Yu-Guan Hsieh, Michal Klein, Marco Cuturi Cameto

Contrasting Multiple Representations with the Multi-Marginal Matching Gap Zoe Piran, Michal Klein, James Thornton, Marco Cuturi Cameto

Data-free Bootstrapping Distillation Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu (University of Pennsylvania), Josh Susskind

Executable Code Actions Elicit Better LLM Agents Xingyao Wang (University of Illinois Urbana-Champaign), Yangyi Chen (University of Illinois Urbana-Champaign), Lifan Yuan (University of Illinois Urbana-Champaign), Yizhe Zhang, Hao Peng (University of Illinois Urbana-Champaign), Ji Heng (University of Illinois Urbana-Champaign)

How Smooth Is Attention? Valérie Castin (ENS), Pierre Ablin, Gabriel Peyré (ENS)

Improved Modelling of Federated Datasets using Mixtures-of-Dirichlet-Multinomials Jonny Scott (Institute of Science and Technology Austria), Aine Cahill

Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models Raviteja Vemulapalli, Hadi Pour Ansari, Fartash Faghri, Sachin Mehta, Mehrdad Farajtabar, Mohammad Rastegari (Meta (work done while at Apple)), Oncel Tuzel

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation Minsik Cho, Mohammad Rastegari (Meta (work done while at Apple)), Devang Naik

On a Practical Implementation of Brenier’s Polar Factorization Theorem and Its Applications to Optimization and Sampling Marco Cuturi Cameto, Nina Vesseron (ENSAE)

On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions Denys Pushkin (EPFL), Raphael Berthier (EPFL), Emmanuel Abbe (Apple/EPFL)

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework Sachin Mehta, Mohammad Sekhavat, Qingqing Cao, Max Horton, Yanzi Jin, Frank Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal (work done while at Apple), Mohammad Rastegari (Meta (work done while at Apple))

Optimization without Retraction on the Random Generalized Stiefel Manifold Simon Vary (UC Louvain), Pierre Ablin, Bin Gao (Chinese Academy of Sciences), Pierre-Antoine Absil (UC Louvain)

Private Vector Mean Estimation in the Shuffle Model: Optimal Rates Require Many Messages Hilal Asi, Vitaly Feldman, Jelani Nelson (University of California Berkeley), Kunal Talwar, Huy Nguyen (Northeastern University), Samson Zhou (Texas A&M University)

Projected Language Models: A Large Model Pre-Segmented Into Smaller Ones David Grangier, Angelos Katharopoulos, Pierre Ablin, Awni Hannun

Revealing the Utilized Rank of Subspaces of Learning in Neural Networks Isha Garg, Eshan Verma, Daniel Ulbricht, Christian Koguchi

Scalable Pre-training of Large Autoregressive Image Models Alaaeldin Mohamed Elnouby Ali, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista Martin, Josh Susskind, Armand Joulin (Google Deepmind (work done while at Apple))

Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation Thomas Merth, Qichen Fu, Mohammad Rastegari (Meta (work done while at Apple)), Mahyar Najibi

Swallowing the Bitter Pill: Simplified Scalable Conformer Generation Yuyang Wang, Ahmed Elhag (University of Oxford), Navdeep Jaitly, Josh Susskind, Miguel Angel Bautista Martin

Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models Xavier Suau Cuadros, Pieter Delobelle (KU Leuven), Rin Metcalf Susa, Armand Joulin (Google Deepmind (work done while at Apple)), Nick Apostoloff, Luca Zapella, Pau Rodriguez Lopez

Demos

MLX

We are demonstrating large model inference and training on device using MLX. MLX is a flexible array framework that is optimized for Apple silicon, and brought to you by Apple Machine Learning Research. It enables training and inference of arbitrarily complex models on Apple silicon powered devices with great brevity and flexibility.

In this demo we showcase fine-tuning of a 7B parameter LLM on an iPhone, image generation using a large diffusion model on an iPad, and text generation using a number of large language models on a M2 Ultra Mac Studio and M3 Macbook Pro.

Private Federated Learning (PFL)

This demo showcases Apple’s Private Federated Learning (PFL) technology. PFL-research is the open-source framework enabling this technology for research simulations, open sourced in March 2024 at the Apple PPML Workshop. Here we are able to show how Siri can play Music and Podcasts on iPhones, which leverages several technologies such as Siri Signals and Siri Inference, Private Federated Learning (PFL) and Differential Privacy (DP). This highlights a user-facing feature in iOS that was shipped, thanks to this framework.

Acknowledgements

Devon Hjelm, Ozan Sener, Pau Rodriguez Lopez, and Tatiana Likhomanenko are Area Chairs for ICML 2024.

Aadirupa Saha is a Co-Organizer for the Models of Human Feedback for AI Alignment workshop.

Rin Metcalf Susa is a Panelist on the Models of Human Feedback for AI Alignment workshop.

Marco Cuturi, Samy Bengio, and Vladlen Kotlun are Senior Meta Reviewers for ICML 2024.

Arno Blaas, Bailin Wang, Fartash Faghri, Gustaf Ahdritz, Hilal Asi, Junpei Zhou, Luca Zappella, Miguel Angel Bautista Martin, Miguel Sarabia del Castillo, Parikshit Gopalan, Qichen Fu, Ray Zhang, Richard Bai, Rin Metcalf Susa, Vimal Thilak, Xavier Suau Cuadros, Yu-Guan Hsieh, Yuyang Wang, and Zoe Piran are reviewers for ICML 2024.

Natalie Schluter is a Senior Workshop Chair for ICML 2024.