View publication

Sign languages are essential for the Deaf and Hard-of-Hearing (DHH) community. Sign language generation systems have the potential to support communication by translating from written languages, such as English, into signed videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures, the absence of facial cues and body language, and insufficient visual and motion fidelity. We address these challenges by building on recent advances in LLMs and video generation models to translate English sentences into natural-looking AI ASL signers. The text component of our model extracts information for manual and non-manual components of ASL, which are used to synthesize skeletal pose sequences and corresponding video frames. Our findings from a user study with 30 DHH participants and thorough technical evaluations demonstrate significant progress and identify critical areas necessary to meet user needs.

Related readings and updates.

ACM Human-Computer Interaction Conference (CHI) 2025

Apple is presenting new research at the ACM annual conference on Human-Computer Interaction (CHI), which takes place in person in Yokohama, Japan, from April 26 to May 1. We are proud to again sponsor the conference, which brings together the scientific and industrial research communities focused on interactive technology. Below is an overview of Apple’s participation at CHI 2025. Schedule Stop by the Apple booth (304 & 305) in the Yokohama…
See event details

Improving How Machine Translations Handle Grammatical Gender Ambiguity

Machine Translation (MT) enables people to connect with others and engage with content across language barriers. Grammatical gender presents a difficult challenge for these systems, as some languages require specificity for terms that can be ambiguous or neutral in other languages. For example, when translating the English word "nurse" into Spanish, one must decide whether the feminine "enfermera" or the masculine "enfermero" is appropriate…
See highlight details