Towards a World-English Language Model

Neural Network Language Models (NNLMs) of Virtual Assistants (VAs) are generally language-, region-, and in some cases, device-dependent, which increases the effort to scale and maintain them. Combining NNLMs for one or more of the categories could be one way to improve scalability. In this work, we combine regional variants of English by building a "World English" NNLM. We examine three data sampling techniques and we experiment with adding adapter bottlenecks to the existing production NNLMs to model dialect-specific characteristics and investigate different strategies to train adapters. We find that adapter modules are more effective in modeling dialects than specialized sub-networks containing a set of feedforward layers. Our experimental results show that adapter-based architectures can achieve up to 4.57% Word Error Rate (WER) reduction over single-dialect baselines on head-heavy test sets and up to 8.22% on tail entities.