View publication

This paper was accepted at the NeurIPS 2023 workshop on Diffusion Models.

We demonstrate how conditional generation from diffusion models can be used to tackle a variety of realistic tasks in the production of music in 44.1kHz stereo audio with sampling-time guidance. The scenarios we consider include continuation, inpainting and regeneration of musical audio, the creation of smooth transitions between two different music tracks, and the transfer of desired stylistic characteristics to existing audio clips. We achieve this by applying guidance at sampling time in a simple framework that supports both reconstruction and classification losses, or any combination of the two. This approach ensures that generated audio can match its surrounding context, or conform to a class distribution or latent representation specified relative to any suitable pre-trained classifier or embedding model.

We show randomly chosen samples for a number of creative applications in Table 1, each conditioned on a given audio prompt. For each task and prompt we show samples from the different models described in the paper.

Task types:

  • infill: replace the middle two seconds of the prompt
  • regeneration: regenerate the middle two seconds of the prompt
  • continuation: generate a new continuation starting from the first 2.4s of the prompt
  • transitions: regenerate a crossfaded section between two tracks
  • guidance: generate a new clip conditioned on the PaSST classifier embedding of the prompt
prompttaskCQTDiff (baseline)latentwaveform
infill
infill
infill
regenerate
regenerate
regenerate
continuation
continuation
continuation
transitions
transitions
transitions
guidance
guidance
guidance

Prompts are drawn from a test split of the Free Music Archive dataset, published by Michaël Defferrard et al. under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

Related readings and updates.

Prompting for a Conversation: How to Control a Dialog Model?

Dialog modelling faces a difficult trade-off. Models are trained on a large amount of text, yet their responses need to be limited to a desired scope and style of a dialog agent. Because the datasets used to achieve the former contain language that is not compatible with the latter, pre-trained dialog models are fine-tuned on smaller curated datasets. However, the fine-tuning process robs them of the ability to produce diverse responses…
See paper details

Mel Spectrogram Inversion with Stable Pitch

Vocoders are models capable of transforming a low-dimensional spectral representation of an audio signal, typically the mel spectrogram, to a waveform. Modern speech generation pipelines use a vocoder as their final component. Recent vocoder models developed for speech achieve a high degree of realism, such that it is natural to wonder how they would perform on music signals. Compared to speech, the heterogeneity and structure of the musical…
See paper details