AudioCraft: An Open‑Source PyTorch Library for Audio Generation with MusicGen, AudioGen, and EnCodec

AudioCraft is a PyTorch library for audio generation that integrates three cutting‑edge AI models: MusicGen for music synthesis, AudioGen for sound effects, and EnCodec, a neural audio codec that compresses and reconstructs audio with high fidelity.

The library can be tried directly on Hugging Face Spaces, where users input textual descriptions or reference audio and generate results after a short processing delay.

Architecturally, AudioCraft simplifies audio generation by using EnCodec to convert raw waveforms into discrete token streams, which are then modeled by an autoregressive language model. This approach efficiently captures long‑range dependencies and produces high‑quality audio.

MusicGen employs a single‑stage Transformer language model with a codebook interleaving (CI) pattern, enabling controlled generation from text or melodic cues. Evaluations using a new chroma‑cosine similarity metric and human listening tests show that conditioning on melody improves control without significantly harming quality.

AudioGen is a self‑regressive model that generates audio from text, addressing challenges such as high‑resolution encoding, inference speed, and text‑audio alignment through techniques like multi‑stream modeling and classifier‑free guidance.

EnCodec is a real‑time neural audio codec that achieves high‑fidelity compression across various sampling rates and bandwidths. The paper introduces a streaming encoder‑decoder architecture, a multi‑scale spectral adversarial loss, and a lightweight Transformer for further compression, demonstrating superior subjective quality and reduced artifacts.

Installation can be performed via pip, with optional installation of the bleeding‑edge version from the GitHub repository. The following command block shows the required steps:

# Best to make sure you have torch installed first, in particular before installing xformers.
# Don't run this if you already have PyTorch installed.
pip install 'torch>=2.0'
# Then proceed to one of the following
pip install -U audiocraft  # stable release
pip install -U git+https://[email protected]/facebookresearch/audiocraft#egg=audiocraft  # bleeding edge
pip install -e .  # or if you cloned the repo locally (mandatory if you want to train).

Further resources include the GitHub repository (https://github.com/facebookresearch/audiocraft) and the API documentation (https://facebookresearch.github.io/audiocraft/api_docs/audiocraft/index.html).