Abstract
AI-driven song generation has advanced rapidly, but practical music production requires iterative editing of existing audio, including segment replacement, track-conditioned reconstruction, lyric revision, and reference-audio style transfer. Supporting these operations within a single autoregressive model is challenging because they require heterogeneous generation behaviors, different objectives across edited and unedited regions, and online boundary decisions when the replacement duration is unknown. We present SongGen-X, a unified framework that augments a frozen autoregressive song-generation backbone with a Mixture-of-Adapters (MoA). The MoA comprises four experts specialized by region function (clone, refine, inpaint, and style) and recombines them across tasks, allowing region-aware routing to separate context preservation from edit-region generation. A Dynamic Routing Head selects the target-side expert at each autoregressive step to support adaptive-duration editing, while Route-Loss Scheduling balances routing and audio modeling over training and Transition-Aware Weighting emphasizes sparse boundary transitions. Trained jointly on approximately 2,550 hours of English songs, SongGen-X uses 23.6M trainable parameters and supports fixed- and adaptive-duration inpainting, track-conditioned refinement, reference-audio style transfer, and density-preserving and density-varying lyric editing. Compared with LoRA fine-tuning on the same backbone and training data, SongGen-X improves lyric intelligibility, musicality, and preservation of unedited audio in both inpainting settings, while providing better partial-track control in refinement. In style transfer, SongGen-X achieves reference-style similarity comparable to ACE-Step 1.5 with fewer lyric errors. Despite being trained only on English songs, SongGen-X transfers reference style to Chinese lyrics in the evaluated setting.
Explore the Audio Demos
Jump directly to a supported editing capability or the exploratory extension.
Demo data. All audio examples below are drawn from our test set; the lyrics were transcribed by SongPrep.
Important Lyric-editing demos are included below
Lyric editing modifies the lyrics within a selected song segment while preserving the surrounding musical context. When the replacement lyrics have a phoneme density similar to the original lyrics, the edit uses fixed-duration inpainting or accompaniment-conditioned refinement; substantially longer or shorter replacements use adaptive-duration inpainting. These examples are therefore integrated into the Inpainting and Refinement sections rather than presented as a separate task. Text shown in red marks target lyrics whose corresponding audio is unavailable in the task-specific reference.
1. Inpainting
Problem definition. Inpainting regenerates a specified segment of a song from its surrounding musical context. In fixed-duration inpainting, the user provides an edit interval, and the generated replacement must preserve the duration of that interval. In adaptive-duration inpainting, the replacement length is not specified in advance, and the model must determine when to end generation and reconnect to the subsequent context. In both settings, the unedited regions should be preserved and the generated content should connect coherently to the preceding and following context.
Columns. Lyrics shows the supplied lyric condition, with the edited content highlighted. Task Reference is the model input: the edit region is replaced with silence for fixed-duration inpainting and cropped for adaptive-duration inpainting. SongGen-X Output is the completed song generated from the task reference and lyric condition. For adaptive-duration references, Context split marks the boundary between the preceding and following audio after cropping; output intervals indicate the generated edit region.
1.1 Fixed Duration
Loading samples…
1.2 Adaptive Duration
Loading samples…
2. Refinement
Problem definition. Refinement completes a specified segment from a partial track, such as a vocals-only, accompaniment-only, or drums-only signal. The model must retain the provided track while generating complementary musical components. It should preserve continuity at the edit boundaries and maintain local consistency between the conditioning track and the generated components.
Columns. Lyrics shows the supplied lyric condition and highlights the edited content. Task Reference contains source audio outside the edit interval and the available partial track—vocals, drums, or accompaniment—inside it. SongGen-X Output is the completed song, retaining the provided track while generating the complementary components.
2.1 Vocal
Loading samples…
2.2 Drum
Loading samples…
2.3 Accompaniment
Loading samples…
3. Style Transfer
Problem definition. Style transfer uses a reference audio clip to guide high-level attributes of the target song, including genre, mood, and instrumentation. The objective is not to reproduce the reference melody, lyrics, singer identity, or exact performance. Instead, the generated song should follow the target lyrics and content conditions while reflecting stylistic characteristics of the reference.
Columns. Source Lyrics shows the English lyrics associated with the style reference. Target Lyrics, when present, shows the rewritten lyrics supplied for generation. Style Reference is the audio clip from which CLaMP3 style features are extracted. SongGen-X Output is generated from the target lyrics under the extracted reference style. In the original-lyrics setting, the source lyrics are also used as the target lyrics.
3.1 Original Lyrics
Loading samples…
3.2 Rewritten Lyrics
Loading samples…
3.3 Chinese Lyrics
Loading samples…
4. Exploratory Extension: Cover Song Generation
Exploratory setup. To explore cover song generation, we extend SongGen-X with an additional adapter set conditioned on Vevo2 prosody tokens while keeping the autoregressive backbone frozen. This experiment is included to demonstrate how efficiently the framework can incorporate a new conditioning signal, rather than to present cover song generation as a mature core capability.
Columns. Lyrics shows the lyric condition. Melody Reference is the vocal audio used to obtain Vevo2 prosody tokens. SongGen-X Output is the generated cover song conditioned on those tokens and lyrics.
This preliminary experiment illustrates the efficient extensibility of SongGen-X, but the results remain unstable. Better melody representations, more training data, and stronger data augmentation strategies require further exploration.
Loading samples…