FoleyForge edits sound effects with words, balancing change against fidelity

Editing rather than starting over

Generating a sound effect from a description is different from refining one that already fits a scene. Recreating a clip can change its timing, texture or character, undermining work a sound designer has already done. FoleyForge tackles that narrower task: take an individual sound effect, follow an instruction, and preserve what the instruction leaves untouched.

Giovana Morais and colleagues at Adobe and New York University describe the system in a preprint submitted to arXiv on November 5, 2026. The supplied record identifies version one and does not establish peer-reviewed publication or conference acceptance.

The model handles signal-processing changes, such as filtering and reverberation, alongside more creative transformations, such as changing a door sound from metal to wood. Its central challenge is not simply making an audible difference, but making the requested difference without losing the source.

Building examples of useful edits

The researchers assembled 464,000 training triplets, each containing an input sound, an edited output and a written instruction. The underlying source material was an internally licensed sound-effects collection with human-written descriptions.

For conventional digital signal processing, or DSP, the team prompted Qwen3-Omni to propose instructions achievable with permitted operations. Software then applied those operations to produce the edited audio.

For transformations beyond DSP, Qwen proposed instructions and descriptions of the desired result. Sketch2Sound synthesized related input and output clips, using a shared energy contour or the same starting noise to encourage similarity. The pipeline restricted these transformations to related subjects, actions, locations or materials rather than unrelated substitutions.

Additional examples combined multiple changes, reversed edits or requested no change. These were intended to improve robustness, including the ability to leave a clip alone when asked.

How the model keeps the source in view

FoleyForge fine-tunes a rectified-flow version of Sketch2Sound. This generative model works with compressed representations of audio rather than directly manipulating every waveform sample.

The system places a clean representation of the source before the noisy representation it will turn into the output. That gives the model access to the original without requiring input and output to share an identical time grid or duration—potentially useful when an effect needs a longer reverberation tail.

Editing begins by adding noise to the encoded input, then generating from that intermediate state. More starting noise allows greater changes but generally weakens resemblance to the source.

Users can also steer relative loudness, edit strength and how closely the output follows the input’s changing amplitude contour. Phrases ranging from “a subtle” to “a drastic” edit provide another way to request magnitude. In control sweeps, stronger settings produced progressively larger changes. These controls interact: following the source’s amplitude contour more closely reduced the range available to other adjustments. The main benchmark configuration did not use the separate loudness and edit-strength scalar controls.

What the tests found

Evaluation used a 100-example Subtle Set with reference outputs and a 90-example Creative Set without reference outputs. Comparisons included SAO-Instruct, SDEdit and EchoEdit+, with automatic measurements, AI judges and a small listening study.

FoleyForge received the highest instruction-adherence score from the Qwen judge on both sets. On creative edits, both Qwen and the separate Gemini judge favored it over the comparison systems. However, the judges disagreed on subtle edits: Gemini rated SAO-Instruct higher.

Preservation was not uniformly better. EchoEdit+ retained more source similarity but often made changes the researchers considered too slight. Standard FoleyForge made stronger edits while retaining more similarity than SDEdit. A variant called FoleyForge+, using EchoEdit+’s coupling method, moved toward greater preservation but lower adherence scores.

Twenty-five headphone-screened listeners assessed ten creative source clips. FoleyForge scored 3.22 out of five for instruction adherence, ahead of SDEdit’s 2.92 and SAO-Instruct’s 2.69. For audio quality, however, SAO-Instruct scored 3.73 against FoleyForge’s 3.43. Those differences were statistically significant in the authors’ analysis.

Where the evidence stops

The tests support a controllable trade-off, not universal superiority. The Subtle Set came from the training-data pipeline, although its examples did not overlap training or validation. Creative instructions were outside the training distribution, but their input audio came from Sketch2Sound, the model family underlying FoleyForge.

Qwen helped author instructions and also judged results, creating a possible evaluation bias; the independent judge and human study provide useful but incomplete checks. Licensing restrictions prevented a subtle-edit listening study, and the creative listening test covered only three editing systems. Internally licensed training material and a proprietary reference corpus also constrain independent reproduction. The demonstrated scope remains individual sound effects, rather than complete soundtracks.