Dynamic Real-Time Video-to-Audio Generation
for Immersive Interactive Worlds
1Monash University 2Nanjing University 3ShengShu Technology
4Tampere University 5Alibaba Token Hub, Alibaba Group 6Tsinghua University
†Corresponding author
Interactive worlds such as games, AR/VR and 3D engines need sound that reacts to every action as frames are rendered. Recent audio-visual world models generate synchronized audio, but their audio is tied to their own video generators and cannot serve external sources such as game-engine renders or live camera feeds. Streaming video-to-audio (V2A) offers a source-agnostic alternative by generating sound directly from any live pixel stream. To be a usable sound engine, however, it must meet three demands at once: imperceptible latency, precise audio-visual alignment and high audio quality. Existing streaming V2A methods trade these off: shrinking the visual receptive field to cut latency degrades alignment, and denoising each audio patch in isolation limits quality.
We present LiveFoley, a plug-and-play sound engine for interactive worlds. For imperceptible latency, LiveFoley reaches an end-to-end latency of 156 ms on a single A800 GPU, below the human perceptual threshold. For precise alignment, a Trailing Sliding Window turns off-the-shelf segment-level encoders into streaming modules without retraining and matches offline models in temporal alignment (DeSync 0.48 vs. 0.47) at a 0.08 s frontend look-ahead. For audio quality, we widen the denoising window of the local diffusion head: History LocDiT, the default, adds past context at no latency cost, while the optional Future-Warmup LocDiT breaks an autoregressive deadlock to further improve quality at a 0.08 s backend look-ahead. In addition, a Unified Varhop Framework lets one model adapt its latency to the scene, and LiveFoley generalizes to AR/VR and game-engine footage with stable long-form generation.
Segment-level alignment at frame-level latency. Anchoring the window tail to the current frame turns off-the-shelf non-causal encoders (e.g., Synchformer) into streaming modules without retraining: look-ahead drops to one hop (0.08 s) while alignment stays on par with offline models (DeSync 0.48 vs. 0.47 for MMAudio-L).
One model, several latency budgets. A Synchronizer aligns tokens from different hop sizes, so a single set of weights can switch between a small hop for shooter games and a large hop for free roaming. The unified model matches single-hop experts in alignment and cuts visual frontend compute by up to 4×.
Widening the denoising window. History LocDiT adds past patches as context at no latency cost and is the default. Including the upcoming patch causes an autoregressive deadlock, since its condition depends on the current patch. The optional Future-Warmup LocDiT approximates this condition with the current hidden state and the next visual token, further improving quality at a 0.08 s backend look-ahead.
How much delay can a listener tolerate? In our user study, a lag becomes noticeable at about 0.34 s for action sounds, 0.49 s for speech and 0.85 s for ambience during free roaming. Shrinking a chunked window to cut latency, however, sharply hurts alignment, whereas our trailing sliding window does not. Segment-level alignment at frame-level latency: anchoring the window tail to the current frame decouples the frontend look-ahead from the receptive field, so it drops to one hop without retraining the encoder. A Synchronizer lets one model switch among hop sizes. We widen the denoising window of LocDiT beyond a single patch. History LocDiT adds past patches as context at no latency cost and is the default. The optional Future-Warmup LocDiT brings the upcoming patch into the window by conditioning it on the current hidden state and the next visual token, further improving quality at a 0.08 s backend look-ahead.
Motivation
Visual frontend
Audio backend
Motivation. (a) User study: the minimum audio delay at which participants notice the sound lagging behind the video, for action sounds, speech and ambient sounds in free roaming (mean ± 95% CI). (b) On the V-AURA backbone, the chunked window cuts latency by shrinking the window w at a sharp cost in alignment, while our trailing sliding window shortens the hop s and preserves alignment.
In streaming, a segment-level encoder such as Synchformer can encode a segment only after it has fully arrived, so its long receptive field translates directly into frontend look-ahead. The trailing sliding window anchors the window tail to the current frame: the frontend look-ahead drops to one hop (s = 1, 2 or 4 tokens, i.e., 0.08, 0.16 or 0.32 s) while the full temporal context is kept. The Unified Varhop Framework uses a Synchronizer to support all these latency modes in a single model.
Streaming visual frontend. (a) Chunked window: a smaller w lowers latency but narrows the receptive field. (b) Trailing sliding window: look-ahead drops to one hop while the full receptive field w is preserved. (c) Unified Varhop: a Synchronizer maps tokens from different hop sizes into a shared latent space.
At each 12.5 Hz step, a causal LM produces a hidden state hi, and LocDiT denoises the current patch conditioned on it. Prior streaming V2A models denoise each patch in isolation; we widen the denoising window in two directions. History LocDiT, our default, adds previously generated clean patches as context and improves quality at no latency cost. Toward the future, the upcoming patch cannot simply join the window, since its condition depends on the current patch, which is still being denoised. Future-Warmup LocDiT breaks this autoregressive deadlock by conditioning the upcoming patch on hi and the next visual token vi+1; at inference, this warmup patch is discarded and regenerated with the true hi+1. This option further improves quality at the cost of a 0.08 s backend look-ahead.
Audio backend. Left: the causal LM fuses the current visual token and the LocEnc embedding of the previous clean patch into hi; conditioned on it, Independent LocDiT denoises the current patch in isolation. Right: History LocDiT (top) adds clean patches i−2 and i−1 as context without backend look-ahead. Future-Warmup LocDiT (bottom) jointly denoises patch i and a warmup patch i+1, which is conditioned on hi and vi+1 and discarded at inference; waiting for vi+1 costs 0.08 s of backend look-ahead.
Temporal alignment on par with offline models at a 0.08 s frontend look-ahead: DeSync 0.48 (LiveFoley-48k) vs. 0.47 for the offline MMAudio-L, far ahead of MMAudio (online) (0.71), V-AURA (0.69), WorldSonus (0.67) and SoundReactor (1.03). LiveFoley also has the best audio quality among streaming systems and the highest PQ of all systems (6.26). In a listening test on 16 clips, LiveFoley outperforms all online baselines (MMAudio (online) / V-AURA / WorldSonus) on both quality (MOS-Q 3.23 vs. 2.79 / 2.75 / 3.04) and alignment (MOS-A 3.58 vs. 3.15 / 3.02 / 2.96), and approaches the offline MMAudio-L in alignment (3.79). At hop size s = 1, LiveFoley reaches an end-to-end latency of 156 ms on a single A800 GPU, and the optional Future-Warmup LocDiT adds a further 0.08 s (an estimated 236 ms); both remain below the human perceptual threshold. All compute stages together take under 80 ms; the rest is the algorithmic wait for one hop of video.
| Distribution matching | Audio quality | Modality alignment | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Type | Method | Frontend look-ahead (s) | FDPANNs ↓ | FDPaSST ↓ | KLPANNs ↓ | IS ↑ | PQ ↑ | DeSync ↓ | IB ↑ |
| Offline | Frieren | 8.00 | 12.13 | 123.75 | 2.73 | 11.32 | 5.82 | 0.86 | 0.21 |
| AudioX | 10.00 | 12.73 | 121.82 | 2.96 | 13.34 | 6.17 | 1.22 | 0.24 | |
| MMAudio-L | 8.00 | 8.48 | 74.16 | 2.18 | 11.62 | 6.18 | 0.47 | 0.31 | |
| Online | MMAudio (online) | 0.64 | 9.82 | 175.92 | 2.11 | 10.11 | 5.93 | 0.71 | 0.29 |
| V-AURA | 0.64 | 16.43 | 245.25 | 2.28 | 10.82 | 5.74 | 0.69 | 0.28 | |
| SoundReactor† | 0.08 | – | 113.00 | 2.37 | – | – | 1.03 | 0.24 | |
| WorldSonus | 0.10 | 16.32 | 197.68 | 2.54 | 8.73 | 6.11 | 0.67 | 0.26 | |
| LiveFoley-16k | 0.08 | 9.56 | 92.48 | 2.24 | 11.30 | 6.14 | 0.50 | 0.29 | |
| LiveFoley-48k | 0.08 | 9.92 | 128.34 | 2.30 | 11.42 | 6.26 | 0.48 | 0.29 | |
Objective metrics
Listening test
End-to-end latency with History LocDiT at hop s = 1 (0.08 s frontend look-ahead), in ms.
Latency
System-level comparison on the VGGSound test set. Frontend look-ahead is the duration of video that must be buffered before the visual encoder can encode a new chunk or hop (WorldSonus: one 100 ms chunk of three frames at 30 fps); offline models see the entire clip. MMAudio (online) is our training-free online adaptation of MMAudio-L. Bold / underline: best / second best within each group. †Quoted from the original paper (no public checkpoint) under its own evaluation protocol; not ranked. Both LiveFoley variants use History LocDiT with hop size s = 1.
Listening test on the 16 clips in the comparison below (5-point scale; the dashed line separates offline from online systems). LiveFoley leads all online systems on quality and alignment; WorldSonus runs at a comparable frontend look-ahead (0.10 s) but trails LiveFoley most clearly in alignment. LiveFoley approaches the offline MMAudio-L in alignment (3.58 vs. 3.79) but still trails it in quality (3.23 vs. 3.62).
Measured on a single A800 with a simulator that releases frames at 25 fps (History LocDiT, s = 1). The frontend look-ahead of one hop is an algorithmic wait independent of hardware; the remaining compute stages together take under 80 ms. The optional Future-Warmup LocDiT adds 0.08 s of backend look-ahead, giving an estimated 236 ms.
Please turn the sound on. Unless noted, all audio below is generated in streaming mode by LiveFoley-48k (48 kHz stereo): one autoregressive step per 0.08 s of video, with no access to future frames beyond the look-ahead. Clips tagged + text also receive a one-sentence text prompt (the optional text conditioning described in the paper). To let you hear the real latency, the audio of every clip is deliberately delayed by 0.25 s. This exceeds the end-to-end latency of our streaming pipeline on one A800 (156 ms by default; an estimated 236 ms with the optional Future-Warmup LocDiT), so the sound arrives no earlier than it would in a live setting and you can judge for yourself whether the lag is noticeable.
The 16 clips from our listening test. Each card plays one silent video; click a system to hear its audio from the start, and click again to pause. Every system receives the same video as its only input; outputs are trimmed to 8 s and loudness-normalized to −23 LUFS. Offline models see the whole clip before generating; online (streaming) models generate while the video arrives.