ComfyUI lab 5: Turn sound into reactive motion
Window a waveform, measure local energy or spectrum, map it to frames, and preserve synchronization at a declared FPS.
Listen to this article
On-device voiceUses the voice built into your browser; no article text leaves this page.
Audio narration is not supported by this browser.
Audio reactivity is signal mapping, not magic. A waveform is amplitude over time. Animation is a sequence of frames. The workflow’s job is to summarize the right slice of audio for each frame and map that measurement into a visual parameter.
Time is the shared coordinate
At 12 fps, frame 24 represents two seconds. Split the waveform into frame-aligned windows. RMS energy gives a stable loudness estimate; a short-time Fourier transform separates bass, midrange, and treble; onset detection finds transients. The included MIT node offers three small baselines: waveform, spectrum bars, and an RMS-driven pulse.
The graph caps duration and frame count deliberately. A 60-second clip at 60 fps is 3,600 full-resolution images, which can exhaust ordinary RAM even without diffusion. Prototype six seconds at 12 fps, prove timing, then raise quality.
Map signals musically
Raw values flicker. Normalize against a rolling percentile, add attack/release smoothing, and use different bands for different controls: bass for scale, snare-band transients for cuts, vocals for color. The advanced MIT RyanOnTheInside pack extends this idea to MIDI, masks, particles, optical flow, and model conditioning. AnimateDiff Evolved can receive scheduled influence after the signal layer is understood.
Export audio beside the animation and verify duration exactly. A beautiful reactive render that drifts a frame every few seconds is still broken.
Run the lab
- Open this graph in Comfy Studio
- Download the exact API-format workflow
- CLI:
app comfy download audio-reactive-visualizer
Record the graph JSON, input assets, seed where relevant, model or service version, custom-node commit, and final artifact together. That bundle—not a screenshot—is the reproducible creative unit.
Open-source map
- Open app.nz reactive node — MIT minimal renderer used by the graph
- RyanOnTheInside — MIT advanced audio, MIDI, masks, particles, and motion
- AnimateDiff Evolved — Apache-2.0 diffusion animation system
Creative Systems Labs: 5 of 12. Previous: Build a talking character pipeline Next: Build style as a system