HOW I BUILT SONUS: AUDIO-REACTIVE WEBGPU SHADERS & 3D CINEMATOGRAPHY

1.1A Ferrofluid-Inspired Material

The vgpu glass-fractal example gave me a starting point for experimenting with materials and light. I shifted the focus toward a single dark orb, so changes in its silhouette and highlights would carry the experience.

I describe the material as ferrofluid-inspired, but the sound is not acting as a magnet by itself. Within the fiction of NODUS, the fluid is suspended inside the prism and manipulated by an electromagnetic field whose strength and pole distribution are modulated by the music.

This distinction gives the material room to behave expressively: it can form magnetic-looking peaks, breathe, recoil, and redistribute its mass without remaining attached to one permanent magnet. I think of it as a magnetoreactive fluid, first confined by glass and later released by the music.

1.2Simulating the Confined Fluid

The render shader uses 256 distributed directions as reference points for the magnetic peaks. Beneath that surface, a 128 × 64 WebGPU compute grid stores radial displacement, velocity, magnetization, and wave age between frames. The orb therefore has memory: an impact can travel across it and decay instead of being recalculated as an unrelated shape every frame.

It is still an artistic simulation rather than a scientific ferrohydrodynamics solver. Bass sustain strengthens the field, transients launch capillary-like waves, and spectral changes alter the surface tension and distribution of peaks. Four inner prism planes constrain the final surface, compressing the fluid as it approaches the glass instead of allowing the mesh to pass through it.

1.3Layered Light & Musical Contrast

The overhead beams help define the orb against the dark background. They are built from layered procedural patterns, with different movement and audio modulation creating the impression of light passing through haze. The orb's key light remains fixed above the prism in world space, so orbiting the camera reveals different facets without moving the source with the viewer.

A blue-noise texture adds fine variation to the beams and helps break up visible banding. The result is a stylized volumetric appearance, with the lighting doing as much visual work as the surface itself.

Movement and brightness have separate roles. Between 1:50.5 and 2:03.5, the beams move at 3.6× their normal speed while their intensity falls to 0.05×. Faster motion can feel tense even in near-darkness: the quieter light lets the free orb carry the scene. Its pale highlights shift toward deep red during this passage, and short ramps soften the lighting changes at both ends.

1.4Live Audio Response & a Choreographed Drop

NODUS combines live surface motion with an authored sequence. Bass, mids, treble, overall energy, spectral flux, and longer rises or falls are analysed during playback and drive the fluid continuously. Kick accents add a restrained pulse to the red lighting; a later passage uses individual kicks to build pressure inside the prism.

The larger transitions follow the playback clock. A blackout begins at 0:55.5; FORMA appears from 57.66 to 58.6 seconds, followed by NOXA from 58.5 to 59.5. At 1:00.75, a brief, restrained flash leads into crimson while preserving the silhouette of the fluid and glass. These cues belong to the original soundtrack.

That combination lets the surface react to the details of the music while the lighting follows a deliberate arc. From 2:05, the red mode gradually returns toward monochrome.

Local audio drives the surface and kick detection without inheriting the original track's blackout, text cues, or prism rupture. Press H to toggle the red hardcore mode while listening to your own music. Mouse and touch gestures rotate the view; release velocity carries the camera with friction before it gradually settles.

1.5Pressure, Fracture & Return

From 1:25, each kick gives the fluid a small elastic expansion and leaves a little pressure behind. The pulse recedes, but the accumulated pressure remains. Over eight kicks the fluid approaches the inner walls, taking on the prism's shape before the glass breaks. The kick times come from analysing the original audio and are stored with the choreography, so jumping forward or replaying reconstructs the same pressure and fracture timing.

I cut sixteen fragments from the original rounded glass surface, keeping its normals and adding thin edges. The intact prism and its fragments share material functions for transmission, absorption, and highlights. Moving fragments can also catch a subtle reflection of the fluid through small planar captures rendered from mirrored cameras. The glass should still feel like the same material after it leaves the orb.

Once released, the orb keeps its surface motion and loses the shadow associated with its enclosure. From 2:05, the fragments return over eight seconds as the fluid settles back inside and the scene fades toward white. Refraction returns gradually after the shell closes, avoiding a doubled image of the orb during reconstruction.

1.6Keeping the Motion Continuous

The simulation advances in small fixed steps at 120 Hz, with a bounded time accumulator between rendered frames. Musical impulses enter once rather than repeating for every simulation step. Camera inertia also uses elapsed time, helping the scene retain its pace across different display refresh rates and frame rates.

Pausing holds the choreography and lighting phase while the fluid relaxes; resuming restores its energy with a short blend. Fracture and reconstruction change the enclosure around the existing simulation, preserving the orb's motion through both transitions. When the track ends, the scene gradually returns to its original white lighting and quiet idle state.

1.7Making Music in Sound Studio

Sound Studio lets you make a small musical loop inside NODUS. A sixteen-step sequencer supplies kick, snare, and hi-hat patterns, while a two-octave piano roll adds melody. Both share one audio clock, tempo, and transport, keeping notes and percussion together as the pattern repeats.

Add notes with a click, drag them to change pitch or timing, and pull their right edge to change duration. Holding a note previews its sound as you move it; right-click deletes it. Generate melody suggests notes in C minor, and Variation changes the drum pattern while retaining its main rhythmic anchors. These are starting points to edit and play with.

The synth shapes that melody with waveform, drive, tone, detune, and wet/dry mix controls. Switch to the XY pad to play it freely over the beat. Both sources feed the scene's audio analysis, so the fluid responds to the music you make; H adds the red hardcore lighting. Separate volume controls use a gradual gain curve, and closing the studio stops its sounds.

2.1Virtual Cinematography as an Instrument

For the second experiment, I wanted to step away from abstract fluid shaders and explore figurative 3D form and automated cinematography.

I chose a classical sculpture of an angel and used framing, light, and timed camera cuts to explore its form. The model stays still; the changing viewpoint gives the sequence its movement and tension.

2.2Six Camera Framings & Motion

I composed six framings around the angel: hero, face, profile, wing, low, and wide. A timeline selects the framing according to the track position, allowing close-ups and wider views to follow the musical phrases.

Within each shot, small sinusoidal offsets keep the camera moving. A subtle forward pulse follows a fixed 150 BPM grid. The final movement blends back toward the hero framing using a smooth transition.

2.3Sculpting the Angel with Light

A low ambient light and three directional lights reveal the sculpture against black. A white main light defines the form, a cooler fill brings detail into the shaded areas, and a rear light helps separate the silhouette.

The lighting rig stays fixed while the camera moves through the scene. Bloom, subtle chromatic aberration, grain, and a vignette give the image its final atmosphere.

2.4A Timeline Shared with the Lyrics

Camera cuts and lyric cues are authored against the playback time of Vladimir Cauchemar’s Laboratoires. They share the same clock, so the images and words can follow the phrasing of the track.

LABORATORES is a choreographed sequence: its camera follows timed cues and a fixed rhythmic grid. NODUS puts more emphasis on live audio analysis. These two approaches let me explore different relationships between music and a 3D scene.

You can also make your own timing edits in the public timeline: move lyric clips, resize their duration, and shift camera cuts. Loop a selected lyric to try a change, then use Undo or Reset to revisit it. Edits last for the current session; export a JSON preset to keep them and import it to continue later.

3.1Light as an Interactive Material

The third experiment explores an interactive scene built around light. I am working on how lighting shapes the space, reveals forms, and changes the atmosphere as you interact with the scene.

3.2Scene Exploration & Development

The current work focuses on the scene, its lighting, and the ways you can interact with them. The visual direction and controls are still being explored; this experiment is in development and is not yet available in the public experience.