Spectral Facade is an interactive media installation exploring architectural transitions through gesture‐based controls, generative diffusion algorithms, and dynamically weighted prompts. The project merges two distinct façade styles—one modern and the other radically deteriorated—into a continuous spectrum that the viewer can control using body posture (specifically the nose’s Y‑position) and hand gestures (open palm vs. closed palm). Below is a detailed overview of the project’s concept, process, and technical implementation.

Concept & Inspiration

• Two Contrasting Facades

  The piece transitions between a sleek, modernist style and a “Bilbo‐looking” dystopian façade. For the modern facade, we drew inspiration from Walden 7, designed by the modernist architect Ricardo Bofill. This bold, geometric style contrasts starkly with the heavily decayed, fantastical aesthetic of the other façade.

• Human Posture & Gesture Inputs

  The primary input to control this transition is the viewer’s nose height (the Y position of the tip of the nose). By moving closer or farther, or by altering posture, participants can shift the façade toward either the modern or the deteriorated state. Additionally, we incorporated hand gestures:

  • Open Palm: Reduces the “strength” of the diffusion prompt, making the output more faithful to the original input images.
  • Closed Palm: Increases the prompt’s influence, pushing the rendered result closer to the fully synthesized façade style.

• Generative Sound & Volume

  The project also ties each façade style to its own musical layer. As viewers shift between open‐ and closed‐palm gestures, the audio transitions—altering volume levels based on the active façade stage.

Process & Implementation

1. Input Image Textures

   • We began by collecting a series of internet memes and blending them into a single, cohesive texture (a collage).
   • This meme collage is then mapped onto a physical model as a base input.
   • A separate “basic façade” texture is also prepared, representing the modern and deteriorated states.

   

   internet meme collection into a cohesive collection

2. Stream Diffusion Module

   • We use what we refer to as “stream diffusion” to synthesize new façade images.
   • Two prompts are simultaneously fed into the diffusion model—one capturing the modernist Walden 7 aesthetic, and the other capturing a deteriorated, dystopian façade.
   • Weighted Prompts: We dynamically adjust the weighting of each prompt in real time, based on posture and gesture data. This allows the final synthesized façade to morph fluidly between the two extremes.

   

   stream diffusion input and prompt weight adjustment

3. Gesture & Posture Control

   • Nose Tracking: A simple face‐tracking algorithm locates the tip of the nose and uses its Y position to set the interpolation value between the two façade states.
   • Open/Closed Palm: A hand‐tracking system detects gestures that modify how many diffusion steps are run for each frame. Closing the palm increases the prompt’s effect (pushing the façade style further), while opening the palm reduces it (preserving more of the original texture).

   

   gesture output from MediaPipe Module

4. Real‐Time Output

   • As the participant moves and changes hand gestures, the system immediately updates both the synthesized image and the soundtrack.
   • The final images appear on a screen (or projection), showcasing an evolving façade that shifts from crisp, modern geometry to fantastical, decaying structures.

   

Outcome & Significance

Spectral Facade demonstrates how generative AI can be used in real‐time, interactive installations. By mapping human gestures and posture to diffusion weights and iteration counts, it blurs the boundary between participant and artwork—allowing viewers to sculpt architectural aesthetics on the fly. This approach offers a glimpse into the future of generative media, where human presence and embodied interaction directly shape design outputs.

If you’d like to learn more or see Spectral Facade in action, feel free to get in touch or explore the additional documentation and videos below.

https://youtu.be/EiQlJGqY754