https://www.youtube.com/watch?v=OAg5alXv8xY

Introduction

This project builds on the earlier work titled "Real-Time Coding Adventure of the Bio-Cybernetic System." The original project revolved around creating a "digital creature" that interacted with user input and environmental changes. Inspired by Physarum Polycephalum, it used mouse and webcam data to track light and motion, generating sequences that shifted the interface’s behavior. By occasionally obstructing the display and then gradually returning to normal after a period of stillness, the system encouraged users to rethink their relationship with the screen. It highlighted how feedback loops can disrupt the usual one-way interaction we have with interfaces, pushing for a more reflective and dynamic experience. (Figure 1)

Figure 1: Real-Time Coding Adventure of the Bio-Cybernetic System Project Slide

The project was grounded in ideas like performative idiom and circular causality. Performative idiom challenges the notion of fixed, pre-defined systems, focusing instead on processes that adapt and evolve in unpredictable ways, much like Ashby’s Homeostat or Pask’s Colloquy of Mobiles. Circular causality, on the other hand, explores how feedback loops create systems that can self-regulate and respond to their environment. Together, these concepts shaped the project’s goal of questioning sedentary, mouse-driven interactions and reimagining them as active, multi-directional exchanges. That said, the original system was limited to what could happen within the boundaries of the computer screen.

The new project takes this further by using Stable Diffusion and Projection Mapping to create a dynamic ambient environment that responds to user interactions in real time. It tracks subtle inputs like mouse location, movement speed, typing rhythm, and posture, and projects them into the physical space around the user. This creates a feedback loop that extends beyond the screen, allowing not only the user but also others nearby to engage with and influence the system.

Some key changes in the new system include:

• Going beyond the screen: The projection mapping takes everyday interactions, like mouse and keyboard movements, and translates them into ambient visuals that fill the surrounding environment.
• Using generative AI: By integrating StreamDiffusion, the system combines both measurable data (like movement speeds and posture) and qualitative inputs (like text prompts or screen captures) to generate visuals that feel imaginative and alive.
• Interactive projections: The projected environment becomes part of the interaction, giving users something new to react to and creating a back-and-forth exchange between the digital and physical spaces.

This project pushes the boundaries of human-computer interaction by moving the focus away from the computer itself and into the surrounding space. It highlights how small, often overlooked actions like typing and mouse movements can shape the environment, inviting users to not only see their interactions differently but also share the experience with others. By breaking out of the screen’s confines, it transforms a closed-loop system into something more open, interactive, and thought-provoking.

Equipment and Computational Pipeline Setup

The workflow begins with the user deciding to interact with the computer, and this interaction is captured through inputs such as mouse and keyboard usage, as well as the user’s physical movements. These interactions generate a range of system inputs, including cursor XY coordinates, cursor movement speed, keyboard typing speed, live PC screen captures, and human contour segmentation. Each type of input is processed and integrated into a computational pipeline, culminating in real-time generative visuals projected back into the user’s environment. (Figure 2, 3)

Figure 2: Overall System Loop Flow Chart (red lines are the human input, grey lines are algorithmic process, green lines are external projections)

Figure 3: Stream Diffusion Script Combining Stream Diffusion (real-time image generation) and MediaPipe (posture recognition)