Mixed Reality (MR) aims to seamlessly blend digital and physical worlds, but the absence of haptic feedback often disrupts the cohesion between visual and tactile experiences. We introduce Prop-Chromeleon, which dynamically repurposes everyday objects as haptic props for virtual assets using generative artificial intelligence (AI). Based on user prompts, Prop-Chromeleon generates virtual models and textures that are fit and overlaid on the physical prop shape. Prop-Chromeleon enables a single physical prop to act as haptic proxies for a wide range of virtual objects that through generative AI can change in real-time without the need for predefined proxies or virtual libraries. We demonstrate Prop-Chromeleon's effectiveness in adaptively transforming physical objects to haptic virtual objects via a technical evaluation of diverse shapes and prompts. We also showcase Prop-Chromeleon's ability to deliver believable passive haptic experiences in an MR user study. Prop-Chromeleon offers a scalable and adaptive approach to improving haptic MR experiences.

The lack of haptic feedback can disrupt the coherence between visual and tactile experiences, as seeing your hand pass through virtual objects can break the carefully crafted immersion in Mixed Reality. One existing approach to address this issue involves using physical objects as passive haptic props for similarly shaped virtual items. However, in real-world settings, it's not practical to find physical props that can perfectly match every virtual object, which limit the scalability of this method.
We aimed to create an MR passive haptic system that overcomes these challenges and adapts to various everyday objects across different user environments, creating on-demand haptic experiences.

To generate 3D content based on user prompts and the shape of real-world objects, we combined a text-to-image AI model with a 3D reconstruction model, along with AR authoring tools and a capture system to create a unique pipeline. The process begins by capturing a depth map of the scene, including the physical object the user wishes to transform. Using the user's text prompt, a text-to-image AI model generates a 2D image that represents the desired virtual asset. This image is then converted into a 3D mesh through a 3D reconstruction model. Simultaneously, the system creates a 3D model of the original physical object to serve as a tracking reference. Finally, using AR authoring tools, the generated virtual mesh is anchored onto the physical object in real-time with six degrees of freedom.

To validate Prop-Chromeleon's effectiveness, we conducted two studies assessing its performance and impact on the haptic experience. In the first study, we tested the system on a variety of shapes and user prompts that might be encountered in real life, demonstrating its ability to adapt to different scenarios. Also the measurements of geometric alignment confirmed high shape similarity between the generated models and the original objects.
In the second study, user tests revealed that Prop-Chromeleon significantly enhanced haptic feedback compared to a baseline without shape alignment. Participants found the system more "realistic," "convincing," and engaging, noting that shape alignment allows for more immersive interactions. For more information, please refer to the paper.