Prop-Chromeleon:
Adaptive Haptic Props in Mixed Reality through Generative Artificial Intelligence

Mixed Reality (MR) aims to blend digital and physical worlds, but the absence of haptic feedback often breaks visual-tactile consistency. We introduce Prop-Chromeleon, a generative AI-based MR system that dynamically transforms everyday objects into adaptive passive haptic props through user-provided text prompts. Our pipeline generates virtual assets and spatially anchors them to physical props, enabling on-demand tangible interactions in MR. Generation takes approximately 20 seconds in the reported setup; it is not instantaneous. We evaluate Prop-Chromeleon’s effectiveness through a generation study using varied object shapes and user prompts, combining quantitative shape similarity metrics with qualitative prompt fidelity analysis. Our user studies further showcase Prop-Chromeleon’s significant improvements in perceived realism, immersion, and enjoyment compared to static baselines. By seamlessly blending physical interaction with virtual transformation, Prop-Chromeleon delivers not only effective haptic feedback but also a playful and creative medium for interaction in MR.

The lack of haptic feedback can disrupt the coherence between visual and tactile experiences, as seeing your hand pass through virtual objects can break the carefully crafted immersion in Mixed Reality. One existing approach to address this issue involves using physical objects as passive haptic props for similarly shaped virtual items. However, in real-world settings, it's not practical to find physical props that can perfectly match every virtual object, which limit the scalability of this method.
We aimed to create an MR passive haptic system that overcomes these challenges and adapts to various everyday objects across different user environments, creating on-demand haptic experiences.

To generate 3D content based on user prompts and the shape of real-world objects, we combined a text-to-image AI model with a 3D reconstruction model, along with AR authoring tools and a capture system to create a unique pipeline. The process begins by capturing a depth map of the scene, including the physical object the user wishes to transform. Using the user's text prompt, a text-to-image AI model generates a 2D image that represents the desired virtual asset. This image is then converted into a 3D mesh through a 3D reconstruction model. Simultaneously, the system creates a 3D model of the original physical object to serve as a tracking reference. Finally, using AR authoring tools, the generated virtual mesh is anchored onto the physical object in real-time with six degrees of freedom.

In the user study, 11 of 12 participants preferred Prop-Chromeleon to the shape-unaware baseline. Questionnaire results favored the system for realism, harmony, and involvement, but not expressivity. Texture mismatches and occlusion problems remained limitations.