Mixed Reality (MR) aims to blend digital and physical worlds, but the absence of haptic feedback often breaks visual-tactile consistency. We introduce Prop-Chromeleon, a generative AI-based MR system that dynamically transforms everyday objects into adaptive passive haptic props through user-provided text prompts. Our pipeline performs real-time generation and spatial anchoring of virtual assets that align precisely with the shape of physical props, enabling on-demand tangible interactions in MR. We evaluate Prop-Chromeleon’s effectiveness through a generation study using varied object shapes and user prompts, combining quantitative shape similarity metrics with qualitative prompt fidelity analysis. Our user studies further showcase Prop-Chromeleon’s significant improvements in perceived realism, immersion, and enjoyment compared to static baselines. By seamlessly blending physical interaction with virtual transformation, Prop-Chromeleon delivers not only effective haptic feedback but also a playful and creative medium for interaction in MR.

The lack of haptic feedback can disrupt the coherence between visual and tactile experiences, as seeing your hand pass through virtual objects can break the carefully crafted immersion in Mixed Reality. One existing approach to address this issue involves using physical objects as passive haptic props for similarly shaped virtual items. However, in real-world settings, it's not practical to find physical props that can perfectly match every virtual object, which limit the scalability of this method.
We aimed to create an MR passive haptic system that overcomes these challenges and adapts to various everyday objects across different user environments, creating on-demand haptic experiences.

To generate 3D content based on user prompts and the shape of real-world objects, we combined a text-to-image AI model with a 3D reconstruction model, along with AR authoring tools and a capture system to create a unique pipeline. The process begins by capturing a depth map of the scene, including the physical object the user wishes to transform. Using the user's text prompt, a text-to-image AI model generates a 2D image that represents the desired virtual asset. This image is then converted into a 3D mesh through a 3D reconstruction model. Simultaneously, the system creates a 3D model of the original physical object to serve as a tracking reference. Finally, using AR authoring tools, the generated virtual mesh is anchored onto the physical object in real-time with six degrees of freedom.

Through technical evaluation and user studies, we demonstrated Prop-Chromeleon’s effectiveness in handling diverse physical geometries and prompt types, while achieving high fidelity in both shape alignment and semantic adherence. Compared to conventional baselines, Prop- Chromeleon significantly enhances realism, immersion, engagement, and overall user preference. Our findings also suggest that incorporating the geometry of physical objects into prompt design leads to more coherent and believable outcomes. Furthermore, our study reveals that Prop-Chromeleon has the potential not merely as a haptic tool, but as a creative platform that bridges generative AI with embodied Mixed Reality experiences. These results highlight the broader applicability of generative AI in enabling passive haptic experiences for MR.