Projection mapping is one of the most profound visual effects that computers can generate; themepark fans will have seen it in effect on the revamped opening scene to the Indiana Jones ride at Disneyland and in the night-time shows that involve painting the whole castle with light (projection mapping is also used to generate the rear-projected faces of the animatronic figures in the new Snow White ride).
But those forms of projection mapping all depend on precise, advance measurement of the projection surface, which is used to pre-deform the video to correctly map overtop of the surface.
In a jaw-dropping demo from Ishikawa Watanabe Labratory, researchers show how they can use a 1,000 FPS projector to remap video in near-realtime (there’s 3ms of latency, well below the threshold for human perception), so that images can be correctly displayed on surfaces in motion, even the kinds of complex motion set up by fabric being flapped in the wind.
The projections are only 8 bits deep, but presumably that’s a temporary limitation that’ll be overcome with advances in projector technology.
Projection mapping attracts attentions as an emerging technology to extend the real world. However, almost realized examples have been limited to static or quasi-static environments. This research aims at overcoming this limitation and realizes dynamic projection mapping in which dynamically-changing real-world and virtual visual information are completely merged in the level of human visual perception. This high-speed dynamic projection mapping requires a high-speed projector enabling high-frame-rate and low-latency projection. In order to meet this demand, we have developed a high-speed projector “DynaFlash” that can project 8-bit images up to 1,000fps with 3ms delay.In particular, as a challenging target for the dynamic projection mapping, we focus on a non-rigid surface. Sensing of non-rigid surface deformation is difficult to be achieved at high speed because it has high degrees-of-freedom and involves self-occlusions as well as external occlusions. Our newly proposed method overcomes this limitation. Our method can obtain the deformation robustly at 1,000 fps by using an originally proposed marker “Deformable Dot Cluster Marker”, even when the target causes large deformation and occlusions.
Using these base technologies including DynaFlash and Deformable Dot Cluster Marker, we realize a new dynamic projection mapping onto deforming non-rigid surface. In this demonstration, by drawing the marker on the target with IR ink, we allow the marker to be invisible to human and enable robust sensing independently of the projected images. In our technology, both the projection and sensing are operated at a speed of 1,000 fps. Therefore, it is possible to keep the projection consistent with the deformation and extend the real world as if the projected image is printed or existed as an original (digital) texture on the target. Especially, focusing on new paradigms in the field of user interface and fashion, we have demonstrated dynamic projection mapping onto a deformed sheet of paper and T-shirt. Also we show that projection to multiple targets can be controlled flexibly by using multiple markers.
Dynamic projection mapping onto deforming non-rigid surface using a high-speed projector
[Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa/Ishikawa Watanabe Labratory]
(via Reddit)