The new AR platform from Disney Research, Magic Bench, will be presented and demonstrated at SIGGRAPH 2017 in Los Angeles.
Magic Bench combines AR and MR, allowing people to see, hear, and even feel characters on and around the bench. It puts the focus on the surroundings being instrumented, rather than the individual user. This allows a whole group of people to share the experience, rather than users being locked in their individual display helmet or holding out a handheld device.
When people sit on the bench they see themselves in a mirrored image on a large screen – this creates a third person point of view. The scene uses a depth sensor so, instead of superimposing one video feed onto another, participants occupy the same 3D space as a computer-generated character or object.
“This platform creates a multi-sensory immersive experience in which a group can interact directly with an animated character,” says Moshe Mahler, principal digital artist at Disney Research. “Our mantra for this project was: hear a character coming, see them enter the space, and feel them sit next to you.”
Researchers employed a colour camera and depth sensor, creating a real-time, HD-video-textured 3D reconstruction of the bench, its surroundings, and the participants. The algorithm is able to reconstruct the scene – it aligns information from the RGB camera with that from the depth sensor.
Potential snags have been overcome. Where depth shadows occur in places where the depth sensor has no corresponding line of sight with the colour camera, a modified algorithm steps in to create a 2D backdrop. The 3D and 2D reconstructions are placed in virtual space and then populated with 3D characters and effects. The real-time rendering is a seamless composite, totally up to the challenge of interacting with virtual physics, light, and shadows.
Mahler explains that the bench itself, which contains haptic actuators, plays a crucial role. “It constrains several issues for us in an elegant way,” he says. “We know the location and the number of participants, and can infer their gaze. It creates a stage with a foreground and a background, with the seated participants in the middle ground.” He says it even serves as a controller. “The mixed reality experience doesn’t begin until someone sits down and different formations of people seated create different types of experiences.”
A large team of Disney Research scientists evolved The Magic Bench. Alongside Moshe Mahler, Kyna McIntosh, John Mars, James Krahe, Jim McCann, Alexander Rivera, Jake Marsico, Ali Israr and Shawn Lawson were also involved in the project.
The research team will present and demonstrate the Magic Bench at SIGGRAPH 2017, the Computer Graphics and Interactive Techniques Conference, in Los Angeles. The conference begins on July 30th.
