Reverse passthrough technology lets you to see the eyes of those wearing a headset, opening up new doors for “social co-presence”.
Lead by research scientist Nathan Matsuda, a team at Facebook has spent the last two years developing a new form of “social co-presence” for VR designed to remove the disconnect between VR users and those in the physical world using a 3D monitor and eye-tracking technology.
Whereas the company has been largely focused on Passthrough+ and its experimental Passthrough API—a feature that allows users to view the physical world while in-headset using several onboard cameras—reverse passthrough VR, as Facebook calls it, allows those outside the headset to view the upper facial expressions of those in VR without the need of an external display.
A microlens-array light field display mounted to the front of the headset allows those out of headset to view a 3D rendered model of the users’ eyes brought to life in real-time thanks to built-in eye-tracking technology.
According to an official report by Facebook Reality Labs Research, the project began with a standard 2D screen displaying a flat projection of the user’s face. Unfortunately, this resulted in a—as Facebook put it—“visually jarring, unnatural effect”.
“My first reaction was that it was kind of a goofy idea, a novelty at best,” said FRL Chief Scientist Michael Abrash after trying the first version of Matsuda’s device back in 2019. “But I don’t tell researchers what to do, because you don’t get innovation without freedom to try new things, and that’s a good thing, because now it’s clearly a unique idea with genuine promise.”
Since then the project has expanded to include various optics, software, and supporties technologies. 3D display technology offers a more realistic experience by displaying the users eyes at their correct position in space in relation to the position of the external viewer, so long as their viewing angle isn’t too severe.
Of course it goes without saying the project is still in its infancy and the visuals are far from lifelike. The system also requires a heft amount of computer power to run and is limited physically by several tethered cables. As such, don’t expect this technology on consumer headsets any time soon. Still, it’s an incredibly cool look at the future of social co-presence, one I can’t wait to see perfected.
For more information check out the official announcement here.
Feature Image Credit: Facebook