VRSUN

VRSun

Mixed Reality

Facebook Details Experimental Mixed Reality & Passthrough API

From uploadvr.com

Facebook shared some details about its experimental Passthrough API to enable new kinds of mixed reality apps for Oculus Quest 2.

The feature may also serve as the foundation for the company’s long-term efforts in augmented reality, effectively turning Quest 2 into a $299 AR developer kit. When asked if the feature is coming to the original Oculus Quest, a Facebook representative replied “today, this is only available for Quest 2.”

The new feature will be available to Unity developers in an upcoming software development kit release “with support for other development platforms coming in the future.”

Facebook says apps using the API “cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors” and raw images form the four on-board cameras “are processed on-device.”

The following capabilities will be available with the passthrough API, according to Facebook:

Composition: You can composite Passthrough layers with other VR layers via existing blending techniques like hole punching and alpha blending.

Styling: You’ll be able to apply styles and tint to layers from a predefined list, including applying a color overlay to the feed, rendering edges, customizing opacity, and posterizing.

Custom Geometry: You can render Passthrough images to a custom mesh instead of relying on the default style mesh—for example, to project Passthrough on a planar surface.

Facebook sees the Passthrough API as enabling more robust examples of productivity apps — remote work solution Spatial was one of the first apps to access the API — alongside the ability to interact with people or pets in your physical room at the same time as VR content, and it could open up new kinds of games that could incorporate more of your physical environment.

GIGXR to Develop XR Training Platform HoloChem for US Air Force Cadets

From vrfocus.com

Having previously developed mixed reality (MR) training solutions such as HoloPatient for medical and nursing schools, GIGXR has announced a new contract to develop a chemistry training simulation for US Air Force Academy (USAFA) cadets. Called HoloChem it’ll be an MR application for USAFA 100 and 200 level chemistry courses.

GIGXR - HoloPatient
HoloPatient. Image credit: GIGXR

GIGXR received a $750,000 USD Phase II Small Business Innovative Research (SBIR) contract from AFWERX to develop HoloChem which will be used by cadets in conjunction with Microsoft HoloLens 2, aiding them in their chemistry studies.

The platform will utilise GIGXR’s mixed reality expertise from its other applications, providing lifelike learning experiments which take traditional chemistry curricula and makes them more engaging. Like most XR teaching applications, HoloChem will utilise gamification techniques to encourage concept retention and critical thinking skills. 

“Traditional chemistry labs lack the immersive, safe-to-fail environments students need to develop the critical thinking skills for using chemistry in the real world,” said Captain Wale Lawal, Ph.D., Assistant Professor at the Department of Chemistry for the U.S. Air Force Academy in a statement. “GIGXR’s mixed reality learning solutions allows USAFA educators to immerse students in ultra-realistic, interactive scenarios to transform their understanding of chemistry from rote memorization or rigid ‘cookbook chemistry’ to real-time problem-solving. This kind of learning mimics the challenges they might face in the future, which would otherwise be impossible to simulate or place our students at risk.”

GIGXR
HoloPatient. Image credit: GIGXR

“Being awarded with the Phase II SBIR Contract is an achievement we’re incredibly proud of and marks a milestone in mixed reality’s valuable impact on education,” said David King Lassman, CEO, GIGXR. “We’re honoured to be working with the U.S. Air Force Academy, whose reputation for academic excellence and technological innovation is second to none. USAFA needed a product that enabled their instructors to provide hands-on, immersive learning and future-proof execution. HoloChem simulates key real-life experiences that labs simply cannot and allows instructors to reach multiple campuses or remote students to match the ultra-realistic learning experience available to the cadets in the instructor’s own classroom.”

HoloChem is slated to be deployed in early 2022 with instructors using HoloLens 2 headsets to share content with students. They’ll also have access to the MR headsets or can connect via Android or iOS smartphones and tablets. For further updates on GIGXR’s latest mixed reality projects, keep reading VRFocus. 

‘XR Casino’ Brings Cross-Platform Gambling To VR & AR

From vrscout.com

Bet on games of Blackjack and Roulette in this blockchain-powered virtual casino coming soon to major VR & AR devices.

XR Casino, Inc (“The Company”) has announced an official release date for its cross-platform gambling experience which allows users on VR and AR devices to bet real money on games like blackjack, roulette, and slots from the comfort of their own holographic casinos.

At launch, XR Casino will feature multiplayer games, public and private rooms, and text and voice chat. In terms of compatibility, the platform is cross-compatible with nearly every major VR & AR device, including the HTC Vive, Oculus Quest 2, Microsoft HoloLens 2, as well as AR-enabled iOS and Android devices; it’s never been easier to take your friends’ money! The platform itself is built on a decentralized blockchain network, offering support for over 100 payment methods, from standard credit and debit cards to popular cryptocurrencies like Bitcoin.

Image Credit: XR Casino

“XR technologies are changing the way we interact with each other and eliminating the need for expensive equipment,” said XR Casino CEO Dan Martinez in an official release. “Tech giants such as Apple, Google, Facebook and Microsoft have invested heavily into AR, VR and MR to speed up mass adoption through both hardware and software solutions.”

“XR Casino has first mover’s advantage in providing XR SaaS solutions to online, traditional casinos and sports betting companies. Our solutions create value for the gambling industry and offer competitive differentiation to attract new Generation X, Y and Z players worldwide.”

Image Credit: XR Casino

Moving forward, the company plans on introducing additional casino games and sports betting to the platform, including baccarat, bingo, craps, keno, and poker. XR Casino will be available on the following devices when the platform launches on August 21st, 2021:

  • HTC Vive & Vive Pro
  • Magic Leap One Creator Edition
  • Microsoft HoloLens 1 & 2
  • Oculus Quest & Quest 2
  • Samsung Gear VR
  • Valve Index
  • Windows Mixed Reality
  • Apple iPhone & iPad
  • 39 Android devices

There’s currently no shortage of immersive gambling experiences available on VR headsets at the moment. That being said, XR Casino offers users the rare opportunity to gamble using real-world currency.

Not only that, but it appears as though the platform is compatible with a wide range of payment methods, making it incredibly easy to transfer funds and jump right back into the action; perhaps a little too easy…

As mentioned previously, XR Casino launches on VR & AR devices on August 21st. For more information visit here.

Feature Image Credit: XR Casino

‘Virtual Taekwondo’ Could Be The Next Big Olympic-Level Sport

From vrscout.com

Virtual Taekwondo combines full-body motion-tracking and video games to offer the next generation of competitive virtual sports.

The 2020 Summer games are finally happening after being delayed for over a year due to the COVID-19 pandemic, which means the long-awaited return of Olympic-level taekwondo.

In honor of International Olympic Day, a celebration of Olympic values in which athletes participate in a variety of sports activities, exhibitions, and seminars, Singapore-based tech company Refract has revealed Virtual Taekwondo, a mixed reality experience that allows fighters to spar virtually using full-body motion-tracking powered by the companies Active XR Interface System (AXIS). Think of it like playing a video game such as Tekken, Mortal Kombat, or Street Fighter, only instead of buttons and joysticks, your body is the controller.

Image Credit: Refract

Virtual Taekwondo uses 10 sensors (or 7 if you’re using a VR headset) spread throughout various parts of your body in order to capture your full body’s movements and then replicate them in-game in real-time. The movements of you and your opponent are displayed on a giant screen in front of each player; it looks a lot like playing a traditional video game, only slightly more exhausting.

This system doesn’t require cables or any type of base station, which means you have full freedom of movement, allowing you to bring your own unique style of fighting to the game. Because your body is the controller, Virtual Taekwondo requires a considerable amount of physical energy, not to mention skill.

Sorry button mashers, luck isn’t going to help you here.

Image Credit: Refract

With the esports market on track to surpass $1.5B by 2023, Refract sees big potential with its AXIS platform. By partnering with the World Taekwondo Federation, the two organizations hope to develop a virtual sports program that can be used to promote physical activity through innovation such as XR. Another goal is to align themselves with the IOC’s Olympic Agenda 2020+5 vision (a roadmap of 15 recommendations that will guide the IOC and Olympic Movement for the next five years) in growing digital engagement with people all around the world. 

In an official press release, President of the World Taekwondo Federation, Dr. Chungwon Choue said, “In today’s society, it has never been more important to transcend physical barriers,” adding, “With Virtual Taekwondo, we promote fitness, wellbeing, and community to a wider, digitally-savvy audience.” 

Image Credit: Refract

Ultimately, Choue hopes that virtual sports will be a medal event at the 2028 Olympic Games in Los Angeles. 

Because AXIS is a digital platform, athletes across several disciplines will no longer be restricted by traditional sports barriers such as location, age, gender, and weight class; instead of having women and men’s divisions—or dividing athletes by age group—virtual sports will pit skill vs skill, regardless of physical form or characteristics.

“We are excited to support World Taekwondo and have our system tested at the highest levels by professional athletes,” said Michael Chng, CEO of Refract Technologies. “With advances in 5G, artificial intelligence, and edge cloud processing, usage of extended reality devices will become a way of life in the near future.” 

Image Credit: Refract

Refract will be showcasing its AXIS technology and Virtual Taekwondo experience for International Olympic Day in collaboration with the World Taekwondo Federation.

For more information, visit the Refract website at www.refract.gg.

Feature Image Credit: Refract

REPORT: Apple Mixed Reality Glasses Launching Mid-2022

From vrscout.com

Tech analyst Ming-Chi Kuo claims that the upcoming headset will launch in Q2 of next year and will feature 15 onboard cameras.

This past April, Apple CEO Tim Cook stated that augmented reality (AR) technology is “critically important” for the future of the company. During last week’s WWDC, the company flexed its AR muscles even further with the reveal of AR navigation for Apple Maps, an easy-to-use 3D modeling tool called RealityKit Object Capture, and ARKit 5 with improved face tracking and motion capture.

It’s obvious that Apple is working hard to build an AR ecosystem of amazing tools powered by products such as the iPhone or iPad.



That being said, there have also been a lot of rumors floating around about the company launching its own set of mixed reality glasses. At one point, rumors began circulating about a potential mid-2021 release. However, according to the latest report from analyst Ming-Chi Kuo, who has over 10 years of experience reporting on Apple, information recently gathered from supply chains now indicates a possible Q2 2022 release for the highly-anticipated mixed reality hardware.

This date makes sense for a couple of reasons. For one, Apple’s latest AR tools are just getting into the hands of developers. Delaying its mixed reality glasses could give creators enough time to familiarize themselves with the new platform and begin crafting AR experiences that will be ready at launch. This timeline also makes sense when you take into account supplier availability. In a new report by MacRumors, the publication points out how Genius Electronic Optical, a key lens supplier for numerous companies, will benefit from the launch of upcoming AR and VR projects from Facebook and Sony.

“We predict that Apple will launch AR HMD [head-mounted display] devices in Q2 2022. The device will provide a video see-through AR experience, so the lens is also needed, and Genius is also a key supplier,” the report says.

Concept art of Apple MR Glasses / Image Credit: The Information

In addition to mixed reality functionality, it’s also possible that the headset will offer hand and eye-tracking thanks to a number of cameras located both inside and outside the glasses. According to past reports, the number of onboard cameras could be as high as 15.

According to Kuo, “As well as collecting data from the outside world, Apple is expected to precisely track the user’s eye movements. This will allow them to increase apparent rendering fidelity by targeting only what the user is currently looking at, rather than the entire human field of view. This technique is called foveated rendering.”

In Kuo’s previous report, he stated that the mixed reality headset would cost approximately $1,000. That detail still hasn’t changed. As for the hardware’s primary target demographic, it’s still too early to tell. It’s entirely possible that Apple’s AR glasses will be marketed as a high-end device designed specifically for enterprise use, and not made for the general consumer. 

Feature Image Credit: Future

Facebook’s VR Spectator Tools Could Evolve Into Mixed Reality Casting

From uploadvr.com

Facebook continues to evolve its spectator tools for virtual reality, but creating mixed reality footage remains an arduous process from the standalone Quest while the company continues to actively explore the approach.

The company recently added a “Live Overlay” feature in the most recent v29 software update for Quest which merges the first-person view in VR with a camera view of the person wearing the headset. The feature is starting to roll out to a limited number of users in the Oculus app “and it will work with any VR app that supports Casting and Recording,” Facebook explained over email.

In addition, a page in the Oculus Quest developer documentation explains “Spectator Camera is extending Mixed Reality Capture (MRC) to the Oculus app casting.” We contacted Facebook to ask for clarification about this feature because the language in this description isn’t clear. The “Spectator Camera” will allow casting the view from VR to the Oculus app while enabling control of the view from the phone using touch controls, Facebook explained. It was announced at Facebook’s last VR developer event but doesn’t include a view of the VR user wearing the headset.

“There aren’t any apps live with this today, but we are actively working with developers to integrate, so that’s why these developer tools are available now,” Facebook explained.

And while the “Live Overlay” feature does feature an external view of the person in VR, it doesn’t include a third-person angle or the careful alignment that’s key to “Mixed Reality Capture.”

Mixed Reality Capture is driven largely by community projects making it possible to merge simulated content streaming from a VR app with an actual camera’s view of the scene. The merging of two realities becomes a single seamless view that’s instantly understandable to spectators. In 2016, Valve essentially popularized the idea with its initial promotional video for SteamVR. Since then, community projects like LIV and Fabio Dela Antonio’s Reality Mixer app enable developers and broadcasters to more easily share what it feels like to play VR games without post-production work. Some of these projects, however, require an arduous alignment process to sync the view of two realities, or requires the addition of extra recording software like OBS.

Dela Antonio maintains an open source repository for his work on the Reality Mixer app and Facebook CEO Mark Zuckerberg this week posted a mixed reality video of himself playing Beat Saber using what appears to be the tool. We reached out to Dela Antonio for comment on the Zuckerberg video and to see what Facebook’s interest in mixed reality broadcasting means for his work. He wrote to us over direct message that he thinks Zuckerberg’s video and the developer page are evidence the company will eventually offer a solution similar to his app.

“I generally see this as a positive thing, it means that these videos will become even more accessible, and that they’ll start making some improvements to the existing Oculus MRC mechanism,” he wrote. “As for my app, its future compatibility with the Quest will depend on the changes they make to the MRC mechanism. I’m still working on the app as normal, I’ve recently added support for Avatars using ARKit’s body tracking and I’m also working on supporting physical green screens…there are some other things I could do to improve the calibration process and to allow the user to move the camera around the scene…but that’d require changes to how the Oculus MRC protocol works, it’d be interesting to know if Oculus has similar plans/ideas…One reason I was able to start working on my app last year was because the Oculus MRC plugin for OBS was open source, and that’s also why I’ve kept my app open source. It’d be nice if they could open source their new mobile Mixed Reality solution…”

Facebook offers the Portal line of video calling devices which include a wide angle camera and a key feature of the hardware zooms in on people in the scene. But years after release, there’s still no integration with Oculus Quest for Casting or any other functionality. Recent tweets from a Japanese group shows how full body tracking might be possible with an external camera and Oculus founder Palmer Luckey recently commented that Portal should be used for the same purpose.

“Overall, we are exploring a variety of ways to help people better capture, represent and share their VR experiences. We believe these kinds of tools help to make VR more fun, and are also good for the ecosystem since they enable people to share the experiences they’re having in VR,” Facebook shared in an email. “Moving forward, you will see us continue to experiment with different features and capabilities in the Oculus App that provide new ways of casting, recording and sharing the VR experience with various different views of reality, virtual reality, camera perspectives, etc. There are many ways that it’s possible to represent and share VR, and every VR app is different – That’s why we’re testing different approaches to learn from our community about what modes of capture best represent them and the VR apps they use. In the future, features like these could possibly evolve into a mobile ‘mixed reality’ casting/recording experience like you describe. We don’t have specific plans or timing to announce for that, but it’s something we are exploring – in fact, we showed a proof of concept for this at OC6, so it’s something we’ve been continuing to explore over time.”

Microsoft Files Patent To Bring Real-World Objects To VR

From vrscout.com

The proposed system would bring physical environment awareness to VR by creating virtual representations of real-world objects.

According to a patent filed to the United States Patent & Trademark Office, Microsoft has been working on a computer system that automatically converts physical objects into digital models viewable in VR.

The patent, entitled “Transitioning into a VR environment and warning HMD users of real-world physical obstacles,” details a system capable of generating virtual representations of real-world objects that come within a certain range of the head-mounted display (HMD). According to the documentation, a proximity detection system would be used to identify physical objects before warning the user in VR by creating a relevant digital model layered over the aforementioned object.

Image Credit: United States Patent & Trademark Office

“Optimizations are provided for facilitating an improved transition between a real-world environment and a virtual reality environment,” states the patent. “Initially, use of a HMD is detected and one or more real-world physical objects within a threshold proximity to the HMD are identified. Subsequently, a replicated environment, which includes virtual representations of the real-world physical object(s), is obtained and rendered in a virtual reality display. The replicated environment is transitioned out of view and a VR environment is subsequently rendered in the virtual reality display. In some instances, rendering of virtual representations of real-world physical objects into the VR environment occurs is response to detected triggering event.”

In theory, this could be a game-changer for VR users dealing with limited free space. Whether it be a small apartment or a cluttered bedroom, any location could become a virtual playing field made unique by its surrounding objects.

Image Credit: Microsoft

Facebook has already begun implementing its own form of physical object detection on Oculus Quest headsets. Recent updates have included new features such as desk and couch detection, allowing users to mark the real-world locations of said objects and generate virtual representations which can be viewed while in VR.

The system described in Microsoft’s patent sounds as though it could take the technology even further with an automatic object-detection system that improves not only the players’ safety but their level of immersion as well.

For more information check out the patent here.

Feature Image Credit: Microsoft

Oculus Quest v29: Mixed Reality, iOS Notifications, And More

From vrscout.com

Facebook’s latest update introduces several long-awaited features designed to improve how we work and play on the Oculus Quest.

While many of us are still wrapping our heads around the many exciting updates introduced as part of the v28 update, Facebook today unveiled yet another batch of game-changing improvements on their way to Oculus Quest headsets, including live overlay capture, in-headset notifications, and much more.

Rolling out soon on the Quest platform, update v29 features plenty to look forward to, whether you’re a hardcore gamer looking for an easier way to share your in-headset experiences with others on the web, or a business professional searching for new methods to improve your work productivity in VR. Let’s talk about the most exciting announcements first.

Image Credit: Facebook, Oculus

Live Overlay

Available soon on via the Oculus app, this unique casting feature lets you ditch the boring first-person gameplay in favor of immersive mixed reality by layering yourself over VR gameplay in real-time; no additional camera equipment or green screen required. Compared to standard mixed reality capture, the process described by Oculus sounds considerably more simple.

All you have to do is access the Live Overlay tool via the Oculus app and aim your smartphone’s camera at yourself to record or cast immersive mixed reality gameplay. You can place your smartphone on a flat surface and capture gameplay using the selfie camera or hand the device off to a friend to create dynamic, moving shots. Live Overlay currently supports iPhone models XS and up; no word yet on whether or not the feature will be available on Android devices.

Image Credit: Facebook, Oculus

Oculus is also introducing the ability to capture your voice using the Quest’s built-in microphone, whether you’re recording/casting standard first-person gameplay or Live Overlay content.

iOS Notifications

Another long-awaited feature among Quest users, v29 will introduce support for in-headset smartphone notifications, beginning with the iOS platform. Those with an iPhone 7 or newer will soon have the ability to enable iOS notifications, including text messages and updates from select mobile apps, via the Oculus Quest app as part of the headsets growing Infinite Office platform. These notifications will be visible only to you, regardless of whether or not you have multi-user support enabled.

According to Facebook, Android support will be arriving in the near future.

Image Credit: Facebook, Oculus

The New Files App

In addition to text and app notifications, v29 will also see the introduction of Files, a new app designed to assist you in accessing, organizing, and sharing various media files located on your headset, such as video captures and screenshots. Facebook also states that the app will make it easier to upload and distribute content to various websites straight from your headset via the Oculus Browser.

While this may not sound like the most thrilling update compared to today’s other announcements, the current method for sharing media files on the Oculus Quest can be somewhat of a headache at times, especially if you’re new to the Quest hardware. Hopefully, the Files app will make uploading and sharing content a more streamlined process for a new generation of VR content creators inspired by the Live Overlay feature.

Image Credit: Facebook, Oculus

Passthrough Shortcut

Another potentially game-changing addition to the Infinite Office platform, the new passthrough environment is designed to give you a quick view of your surroundings while working in VR. By selecting the “Passthrough Home” icon located in your Quick Settings menu, you’ll have the ability to instantly switch between your VR office and real-world surroundings, whether it be to check some important paperwork or simply take a quick sip of your coffee.

Ads on the Oculus App

Facebook capped off today’s update with the reveal of dedicated ads for select VR content. The company will soon begin testing a new ad system designed to provide further exposure for sponsored games and apps by suggesting relevant content via the Oculus app. Naturally, the company realizes that privacy is a major concern for many users, which is why it’s offering several options for managing your ad experience.

For more information regarding ad management, visit here.

Image Credit: Facebook, Oculus

In addition to the above-mentioned features, Facebook will also be launching multi-user support for the original Oculus Quest, allowing multiple users to save their in-game settings and progress using a single headset.

For more information on today’s announcements, check out the official blog post.

Feature Image Credit: Facebook, Oculus

NuEyes’ AR Glasses Are Made For Hardcore PC Gamers

From vrscout.com

Work and game on a 148″ augmented reality screen using the companies Pro 3e 5G smart glasses.

AR company NuEyes is taking its passion for AR and gaming to the next level with the launch of the Pro 3e 5G-enabled smart glasses through their gaming division, NuGaming. Unlike existing enterprise-focused headsets like the Microsoft HoloLens 2, the Pro 3e smart glasses are designed for use by consumers as a PC gaming and entertainment device.

Designed to look like a standard pair of sunglasses, the Pro 3e uses AR to project a 148” viewing screen in full 1080p HD at a smooth 60 FPS. There’s also a deep color contrast of 100,000 to 1, as well booming stereo speakers built into the frames. Similar to existing VR headsets like the Oculus Quest, the Pro 3e allows you to adjust the lenses to fit your personal IPD measurements.

Image Credit: NuEyes

I went hands-on with the device recently and the first thing I noticed was the lightweight and comfortable design. They didn’t feel bulky or uneven. Honestly, it felt like I was wearing a pair of conventional glasses. Thanks to the foldable arms, you can toss them in your backpack or tuck them descretely into your shirt pocket, making it easy to carry them with you throughout your day.

Adjusting the settings on the AR glasses is simple. There are volume and brightness controls built into both arms of the frames as well as two diopter adjustment knobs located at the top of the glasses that you can access by pulling back the covers.

NuGaming’s Pro 3e uses a USB-C plug so you can connect your glasses to the Samsung S20 and other mobile devices. You can also connect the device to your PC and project your computer screen directly to your glasses. NuEyes’s Pro 3e even works with the MacBook Pro and iPad Pro. There are other devices that you could plug your glasses into, but you might need a power boost adapter depending on the device, which will be available on the NuGaming site soon.

Image Credit: NuEyes

In terms of gaming, the Pro 3e can sync to almost any type of gaming controller, including Xbox, Google Stadia, and PlayStation. There’s even support for the Finch Ring as well as various Bluetooth keyboard and mouse solutions.

In an official press release, Mark Greget, Founder and CEO of NuEyes, said, “There are so many great cloud-based streaming services now, and to be able to tether my controller to my phone or tablet and stream games in 5G while enjoying the content on a crisp 148” viewing screen in full 1080p HD vs a smaller 8” or 10” screen or a cumbersome VR device is the next evolution in technology for this space.”

Crystal clear visuals and gaming support aside, the Pro 3e does feature its fair share of technical limitations. Unlike other popular AR headsets, for instance, the Pro 3e does not allow you to pin interactive 3D content around your real-world space due to its lack of external cameras. This lack of inside-out tracking means the generous 1080p HD screen remains in front of you at all times, regardless of whether or not you’re moving around.

Image Credit: NuEyes

In addition to gaming, the Pro 3e features plenty of light enterprise appeal. For example, the ability to quickly and privately access work-related content could make the device a valuable tool for business professionals on the move; while it may look like you’re soaking up the sun in pair of cool-looking shades, you’re actually balancing the company books and working on a proposal in MS Office. And whenever you need a break, the glasses can be used to access HBO Max, Netflix, Hulu, and other popular streaming platforms. I have to say, watching Star Wars: The Bad Batch on a 148″ screen is spectacular!

What NuGaming delivers with its 5G AR smart glasses is a streamlined immersive experience designed to meet the needs of both gamers and business professionals.

If you want to learn more about the NuGaming Pro 3e AR smart glasses, check out their website or follow them on Twitter.

Feature Image Credit: NuEyes