The BMW Group Technology Office USA in Mountain View and Meta’s Reality Labs Research have demonstrated for the first time the ability to display stable virtual reality (VR) and mixed reality (MR) content accurately to passengers in a fast-moving car, even when making turns, going over speed bumps, and accelerating.

By integrating Meta Quest’s tracking system with BMW’s sensor data, the researchers enabled a series of stable “car-locked” gaming, entertainment, productivity, and meditation experiences.

It is too early to tell exactly how or when this technology will make it into customers’ hands, but we envision a number of potential use cases for XR devices in vehicles—from assisting the driver in locating their car in a crowded parking lot to alerting them to hazards on the road and surfacing important information about the vehicle’s condition. The implications of future AR glasses and VR devices—for passengers as well as drivers—are promising. The research partnership with Meta will allow us to discover what immersive, in-vehicle XR experiences could look like in the future and spearhead the seamless integration of such devices into cars.

—Claus Dorrer, Head of BMW Group Technology Office USA

VR headsets use both inertial motion sensors (IMUs) and cameras to estimate precisely the headset’s location and motion. However, in a moving environment—i.e., in a non-inertial reference frame—these two modalities are in conflict as the cameras observe motion relative to the inside of the car, while the IMUs measure acceleration and rotational velocity relative to the world. Simply put, the headset’s motion sensors think it is moving, but its cameras think it is standing still when looking at the car’s interior.

That mismatch means today’s VR headsets can’t display stable virtual content when traveling inside a fast-moving vehicle, particularly if the vehicle accelerates, turns fast, or encounters bumps on the road.

To solve this challenge, BMW and Meta researchers collaborated to incorporate information from a BMW car’s sensor array into the tracking system of a Meta Quest Pro. That allowed the tracking system to calculate the device’s location relative to the car and the moving car’s location relative to the world simultaneously, enabling it to anchor virtual objects accurately to a moving car and demo compelling VR and MR passenger experiences.

Combining the movement data generated by the vehicle with XR headset sensor capabilities allowed subtracting the vehicle motion and more accurately tracking the headset’s movement in a vehicle. That in turn enabled the researchers to render content which is synced to the movement of the vehicle in real-time, both inside and outside of the vehicle at high speeds, regardless of whether the vehicle is on a flat, straight street or a bumpy road with sharp turns.

This proof-of-concept prototype demonstrates an in-car full range of a person’s movements (known as six degrees of freedom, 6DOF) and an XR experience that simultaneously calculates a headset’s location relative to the car and the moving car’s location relative to the world by integrating with the car’s sensors.