Skip to main content

Concept of Relocalization

In order to create a shared AR experience, we need to apply three main concepts: tracking, mapping, and relocalization.

Tracking and Mapping​

When a device (like a mobile phone) is placed in an unknown environment with the purpose of overlaying 3-dimensional content, there are two main questions:

  • Where am I?

  • What is in my surrounding?

To answer these two questions, we need to find the relative position between the device and items in the environment. We perform this by simultaneously tracking the device in the environment, and mapping this environment.

  • Tracking: By analyzing the difference between each camera frame, we can understand the motion of the device, and build a trajectory

  • Mapping: By finding interesting (feature) points in the camera frames, and triangulating their position in 3D, we can recreate a spatial structure of the environment

Relocalization​

The tracking and mapping processes are sufficient to display arbitrary 3D content in an AR application. However, if we were to save the 3D position of the virtual content, restart the AR application and attempts to display the content at the same position, all the objects would appear to have moved. Why?

This situation happens because restarting the AR application changes the coordinate system. Indeed, during localization and mapping, a coordinate system is created with an arbitrary origin, which is different for each session. So how do we create a consistent coordinate system so that 3D objects can persist at the same position?

Assuming we have scanned the environment beforehand, we can apply a relocalization process that analyzes the camera image and tries to match the feature points from that image to the existing scan. Once the process is done, we obtain the position of the device relative to its environment. By utilizing this information, and a concept called shared anchor (see Shared Anchor), we can recover the same coordinate system in the same environment. If the coordinate system is consistent across AR sessions, we can save virtual objects positions, and enable persistent AR experiences.

Similarly, if multiple users want to view the same AR content simultaneously, they need to synchronize their coordinate system. By applying the same relocalization process to every users, we can synchronize their virtual scene, and enable shared AR experiences.