AR-based Indoor Navigation
By Maria Nilga, Dmytro Rusin
It is well known that GPS sensors don’t always provide accurate location information. This is especially the case when inside buildings that block clear line of sight to the satellites that calculate your position. Signals that do enter buildings through windows are often unreliable and can produce location errors up to hundreds of meters. This is why an entirely different technology solution is required in order to provide accurate directions to indoor locations.
Indoor navigation is a solution that provides turn-by-turn directions to locations or objects where GPS and other technologies cannot accurately work.
Where to use indoor navigation
Indoor navigation can be useful in various environments and applications including:
- Office buildings: Employees and guests can easily locate conference rooms, restrooms, etc., in addition to, people, places, or assets like tools and equipment.
- Shopping malls and large stores: Turn-by-turn directions in stores can be particularly helpful as it can lead customers and employees to specific products, which allows use for collecting useful marketing data. With this information, you can make popular items or stores easier to find, or even reconfigure the store layout based on behavior.
- Hospitals: Hospitals often have multiple departments and sections spread across multiple floors or even multiple buildings. As people tend to visit hospitals infrequently, indoor navigation can be extremely helpful in finding specific locations they are trying to find for the first time.
- Airports: Today’s international airports are enormous, sprawling structures that can be miles across. When people have limited time to reach flights, having turn based navigation can help people move to gates far more quickly and efficiently.
- University campuses: Virtual objects can be attached to specific positions in campus buildings to help students, guests, and building managers alike.
Frameworks
The ARKit (iOS) or ARCore (Android) frameworks can be used for visual recognition of various objects such as text, images, or other 2D or 3D shapes. These frameworks provide for the rotation or positioning of each object and give useful information about detected vertical and horizontal planes.
The SceneKit (iOS) or Sceneform (Android) frameworks are designed to build and render 3D objects on the camera view. Using these frameworks it’s possible to show any visual information related to a specific location inside a building.
Restrictions of the ARCore is that it only works on supported devices with high-quality camera, motion sensors, and proper architecture and hardware to support performance.
Indoor navigation solutions overview
There are several approaches that can be taken to implement Indoor navigation. We’ve presented a review of the advantages and disadvantages of each approach:
Beacon and Wi-Fi RTT based indoor navigation
Beacon technology is a current buzzword when it comes to indoor navigation. The most common example is the 2,000 battery-powered Bluetooth Low Energy beacons installed at Gatwick airport in the UK which gives, as claimed, +/-3 metre accuracy.
Disadvantages:
Beacons give only an approximate distance value and we need at least three nearby sensors to determine our location using triangulation, which means the approximation error will be even greater. This solution will not provide acceptable accuracy levels.
When you consider other factors such as cost ($10–20 per item), battery replacement (once every 1–2 years) and the working range (10–100 metres), it’s clear that the use of beacons for indoor navigation is effective only under certain conditions.
Built-in mobile sensors
This solution relies on built-in mobile sensors that can provide users with an estimated value based on an interpretation of the electromagnetic field in the immediate area around the user.
Disadvantages:
The magnetic field can change over time due to external factors, so you need to repeatedly check and update the magnetic map of the building.
AR Cloud with visual world saving
The next two solutions are based on augmented reality technology. The cloud solution provided by Google allows adding virtual objects to an AR scene. Multiple users can then view and interact with these objects simultaneously from different positions in a shared physical space. The position of the location around which the world is anchored is saved to the cloud. In this case, what is meant by “the world” is images that are taken from the camera view.
Disadvantages:
- Cloud anchors can be resolved for no longer than 24 hours after they are hosted. Google is currently developing persistent Cloud Anchors, which can be resolved for much longer time periods.
- This solution only works with a stable network connection.
- The identica (not unique) environment can cause inaccurate results.
Visual Markers — AR-based indoor navigation
This solution is based on creating visual markers, also known as AR Markers, or ARReferenceImage, and then detecting their position. In this case all you need is your camera. With just a camera, you can process data and apply any logic you want. That’s why we ultimately selected this method.
A visual marker is an image recognized by Apple’s ARKit, Google’s ARCore, and other AR SDKs. Visual markers are used to tell the app where to place AR content. If we place a visual marker somewhere in space, on a floor or wall surface for example, then when scanning we will not only be able to see the marker represented in AR but also its exact coordinates in the real world.
So let’s take a more detailed look at this approach in the GD Navigator App and how it can be applied to indoor navigation problems.
GD Navigator App
We selected the Visual Markers solution for our application primarily because this approach doesn’t require GPS, additional hardware, or a permanent network connection. The data can be downloaded to the device on demand, for example due to changes on the server side. Once data is downloaded and cached, a route represented by visual markers can be created on the device without an internet connection. The app has two modes — Admin and User. In Admin mode, an admin manually creates routes with different destinations. In User mode, users can choose their desired destination, define their current location inside the building, and then see the route represented by arrows on the screen.
Admin Mode
1. Create visual markers
In the first step, an admin creates a list of images that can be recognized by ARCore/ARKit. Images can be selected from a gallery or captured from the camera.
Images are recognized as visual markers if they are in the Augmented Image Database format. All images are uploaded to the server, where they are converted to Augmented Image Database using the arcoreimg tool. Each database can store information for up to 1,000 reference images. There’s no limit to the number of databases, but only one database can be active at any time. Then the database is downloaded to the device.
Every object in the real world could be considered as a visual marker, provided it meets certain requirements and cannot be easily confused with another object.
In the app image database, it shows a list of rooms from which the user can choose their desired destination.
2. Create the routes
After creating the list of images (visual markers), the admin can use them to create routes between the rooms. In our implementation a route is represented as a set of items of two kinds: Arrow or Room. To create a route, we need to determine a starting position.
The admin sets the starting position (starting room) by scanning the room sign using the camera. The room sign is recognized by ARCore/ARKit since this image has already been added to the Augmented Image Database. It will be the first anchor that the admin uses to create a route. Having recognized the image, we place a virtual object near the door sign.
To add new items to the route, the admin just needs to tap the screen where they want to leave the arrows, which can be any type of rendered 3D model.
All routes created by the admin are downloaded to the server and saved. The admin can create many routes within one building or across different buildings.
User Mode
In this mode we need to have the map on the app before we can proceed. It can be downloaded from the server or taken from cache and updated if needed.
The first step for the user is to scan a room sign. As each sign is unique, we are able to determine the user’s position on the map after the successful scanning. All coordinates of objects are recalculated relative to the entry point. Of course, distance and the relative position between any two objects remains the same as it was set in the admin mode.
The second step is to select a start and finish point. The user is able to choose between any two different objects on the map. Our task has been simplified to finding the shortest path in the graph. We know that all weights of edges are positive because the distance can’t be negative or zero. It allows us to use Dijkstra’s algorithm and receive the shortest path from the entry point to the destination point.
Now it’s only about rendering the result. Our approach is to add an intermediate object every 2 meters along the path and to render it as an arrow pointing to the next object.
So, in two steps the user is able to get visual step-by-step navigation between two rooms, which means we have effectively implemented indoor navigation.
Difficulties
Creating an AR experience depends on being able to construct a coordinate system for placing objects in a virtual 3D world that maps to the real-world position and motion of the device. When you run a session configuration, ARKit/ARCore creates a scene coordinate system based on the position and orientation of the device. Any objects you create or that the AR session detects are positioned relative to that coordinate system. It means that coordinate systems in different sessions are also different. Thus, we don’t know how much the direction of the coordinate axes has changed between two different AR sessions.
The first approach is to use a built-in device compass and align the axes according to it. In this case, axis rotation remains the same between sessions. So, applying position offset calculations will be enough to determine the object locations. In practice, the compass data is not perfect and is being constantly changed during device usage, which makes all our virtual worlds visually move during the session. Besides, not all devices have a compass. In addition, this type of method is not implemented in ARCore by default, and our requirement was to use only camera data. So we needed another approach to solve this issue.
The second approach is to use some other coordinate system that will remain the same in all sessions. Hence, by choosing a stable object that can be detected by AR (for example, a unique image on a door), we can use this object’s local coordinate system to place all other objects in relation to it. After detecting this object we can use its coordinate system, and it will be the same in any session. That was our solution.
Conclusion
After encountering various technical challenges, we now have a fully functioning application that can provide turn-by-turn navigation to any room on one of our office floors. This tool can be highly valuable to our newcomers and guests as they can locate rooms faster and more easily, while not needing to seek the assistance of a stranger.
Our application could be scaled and adapted to offer indoor navigation in any environment. If you are interested in this solution or us as a company, don’t hesitate to reach out!
Demo video
We have a demo that demonstrates the features described above. Check out the video, and see how useful augmented reality apps will be for shoppers!
