Augmented Reality (AR) and Mixed Reality (MR) promise to provide new experiences for immersive media services. The form factors of the devices for these services are typically not expected to deviate significantly from those of typical glasses, resulting in less physical space for the various required components such as sensors, circuit boards, antennas, cameras, and batteries, when compared with typical smartphones. Such physical limitations also reduce the media processing and communication capabilities that may be supported by AR/MR devices, in some cases requiring the devices to offload certain processing functions to a tethered device and/or a server.
This report addresses the integration of such new devices into 5G system networks and identifies potential needs for specifications to support AR glasses and AR/MR experiences in 5G.
The focus of this document is on general system aspects, especially targeting visual rendering on glasses, and may not be equally balanced or equally precise on all media types (e.g. on haptics, GPUs).
The present document collects information on glass-type AR/MR devices in the context of 5G radio and network services. The primary scope of this Technical Report is the documentation of the following aspects:
Providing formal definitions for the functional structures of AR glasses, including their capabilities and constraints,
Documenting core use cases for AR services over 5G and defining relevant processing functions and reference architectures,
Identifying media exchange formats and profiles relevant to the core use cases,
Identifying necessary content delivery transport protocols and capability exchange mechanisms, as well as suitable 5G system functionalities (including device, edge, and network) and required QoS (including radio access and core network technologies),
Identifying key performance indicators and quality of experience factors,
Identifying relevant radio and system parameters (required bitrates, latencies, loss rates, range, etc.) to support the identified AR use cases and the required QoE,
Providing a detailed overall power analysis for media AR related processing and communication.
The following documents contain provisions which, through reference in this text, constitute provisions of the present document.
References are either specific (identified by date of publication, edition number, version number, etc.) or non-specific.
For a specific reference, subsequent revisions do not apply.
For a non-specific reference, the latest version applies. In the case of a reference to a 3GPP document (including a GSM document), a non-specific reference implicitly refers to the latest version of that document in the same Release as the present document.
H. Chen, Y. Dai, H. Meng, Y. Chen and T. Li, "Understanding the Characteristics of Mobile Augmented Reality Applications," 2018 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2018, pp. 128-138.
Serhan Gül, Dimitri Podborski, Jangwoo Son, Gurdeep Singh Bhullar, Thomas Buchholz, Thomas Schierl, Cornelius Hellge, "Cloud Rendering-based Volumetric Video Streaming System for Mixed Reality Services", Proceedings of the 11th ACM Multimedia Systems Conference (MMSys'20), June 2020
S. N. B. Gunkel, H. M. Stokking, M. J. Prins, N. van der Stap, F.B.T. Haar, and O.A. Niamut, 2018, June. Virtual Reality Conferencing: Multi-user immersive VR experiences on the web. In Proceedings of the 9th ACM Multimedia Systems Conference (pp. 498-501). ACM.
E. Rublee, V. Rabaud, K. Konolige and G. Bradski, "ORB An efficient alternative to SIFT or SURF,"! 2011 International Conference on Computer Vision, 2011, pp. 2564-2571, doi: 10.1109/ICCV.2011.6126544.
For the purposes of the present document, the terms and definitions given in TR 21.905 and the following apply. A term defined in the present document takes precedence over the definition of the same term, if any, in TR 21.905.
5G AR/MR media service enabler:
A 5G AR/MR media service enabler is supporting an AR/MR application to provide AR/MR experience using at least partially 5G System tools.
5G System (Uu):
Modem and system functionalities to support 5G-based delivery using the Uu radio interface.
a software application that integrates audio-visual content into the user's real-world environment.
AR/MR content consists of a scene with typically one or more AR objects and is agnostic to a specific service.
Data generated by the AR Runtime that is accessible through API by an AR/MR application such as pose information, sensors outputs, and camera outputs.
AR Media Delivery Pipeline:
pipeline for accessing AR scenes and related media over the network.
An AR/MR object provides a component of an AR scene agnostic to a renderer capability.
a set of functions that interface with a platform to perform commonly required operations such as accessing controller/peripheral state, getting current and/or predicted tracking positions, general spatial computing, and submitting rendered frames to the display processing unit.
Lightweight Scene Manager:
A scene manager that is capable to handle a limited set of 3D media and typically requires some form of pre-rendering in a network element such as the edge or cloud.
Media Access Function:
A set of functions that enables access to media and other AR related data that is needed in the scene manager or AR Runtime in order to provide an AR experience.
The collection of sensors, cameras, displays and other functionalities on the device that provide a physical connection to the environment.
a set of functions that supports the application in arranging the logical and spatial representation of a multisensorial scene with support of the AR Runtime.
Simplified Entry Point:
An entry point that is generated by 5G cloud/edge processes to support offloading processing workloads from UE by lowering the complexity of the AR/MR content.
AR functions which process sensor data to generate information about the world 3D space surrounding the AR user.
XR Spatial Description:
a data structure describing the spatial organisation of the real world using anchors, trackables, camera parameters and visual features.
XR Spatial Compute Pipeline:
pipeline that uses sensor data to provide an understanding of the physical space surrounding the device and uses XR Spatial Description information from the network.
XR Spatial Compute server:
an edge or cloud server that provides spatial computing AR functions.
XR Spatial Description server:
a cloud server for storing, updating and retrieving XR Spatial Description.
For the purposes of the present document, the abbreviations given in TR 21.905 and the following apply. An abbreviation defined in the present document takes precedence over the definition of the same abbreviation, if any, in TR 21.905.
5G Media Streaming
Advanced Audio Coding
Application Programming Interface
Augmented Reality Framework
Terminal Audio quality performance and Test methods for Immersive Audio Services
Asynchronous Time Warp
Advanced Video Coding
Bluetooth Low Energy
Based Media File Format
Computer Aided Design
Computer Generated Imagery
Common Media Application Format
Centre of Mass
Central Processing Unit
Call Session Control Function
Dynamic Adaptive Streaming over HTTP
Data Channel Multimedia Telephony Service over IMS
Draft International Standard
Degree of Freedom
Datagram Transport Layer Security
Edge Application Server
Evolution of IMS Multimedia Telephony Service
Streaming Architecture extensions for Edge processing
Essential Video Coding
Final Draft International Standard
For Future Study
Framework for Live Uplink Streaming
Field of View
Frame Per Second
Geometry-based Point Cloud Compression
Guaranteed Bit Rate
Graphics Library Transmission Format
Global Positioning System
Graphics Processing Unit
High Density Camera Array
High Efficiency Video Coding
HTTP Live Streaming
HyperText Markup Language
HyperText Transfer Protocol
Interactive Connectivity Establishment
IP Multimedia Subsystem
Inertial Measurement Unit
Industry Specification Group
ISO Based Media File Format
Immersive Teleconferencing and Telepresence for Remote Terminals