illustrates the generic procedure diagram for cognitive immersive services for both STAR-based and EDGAR-based UEs.
Prerequisites and assumptions:
The AR/MR Scene Manager includes immersive media rendering and scene graph handling functionalities.
The Media Player includes immersive content delivery and immersive media decoding functionalities.
The AR/MR Application in the UE is run by the user.
The UE initialises AR registration (starts analysing the surroundings where a user/UE is located), it namely:
captures its surroundings via camera(s)
analyses where the device is located
registers the device into the analysed surroundings.
AR/MR Application and AR/MR Application Provider have exchanged some information, such as device capability or content configuration, for content rendering. The exchange procedures for device capability and content configuration are FFS.
AR/MR Application Provider has established a Provisioning Session and its detailed configurations has been exchanged.
AR/MR Application Provider has completed to set up ingesting immersive contents.
The Scene Server context is established, and scene content is ingested by the Media AS.
Service Announcement is triggered by AR/MR Application. Service Access Information including Media Client entry or a reference to the Service Access Information is provided through the M8d interface.
Desired media content is selected.
Optionally, the Service Access information is acquired or updated.
The AR/MR Application initializes the Scene Manager with the entry point (full scene description) URL.
The Media Client establishes the transport session for the scene session between the Scene Manager in the UE and the Scene Server.
The Media Client requests and receives the full scene description. The entry point (scene description) is processed by the AR/MR Scene Manager, and a scene session is created.
The AR/MR Scene Manager requests the creation of a new AR/MR session from the AR Runtime.
The AR Runtime creates a new AR/MR session.
Scene session loop, steps 10~24, send the interaction and pose information and receives and renders the updated scenes accordingly:
The latest sensor data (e.g. captured media) is acquired by the AR/MR Scene Manager and shared with the Media Client. The Media Client sends this information to the Media AS and AR/MR Application.
The AR/MR Application performs cognitive processing according to the sensor data from the UE. Depending on the outcome, the current scene may be updated or replaced.
When needed, one of the following steps:
The Scene Server sends a new scene entry point to the AR/MR Scene Manager through the Media AS and Media Client (go to step 7), or
The Scene Server sends a scene update (updating streams/objects) to the AR/MR Scene Manager through the Media AS and Media Client.
The AR/MR Scene requests to create additional streaming sessions if needed for new media objects in the scene.
The number of the additional streaming sessions depends on the delivery mechanism. One or more media objects may be delivered through a single manifest and/or use of the same connection. Therefore, introduction of every new media object may not need an additional streaming session.
The Media Session Handle establishes the additional streaming sessions based on the received request.
Streaming session, steps 15~18 establish the transport sessions for media objects and configure the media pipelines
For the required media content, the Media Client establishes the transport session(s) to acquire delivery manifest(s) information.
The Media Client requests and receives the delivery manifest(s) from the Media AS.
The Media Client processes the delivery manifest(s). It determines for example the number of needed transport sessions for media acquisition. The Media Client is expected to be able to use the delivery manifest(s) information to initialize the media pipelines for each media stream.
The AR/MR Scene Manager and Media Client configures the rendering and delivery media pipelines.
The Media Client establishes the transport session(s) to acquire the media content.
Media session loop includes steps 20~24 which are for streaming, decoding and rendering media components:
The Media Client requests the media data according to the delivery manifest processed, possibly taking into account pose information (e.g., viewport dependent streaming).
The Media Client receives the media data and triggers the media rendering pipeline(s), including the possible registration of AR content into the real world accordingly (depending on the device type).
The Media Client decodes and processes the media data. For encrypted media data, the Media Client may also perform decryption.
The Media Client passes the media data to the AR/MR Scene Manager.
The XR Spatial Compute Pipeline as specified in
The AR scene data and XR Spatial Compute data are combined for composition and rendering.