Content for  TR 26.928  Word version:  17.0.0

Use Case Description: Streaming of Immersive 6DoF Alice consumes a recorded highlight of a basketball match being seated close to the court by using an application on the 5G enabled HMD. For this, Alice wears an HMD together with a 6DoF manual controller. The HMD is connected to a 5G network but has no other tethered connection. The 6DoF controller allows to change the viewing position (i.e. the seat) and looking at the action from different angles. In addition, restricted local 6DoF movement of Alice at a location enables to locally interact with the scene based on HMD sensors. Even more the controller allows to rewind, slow mo and pause the scene. In the pause or slow-motion mode, the scene can be viewed from different angles using the controller and head motion. The scene is overlaid with information that helps Alice to navigate through the scene. Alice feels present in the scene. In an extension to the use case, the game is consumed in a live mode. Categorization Type: VR Degrees of Freedom: 3DoF+, 6DoF Delivery: Streaming, Interactive, Split Device: HMD with a controller Preconditions

Application is installed that permits to consume the scene

The application uses existing HW capabilities on the device, including A/V decoders, rendering functionalities as well as sensors. Inside-out Tracking is available.

Media is captured properly and accessible on a server, preferably on a CDN.

Requirements and QoS/QoE Considerations

Required QoS:

Bitrates and Latencies that are sufficient to render the viewport within the immersive limits.

Some numbers are provided here: https://www.roadtovr.com/nextvr-latest-tech-is-bringing-new-levels-of-fidelity-to-vr-video/

in the best case scenario with 8 Mbps bandwidth, the company can now stream 20 pixels per degree. Keep in mind, that's also in stereo and at 60 FPS

plans to roll out this higher-res playback

If full 6DoF with presence needs to be enabled, up to 100 Mbit/s may be necessary.

However, with viewport adaptive streaming and/or split rendering architectures, the requirements on bitrates may be lower, but the latency requirements may increase. A more detailed study is necessary.

Required QoE:

Fast startup of the service,

fast reaction to manual controller information,

reaction to head and limited body movement within immersive limits,

seamless experiences when moving across positions

providing sufficient AV experience to enable presence. https://xinreality.com/wiki/Presence

highest image quality, stereoscopy

should also work in slow motion

Feasibility

Content generated in 6DoF

6DoF content is generated by companies such as NextVR ™:

https://www.roadtovr.com/nextvr-latest-tech-is-bringing-new-levels-of-fidelity-to-vr-video/

https://www.digitaltrends.com/home-theater/nextvr-nba-league-pass-writing-future-of-vr/

https://www.vrfocus.com/2019/02/nextvr-and-qualcomm-to-demo-5g-6dof-vr-streaming-at-mwc19/

"Fearless is designed to play on a 5G enabled handset powered by the Qualcomm Snapdragon 855 Mobile Platform and features six-degrees-of-freedom (6DoF) streaming."

https://www.benzinga.com/pressreleases/19/02/r13233697/nextvr-to-demonstrate-6dof-vr-streaming-over-5g-and-new-ar-portal-at-m

To create Fearless ™, NextVR™ used a state-of-the-art, proprietary camera that generates the 6DoF volume in ultra-high resolution.

This includes 6DoF captured audio and video

Selected Devices/XR Platforms supporting this:

Oculus Quest ™ is announced https://www.oculus.com/quest/

Vive Cosmos ™

https://uploadvr.com/vive-cosmos-everything-we-know/

Qualcomm ™ reference design:

https://www.vrandfun.com/the-qualcomm-snapdragon-855-will-be-able-to-deliver-up-to-8k-360-video-playback/

https://www.roadtovr.com/qualcomm-reference-headset-2x-pixels-vive-pro-ces-2018/

https://venturebeat.com/2019/02/25/qualcomms-5g-xr-viewers-will-stop-the-wave-of-mediocre-ar-headsets/

https://www.i4u.com/2019/02/130947/qualcomm-pushes-5g-connected-ar-and-vr-viewers

Potential challenges to make this happen within 3 years

Broadly available high-quality 6DoF and volumetric capturing systems. There are still not enough variety of volumetric content to get a feel for how it would handle more challenging scenes like those with closer and/or faster moving objects

Broad availability of HMDs and end devices supporting the playback

Availability of access bandwidth to stream such services

Potential Standardization Status and Needs

The following aspects may require standardization work:

Coded Representation of Audio/Video Formats as well as geometry data

Scene composition and description

Storage and Cloud Access Formats

Delivery Architectures to support 6DoF Streaming

Content Delivery and Streaming Protocols

Decoding, rendering and sensor APIs

Network conditions that fulfill the QoS and QoE Requirements

Use Case Description: Emotional Streaming Bob is watching a horror movie using a 5G connected HMD. He is fascinated, but his body reaction, eye rolling, and other attributes are collected and are used to create a personalized story line. Movie effects are adjusted for personal preferences while reactions are collected when watching the movie. Bob's emotional reactions determine the story-line. Alice is watching the same story on her newest 5G connected smart phone. Categorization Type: 2D interactive, VR and AR Degrees of Freedom: 2D, 3DoF+, 6DoF Delivery: Streaming, Interactive, Split Device: Phone and HMD Preconditions

Application is installed that permits to consume the story

The application uses existing HW capabilities on the device, including A/V decoders, rendering functionalities as well as sensors

The application uses AI functionalities to extract personalized reactions based on sensor tracking

Requirements and QoS/QoE Considerations

QoS:

Bitrates and Latencies that are sufficient to render the viewport within the immersive limits or at least to react to the emotions

QoE:

Sufficiently fast reaction to body emotion feedback,

for HMD, reaction to head movement within immersive limits,

providing sufficient AV experience to enable presence.

Streaming with seamless transitions from one scene to either of the choices

Feasibility

https://www.cnet.com/news/with-5g-you-wont-just-be-watching-video-itll-be-watching-you-too/

Interactive and branching content

Netflix's Bandersnatch ™ provides an example for content interactive streaming.

Also games use similar decision making trees. Examples are provided here:

http://skipabeatgame.com/

https://www.digitaltrends.com/cool-tech/bring-to-light-heart-rate-vr/

https://vrscout.com/news/vr-horror-game-tracks-heart-rate/

Device Features

Facial expression tracking with AI is available on mobile devices

Eye Tracking combined with AI is available on mobile devices

IoT/Wearable devices provide the ability to measure biometric metrics such as heart beat and other stress detecting factors (skin changes, etc.) and may be connected with app

Biometric and Emotion Tracking Technologies are summarized:

https://blog.therachat.io/emotion-tracking/

https://www.aplanforliving.com/6-wearables-to-track-your-emotions/

https://www.inc.com/magazine/201607/tom-foster/lightwave-monitor-customer-emotions.html

Potential Standardization Status and Needs

The following aspects may require standardization work:

Coded Representation of Audio/Video Formats

Seamless splicing and smooth transitions across storylines

Scene composition and description

Storage and Cloud Access Formats

Content Delivery Protocols

Decoding, rendering, sensor and emotion tracking APIs

Biometrics and Emotion Metadata definition and delivery

Use Case Description: Untethered Immersive Online Gaming 100 friends play Fortnite Battle Royal ™. Of the the 100 friends, several are on travel and connect on a stand-alone HMD. The HMD has a with 5G connection. Fortnite Battle Royale ™ is a free-to-play battle royale video game. As a battle royale game, Fortnite Battle Royale features up to 100 players, alone, in duos, or in squads of up to four players, attempting to be the last player or group alive by killing other players or evading them, while staying within a constantly shrinking safe zone to prevent taking lethal damage from being outside it. Players start with no intrinsic advantages, and scavenge for weapons and armor to gain the upper hand on their opponents. The game features cross-platform play between the platforms that was limited for the first five seasons, before the restrictions were eased. Other popular VR games are here:

https://veer.tv/blog/30-best-vr-games-for-playstation-vr-oculus-rift-htc-vive-in-2018/

https://uploadvr.com/best-psvr-games/

https://www.digitaltrends.com/gaming/best-psvr-games/

Population: One ™

https://www.ign.com/articles/2019/01/11/population-one-isnt-quite-fortnite-vr-but-its-pretty-convincing

http://www.populationonevr.com/

https://uploadvr.com/ces-population-one-preview/

https://vrgames.io/game/

Categorization

Type: VR

Degrees of Freedom: 6DoF

Delivery: Streaming, Interactive, Split

Device: HMD with a Gaming controller

Preconditions

Gaming client is installed that permits to consume the game

The application uses existing HW capabilities on the device, including game engines, rendering functionalities as well as sensors. Inside-out Tracking is available.

Connectivity to the network is provided.

Requirements and QoS/QoE Considerations

Collected Statistics:

https://www.zdnet.com/article/how-fortnite-approaches-analytics-cloud-to-analyze-petabytes-of-game-data/

Fortnite ™ processes 92 million events a minute and sees its data grow 2 petabytes a month

Akamai ™ said Fortnite set a game traffic record on its network July 12 with 37 terabytes

https://www.techadvisor.co.uk/feature/game/how-much-data-does-fortnite-use-3683618/per second delivered across its platform.

Techadvisor IDG © Copyright 2019 IDG UK. All Rights Reserved has checked our data usage, and according to the tool, the 15-minute session used 12.4MB of mobile data. That may sound like a lot, but it's the equivalent of streaming a one- or two-minute video on YouTube ™. It may vary slightly depending on a number of factors, but Techadvisor IDG © estimate that Fortnite uses between 10-15MB per 15 minutes of gameplay, or around 50-60MB per hour.

Required QoS:

https://broadbandnow.com/guides/best-internet-service-setup-serious-gamers

Any connection over 2 mbps with less than 75ms ping should work well for 99% of games.

the main factors affecting your gameplay are:

Efficiency of your network

Distance to other players in multiplayer games

QoS and network prioritization might not matter much for the average Internet user, but for gamers it can make a big difference in network lag.

Ping is king.

Different scenarios need to be looked at, for example where the rendering is happening.

Required QoE:

fast reaction to manual controller information,

reaction to head movement within immersive limits,

providing sufficient gaming rendering experience to enable presence.

https://xinreality.com/wiki/Presence

supporting frame rate not lower than 60 FPS and resolution not lower than 8K

The TR 22.842 provides some information as well, please refer to clause 5.3.1. Summary of some discussions:

Latency requirements for online games may be very tight. Examples

Current mainstream FPS (First Person Shooter) game requires 60 frames per second, which means frame interval is 16.67ms. If rendering is done in the cloud and taking out the delay for rendering and encoding/decoding processing, the network round trip time (RTT) delay should be less than 5ms.

MOBA (Multiplayer Online Battle Arena) game requires 20ms RTT.

Resolutions and frame rates need to be sufficiently high: higher than 60 FPS and 8K resolution

Packet loss rates should be low as game experiences degrade quickly

And some references from TR 22.842

O. Abari, D. Bharadia, A. Duffield, and D. Katabi, "Cutting the Cord in Virtual Reality," in Proceedings of the 15th ACM Workshop on Hot Topics in Networks. ACM, 2016, pp. 162-168.

E. Bastug, M. Bennis, M. M´dard, and M. Debbah, "Toward Interconnected Virtual Reality: Opportunities, Challenges, and Enablers," IEEE Communications Magazine, vol. 55, no. 6, pp. 110-117, 2017.

Athul Prasad, Mikko A. Uusitalo, David Navrátil, and Mikko Säily, "Challenges for Enabling Virtual Reality Broadcast Using 5G Small Cell Network", IEEE Wireless Communications and Networking Conference Workshops, pp. 220-225, 2018.

Mohammed S. Elbamby, Cristina Perfecto, Mehdi Bennis, and Klaus Doppler, "Toward Low-Latency and Ultra-Reliable Virtual Reality", IEEE Network, March/April, 2018.

Orlosky, Jason & Kiyokawa, Kiyoshi & Takemura, Haruo, "Virtual and Augmented Reality on the 5G Highway", Journal of Information Processing, 25. 133-141. 10.2197/ipsjjip.25.133.

J. Huang, Z. Chen, D. Ceylan and H. Jin, "6-DOF VR videos with a single 360-camera", 2017 IEEE Virtual Reality (VR), Los Angeles, CA, 2017, pp. 37-44. doi: 10.1109/VR.2017.7892229.

Impact of Packet Losses on the Quality of Video Streaming, https://www.diva-portal.org/smash/get/diva2:831420/FULLTEXT01.pdf

New Study from GSMA and CAICT Forecasts That China Will Be the World's Largest 5G Market by 2025, https://www.webscalenetworking.com/news/2017/06/27/8571173.htm

Feasibility

Available Games

Fortnite ™ is available as a game and can be downloaded

Other VR games are also available or in beta:

Population: One ™

https://www.ign.com/articles/2019/01/11/population-one-isnt-quite-fortnite-vr-but-its-pretty-convincing

http://www.populationonevr.com/

https://uploadvr.com/ces-population-one-preview/

https://vrgames.io/game/

Selected Devices/XR Platforms supporting this:

Oculus Rift ™, Playstation VR ™, HTC Vive ™

These are tethered and connected devices

Specifications are here: https://www.digitaltrends.com/virtual-reality/oculus-rift-vs-htc-vive/

Oculus Go ™

Oculus Quest ™ is announced https://www.oculus.com/quest/

An important aspect is that the processing power of untethered devices is typically lower as all processing needs to be done on the device. The feasibility is likely improved by supporting the device with additional network processing.

Demos and Architectures are provided that show cloud and split rendering:

NVIDIA ™ Cloud Rendering: https://www.nvidia.com/object/gpu-cloud-rendering.html

Google ™ Cloud Rendering: https://www.zyncrender.com/

Split Rendering: https://www.qualcomm.com/news/onq/2018/09/18/boundless-xr-new-era-distributed-computing

Potential Challenges:

Getting end-to-end workflow in place

Operational costs

Potential Standardization Status and Needs

The following aspects may require standardization work:

Network conditions that fulfill the QoS and QoE Requirements

Content Delivery Protocols

Decoding, rendering and sensor APIs

Architectures for computing support in the network

TR 22.842 provides a gap analysis in clause 5.3.6 that is in line with these needs