Content for TR 26.998 Word version: 18.0.0

0… 4… 4.2… 4.2.2… 4.2.2.2 4.2.2.3 4.2.2.4 4.2.3… 4.3… 4.4… 4.5… 4.6… 4.6.4… 4.6.5… 4.6.8… 5 6… 6.2… 6.2.4… 6.2.4.2 6.2.5… 6.3… 6.3.4… 6.3.4.2 6.3.5… 6.4… 6.4.4 6.4.5… 6.5… 6.5.4 6.5.5 6.5.6… 6.6… 6.6.4 6.6.5… 7… 8… 8.9 9 A… A.2 A.3… A.4 A.5 A.6 A.7…

A.5 Use Case 20: AR IoT control
...

A.5 Use Case 20: AR IoT control p. 114

Use Case Description: AR IoT control
Many IoT devices are present in the home and several of them such as smart light bulbs, smart curtains, air conditioning systems, heaters or multimedia devices are present in multiple numbers in different rooms within the home. While some IoT devices are at a fixed position and rarely move (light bulbs, heaters, curtains…) others are portable and nomadic by nature (portable speakers, vacuum cleaner robot, wearable devices…). There are today many protocols that can be used for IoT (Wifi, Bluetooth, Zigbee TM, io-homecontrol TM, Z-wave TM but also LTE, NB-IoT and now Sidelink).and in the future there will be more and more IoT devices with 5G connectivity. While it is likely all IoT devices within a single home are interconnected through at least one (or more) gateway, D2D communications may also be used when available as it is likely to be more battery efficient for AR glasses. As the user walk through his home, his AR glasses regularly scan the indoor environment to track the user's position in the home. While there could be several users wearing AR glasses in the same room, the environment reconstruction may also be using volumetric information from other IoT devices in the room (for instance security cameras with depth sensors). In terms of data exchange, upload data to 5G network may be video, depth-maps, sparse point clouds and also sensor information such as gyroscope or accelerometer. Additionally, thanks to AR glasses' scanning, the home IoT system keeps track of IoT devices positions. Actual identification of an IoT device may also be done in the AR glasses themselves Typically, the IoT home system runs on the edge network and information sent back to the AR glasses includes metadata information about IoT devices and environment, simple textual overlays for UIs. More advanced UIs would also probably make use of elements such as video or 3D objects And finally, for D2D communications with 5G enabled IoT devices, control and status information is also exchanged between IoT devices and the AR glasses. The use case addresses several scenarios: The user controls a specific IoT device by just looking at it through the glasses and operates it via an AR displayed user interface and user controls such as touch or voice controls available on the glasses. Since home IoT system can know in real-time the position of the user in the home as well as which IoT devices are present in the same room as the user, the user can control IoT device with simple voice control without even targeting a specific IoT device. For instance, the user can say: "switch off the light" to operate the main light of the room he is currently in.
Categorization
Type: AR Degrees of Freedom: 6DoF Delivery: Interactive, Split, device-to-device Device: AR Glasses
Preconditions
IoT Home Application is installed on the AR glasses or phone connected to AR glasses The application uses existing HW capabilities on the device, rendering functionalities as well as sensors (audio, video, Lidar). Inside-out Tracking is available thanks to AR glasses sensors and may also be enhanced thanks to similar sensing capabilities on IoT devices. Connectivity to the network is provided on the glasses or through the connected phone. Wayfinding and SLAM is provided to locate user and map the environment and may be provided with split-processing (tethered device or 5G edge network). AR and AI functionalities are provided for example for Image & Object Recognition in order to track IoT devices positions. Connectivity to IoT devices may be device-to-device (Bluetooth, Sidelink, …), device to local IoT gateway (WiFi) or device-to-edge (5GNR).
Requirements and QoS/QoE Considerations
5G's low-latency high-bandwidth capabilities are used to provide real-time operation of IoT devices and high-speed upload of sensor information (video, Lidar point clouds). Continuous connectivity is required for the sharing of local information to keep tracking user's position and position of IoT devices in the home so that home IoT system can maintain a real-time map of the home and its user. The underlying AR maps are expected to be accurate and up to date. In order to optimize energy consumption on the glasses, split processing is favoured and efficient compression technology is required for exchanges with the home IoT system. Device-to-device communication may also be used with some IoT devices.
Feasibility
Google Visual Positioning Service: https://www.roadtovr.com/googles-visual-positioning-service-announced-tango-ar-platform/ XR clients continuously send sensing data to a cloud service. The service constructs a detailed and timely map from client contributions and provides the map back to clients. Example is Google's Visual Positioning Service Drivenet Maps - Open Data real-time road Maps for Autonomous Driving from 3D LIDAR point clouds: https://sdi4apps.eu/2016/03/drivenet-maps-open-data-real-time-road-maps-for-autonomous-driving-from-3d-lidar-point-clouds/ An XR HMD receives a detailed reconstruction of a space, potentially captured by a device(s) with superior sensing and processing capabilities. An example of navigation is given in the MPEG-I use case document for point cloud compression (w16331, clause 2.6) Xiaomi MIOT Ecosystem - Around the MiHome application, users can control all MIOT devices from many brands : https://xiaomi-mi.com/ecosystem/ MIOT devices can be controlled locally or through remote locations thanks to cloud servers. Some MIOT devices such as security cameras or vacuum cleaner robots have the capability to track objects and capture the environment (Audio, Video, Lidar) and make it available to the IoT home system.
Potential Standardization Status and Needs
The following aspects may require standardization work: Data representations for AR map of the home Collected sensor data to be uploaded to edge or tethered device (with dedicated compression solutions) Scalable streaming and storage formats for AR maps Content delivery protocols to access AR maps and content items Content delivery protocols to send AR UI elements when IoT application runs on tethered device or edge. Network conditions that fulfil the QoS and QoE Requirements Device-to-device communications.