Content for TR 26.854 Word version: 19.0.0
10 QoS and QoE
10.1 Introduction
10.2 Asynchronicity between haptics and other media
10.3 QoS requirements for haptic media enhanced services
10.4 PDU Set Guidelines for haptics media
...
...
10 QoS and QoE p. 37
10.1 Introduction p. 37
The QoS factors influencing the QoE for haptics media enhanced services as described in the use-cases of clause 5 are mostly similar to those used for traditional AV and immersive AV services (jitter, delay, packet loss, etc). The asynchronicity between haptics and other media is a prevalent parameter to be considered when using haptics along with other media. Another characteristic of parametric haptics media is the presence of silent units alongside temporal and spatial units; which can be taken in account when setting PDU Set Importance.
10.2 Asynchronicity between haptics and other media p. 37
In Haptics enhanced media services, the quality of experience may be impacted by the asynchronicity of the haptics media with either audio, or video. This asynchronicity may be due to various QoS factors which are for further study.
Studies demonstrate (such as [21], [20] and [40]) that haptics effects perceived prior to visual cues is less detrimental to the user experience than the reverse and that asynchronicity is more tolerable in scenarios involving passive user involvement than active user involvement.
The suitable threshold for asynchronicity [20], [21], TR 22.847, TR 22.856 is listed in Table 10.2-1.
| Use case | Media | Tolerable asynchronicity threshold (note 1) | |
|---|---|---|---|
| Haptic enhanced media distribution (5.2) [40] | audio-haptics | audio delay: 100ms | haptic delay: 50ms |
| visual-haptics | visual delay: 80ms | Haptic delay: 60ms | |
| Haptic enhanced communication (5.3) [4] | audio-haptics | audio delay: 3 frames (25ms) | haptic delay: 1 frame (12ms) |
| Visual-haptics | Visual delay: 20ms | Haptic delay: 30ms | |
| Immersive games and Immersive multimodal XR and metaverse (5.4 and 5.5) TR 22.847 | audio-haptics | audio delay: 50 ms | haptic delay: 25 ms 1 frame for gaming |
| visual-haptics | visual delay: 15 ms | Haptic delay: 50 ms | |
| Immersive entertainment (5.4) TR 22.856 | audio-haptics | audio delay: 25 ms | haptic delay: 12 ms |
| visual-haptics | visual delay: 20 ms | Haptic delay: 30ms | |
|
NOTE:
For each media component, "delay" refers to the case where that media component is delayed compared to the other.
|
|||
10.3 QoS requirements for haptic media enhanced services p. 37
The following from TR 22.847 is adjusted to the use cases in clause 5.
1.
Users in an enhanced haptics communication services are equipped with multiple devices to receive and/or capture and transmit audio, video and haptics media.
2.
When a session starts, multiple streams are established over the wireless network between the corresponding UEs. Table 10.3-1 depicts the typical QoS requirements that need be fulfilled for the users' QoE to be satisfactory.
| Haptics | Video | Audio | |
|---|---|---|---|
| Jitter (ms) | ≤ 2 | ≤ 30 | ≤ 30 |
| Delay (ms) | ≤ 50 | ≤ 400 | ≤ 150 |
| Packet loss (%) | ≤ 10 | ≤ 1 | ≤ 1 |
| Update rate (Hz) | ≥ 2000 | ≥ 24 | ≥ 20k |
| Packet size (bytes) per channel | 60-350 compressed parametric 50-1500 time sampled | ≤ MTU | 160-320 |
| Throughput (kbit/s) per channel | n*(fe*16) (NOTE 1) bits/s time sampled 16-32kbps for compressed parametric (NOTE 2) | 2500 - 40000 | 64-128 |
|
NOTE 1:
Where n is the number of channels and fe the sampling frequency synthetized in 16 bits.
NOTE 2:
Throughput for compressed parametric vary based on density and placement of keyframe.
|
|||
3.
Whether the 5QIs defined in TS 23.501 for media streaming and communication services are sufficient to enable haptics media enhanced services may need to be studied further.
Haptics media (MIHS), video and voice or audio are captured with different sampling intervals. A pacer may be configured to transmit the haptic media unit (MIHS) earlier than the media units (MUs) of audio and video, which may result in different arrival time of the MUs of different modalities. Conversely, if the receiver outputs the MUs synchronously, the output of the haptics media MU may need to be delayed until the voice, audio and or video MUs arrive at the destination. Synchronization at the rendering time may be performed by buffer and jitter management. Other mechanisms are for further study.
4.
Synchronization between the media streams becomes increasingly challenging when considering asynchronicity between two or three media types, and with the increasing demand from the application itself, for example in an immersive XR experience, or in dense haptics enhanced media services. . Whether any additional information needs to be exchanged with the 5G network for the assistance of the synchronization at a suitable granularity needs to be further study.
10.4 PDU Set Guidelines for haptics media p. 38
10.4.1 PDU Set Importance (PSI) p. 38
In general, whenever the RAN needs to discard packets (e.g., under congestion situations), it is better to discard packets of lower importance rather than discarding packets randomly. If a discarded packet is critical for the media stream, the QoE may be severely degraded. For this reason, the PDU Set Importance (PSI) field TS 26.522 can be used to mark PDU Sets with their importance level. The PSI field can then be used by the RAN to discard PDU sets. In case of congestion, PDU Sets with higher PSI values are more likely to be discarded.
The RTP payload format for the transport of MPEG-I haptic data is defined in IETF draft-ietf-avtcore-rtp-haptics [29]. The RTP payload header format allows for packetization of MIHS units in an RTP packet payload as well as fragmentation of an MIHS unit into multiple RTP packets.
The RTP Payload Header of a haptics media RTP packet follows the RTP header. Figure 10.4.1-1 describes the RTP Payload Header for Haptics media as described in RTP Payload format for Haptics specification.
The haptic media characteristics may include a MIHS unit type, a dependent MIHS unit and a MIHS unit Layer information. The PSI field in a PDU Set Marking RTP HE may be assigned based on the haptic media characteristics.
An initialization MIHS unit contains MIHS packets carrying metadata necessary to reset and initialize a haptic decoder, including a timestamp. So, initialization MIHS units are essential to configure the decoder at the beginning of the decoding process. In a haptic media RTP packet containing the initialization MIHS units, the unit type field of the RTP payload header is set with value 1. When the unit type field value in the RTP payload header of an RTP packet is 1, then the corresponding PDUs in that PDU Set should be set with higher importance relative to other haptic media data PDUs.
A spatial MIHS unit may contain one or more MIHS packets providing time-independent effects, such as vibrotactile texture, stiffness, and friction. The spatial MIHS packets present in a haptics media bitstream are independently decodable by a haptics media decoder. Hence the spatial MIHS units present in a PDU Set may be delivered with higher priority. In a haptic media RTP packet containing the spatial MIHS units, the unit type field of the RTP payload header is set with value 3. When the unit type field value in the RTP payload header of an RTP packet is 3, then the corresponding PDUs in that PDU Set should be set with higher importance relative to temporal and silent haptic media data PDUs and equal or slightly lesser importance relative to initialization haptic media data PDUs.
A temporal MIHS unit may contain one or more MIHS packets defining time-dependent effects and providing modalities such as pressure, velocity, and acceleration. Temporal MIHS units can be dependent or independent units. Independent temporal MIHS units can be decoded independently using a haptics media decoder. Whereas a dependent temporal MIHS unit is the continuation of previous MIHS units and cannot be independently decoded and rendered without having decoded previous MIHS unit(s). In a haptic media RTP packet containing the temporal MIHS units, the unit type field of the RTP payload header is set with value 2. When the temporal MIHS unit is independently decodable, then the dependency field value in the RTP payload header of such haptics media RTP packet is set to 0. When the temporal MIHS unit is not independently decodable, then the dependency field value in the RTP payload header of such haptics media RTP packet is set to 1. When the unit type field value in the RTP payload header of an RTP packet is 1 and the dependency field value is zero, then the corresponding PDUs in that PDU Set should be set with higher importance relative to dependent temporal haptic media data PDUs and equal importance relative to spatial haptic media data PDUs. When the unit type field value in the RTP payload header of an RTP packet is 1 and the dependency field value is 1, then the corresponding PDUs in that PDU Set should be set with higher importance relative to silent haptic media data PDUs and lesser importance relative to initialization, spatial and independent temporal haptic media data PDUs.
A silent MIHS unit indicates that there is no effect during a time interval. The silent packets are of less importance relative to other packets as they are transmitted to the receiver to indicate that there is no haptic effect during a period of time. In a haptic media RTP packet containing the silent MIHS units, the unit type field of the RTP payload header is set with value 4. When the unit type field value in the RTP payload header of an RTP packet is 4, then the corresponding PDUs in that PDU Set should be set with lower importance relative to other haptic media data PDUs.
The MIHS Layer field present in the RTP payload header of a haptic RTP packet indicates the priority order of the MIHS unit included in the RTP payload, based on application-specific needs. The value zero for the MIHS Layer field corresponds to the highest priority. When the MIHS Layer field value in the RTP payload header of an RTP packet is zero, then the corresponding PDUs in that PDU Set shall be set with higher importance or with a lower PSI value compared to the PDUs or PDU Sets with MIHS Layer field value greater than zero. When a list of PDUs or PDU Sets contain the same MIHS Layer field value in the RTP payload header, then the PSI value of such PDUs or PDU Sets are set based on the Unit Type field and the dependency field values present in the RTP payload header.
10.4.2 IP packet filtering p. 39
The TS 29.514 specification defines various media types that can be carried by an application flow. The ApplicationFlowDescription data type lists the types of the media content present in a specific application flow.
For haptics media, the definition of MediaType present in clause 5.6.3.3 of TS 29.514 can be extended as below.
| Enumeration value | Description | Applicability |
|---|---|---|
| AUDIO | The type of media is audio. | |
| VIDEO | The type of media is video. | |
| DATA | The type of media is data. | |
| APPLICATION | The type of media is application data. | |
| CONTROL | The type of media is control. | |
| TEXT | The type of media is text. | |
| MESSAGE | The type of media is message | |
| HAPTICS | The type of media is haptics | |
| OTHER | Other type of media. |
When haptics media content is transported using RTP packets, the media name in the "m=" line of the SDP message is set to haptics as defined in RTP Payload format for Haptics specification.
An example of media representation corresponding to the hmpg RTP payload in SDP is as follows:
The Payload Type of the haptics media is identified by parsing the m=haptics line in the SDP message. In the above SDP example, the Payload Type value for haptics media is 115.
When an application data flow contains haptics media and other media content, the Payload Type (PT) values in the RTP header of RTP packets may be parsed to detect the PDUs or PDU Sets belonging to haptics media.
m=haptics 43291 UDP/TLS/RTP/SAVPF 115 a=rtpmap:115 hmpg/8000 a=fmtp:115 hmpg-profile=1;hmpg-lvl=1;hmpg-ver=2023
