Content for TS 26.114 Word version: 18.5.0

0… 3… 4… 5… 6… 6.2.3… 6.2.5… 6.2.7… 6.2.10… 7… 7.5… 8… 9… 10… 10.2.1.6… 10.2.2… 10.3… 10.4… 11… 12… 12.3… 12.7… 13a… 16… 16.5… 17… 18… 19… A… A.3… A.4… A.5… A.10… A.14… A.15… B… C… C.1.3… C.1.3.5 C.2… D E… E.18… E.31… G… K… L… M… N… O… P… P.3 Q… R… S… T… U… V… W… X… Y… Y.6… Y.6.4… Y.6.5… Y.7…

Y.6.4 Overlays Y.6.4.1 General Y.6.4.2 Visual Media Y.6.4.3 Overlay Configuration Y.6.4.3.1 General Y.6.4.3.2 Sphere-relative Overlay Configuration Y.6.4.3.3 Viewport-relative Overlay Configuration Y.6.4.3.4 Overlay info parameter Y.6.4.3.5 Additional Overlay Configuration Y.6.4.4 Captured Content Replacement
...

ITT4RT clients supporting the 'Overlay' feature may define an overlay source and overlay configuration in the SDP. An ITT4RT client that supports overlays shall support a video or image stream indicated by a media line in the SDP as the source of an overlay.

Y.6.4.2 Visual Media p. 473

Any visual media that is defined with the 'm=video' line, includes the attribute mid and does not have the attribute "a=3gpp_360video" in the SDP may be rendered as an overlay by an ITT4RT-Rx client. An ITT4RT client shall include the attribute mid in the overlay media description. If an overlay is to be associated with a particular overlay configuration, the mid shall be used to associate the overlay media description to the the overlay configuration, which is later described in clause Y.6.4.3.

Visual media may consist of an encoded video bitstream or HEVC encoded images/image sequences, both of which are defined with the 'm=video' line in SDP. ITT4RT clients that support images shall use the "a=imageseq" attribute as defined in clause 6.2.11.

Y.6.4.3 Overlay Configuration p. 473

Y.6.4.3.1 General p. 473

ITT4RT clients may support the following types of rendering for overlays, as defined in the OMAF specification [179] clause 7.14.

viewport-relative overlay, specifying that the overlay is displayed on a rectangular area at an indicated position relative to the top-left corner of the viewport;
[sphere-relative projected omnidirectional overlay, specifying that the overlay is displayed on a sphere surface at an indicated position within or on the unit sphere];
sphere-relative 2D overlay, specifying that the overlay is displayed on a plane at an indicated position within the unit sphere.

An ITT4RT client supporting overlays may include in its SDP media description the attribute 3gpp_overlay to define one or more parameters for configuring the rendering properties of an overlay. The 3gpp_overlay attribute has the following syntax:

att-field = "3gpp_overlay"
att-value = [free_overlay SP] overlay_id SP type SP overlay_config
 [SP overlay_info] [SP overlay_overlap_flag]
free_overlay = "free_ovelay=" BIT
overlay_id = token
 ; token is defined in RFC 4566
type = BIT
overlay_config = (sph_rel_overlay_config / vp_rel_overlay_config)
overlay_info = "overlay_info=" 5BIT
overlay_overlap_flag = "overlap=" BIT
;sub-rules for sphere_relative_overlay_config
sph_rel_overlay_config = "[" Overlay_azimuth "," Overlay_elevation "," Overlay_tilt 
 "," Overlay_azimuth_range "," Overlay_elevation_range
 "," Overlay_rot_yaw "," Overlay_rot_pitch "," Overlay_rot_roll
 "," region_depth_minus1 "," timeline_change_flag "]"
Overlay_azimuth = degminus180to180
Overlay_elevation = degminus90to90
Overlay_tilt = degminus180to180
Overlay_azimuth_range = deg0to360 
Overlay_elevation_range = deg0to180 
Over_rot_yaw = degminus180to180
Over_rot_pitch = degminus90to90
Over_rot_roll = degminus180to180
region_depth_minus1 = pctin2tominus16
timeline_change_flag = BIT
;sub-rules for viewport_relative_overlay_config
vp_rel_overlay_config = "[" Overlay_rect_left_percent "," Overlay_rect_top_percent 
 "," Overlay_rect_width_percent "," Overlay_rect_height_percent
 "," Relative_disparity_flag "," Disparity
 "," media_alignment "," layering_order "," opacity
 "," overlay_priority "]"
Overlay_rect_left_percent = pctIn2ToMinus16
Overlay_rect_top_percent = pctIn2ToMinus16
Overlay_rect_width_percent = pctIn2ToMinus16
Overlay_rect_height_percent = pctIn2ToMinus16
Relative_disparith_flag = BIT
Disparity = (Disparity_in_percent / Disparity_in_pixels)
Disparity_in_percent = pctIn2ToMinus16
Disparity_in_pixels = integer
media_alignment = "0"/pos-integer
layering_order = pos-integer
opacity = "0"/pos-integer
overlay_priority = pos-integer
;generic sub-rules
integer = "0"/["-"]pos-integer
pctin2tominus16 = "0"
 / POS-DIGIT *4DIGIT
 / "6" ("0"/"1"/"2"/"3"/"4") 3DIGIT
 / "65" ("0"/"1"/"2"/"3"/"4") 2DIGIT
 / "655" ("0"/"1"/"2") 1DIGIT
 / "6553" ("0"/"1"/"2"/"3"/"4"/"5")
 ; 0 to 65 535, inclusive
; pos-integer, degminus180to180, degminus90to90, deg0to360 and deg0to180 are from Y.6.2.1

The 3gpp_overlay attribute is included as part of the 360-degree video media description. More than one 3gpp_overlay attribute may exist as part of the media description if more than one overlay is to be configured. The overlay_id in 3gpp_overlay is set to the mid of the media description of the overlay source for which the configuration is provided. An ITT4RT-Tx client may include more than one 3gpp_overlay attributes with the same non-zero overlay_id (associated with an overlay source) in an SDP offer. The overlay configurations are listed in order of preference. An ITT4RT-Rx client that receives an SDP offer with multiple 3gpp_overlay attributes with the same non-zero overlay_id shall reply with only one acceptable 3gpp_overlay line for one acceptable overlay source. If an overlay source is rejected, all 3gpp_overlay attribute lines with the overlay_id associated with that source are excluded in the SDP answer.

An ITT4RT-Tx client may set the free_overlay flag to indicate that no media source is associated with this overlay configuration by the ITT4RT-Tx client and the corresponding overlay region in the video may be used by the ITT4RT-Rx client to overlay any media including external media sources (e.g., a text notification from an external application, an external video overlay or a source describing an occlude-free region that any overlay on top would not result in any significant loss of information). The overlay_id in 3gpp_overlay in case of overlays with the free_overlay flag with a value of 1 shall not be equal to any mid in the SDP media description. In case of overlay_id which is not associated with an overlay source (to identify an external source overlay and/or an occlude-free region), the free_overlay flag shall be included with a value equal to 1. If the free_overlay flag is not present or the free_overlay flag has a value of 0, the overlay_id shall be a value which corresponds to at least one mid of the media description of the overlay source. An ITT4RT-Tx client may include more than one 3gpp_overlay attributes with the same non-zero overlay_id and with the free_overlay flag set to 1 in an SDP offer. The implementation details for use of overlay region with the free_overlay flag set to 1 is left to the application.

An ITT4RT-Rx client that sends an SDP offer to receive 360-degree media and overlay source media shall not include the 3gpp_overlay attribute. An ITT4RT-Tx client that receives such an offer may initiate an SDP renegotiation offer to configure the overlays for the overlay sources as indicated by the itt4rt_group attribute.

An ITT4RT client that receives an SDP offer including overlay sources without the 3gpp_overlay attribute may decline the offer and send a renegotiation offer with the 3gpp_overlay attribute.

The type shall have the value '0' for viewport-relative overlays and '1' for sphere-relative overlays. Depending on the value of type, the 3gpp_overlay attribute may further include the corresponding configuration information sph_rel_overlay_config (type = '1') or the vp_rel_overlay_config (type = '0').

A set of flags describing interactivity of the overlay may be included in the optional overlay_info parameter defined in Annex Y.6.4.3.4.

Y.6.4.3.2 Sphere-relative Overlay Configuration p. 475

An ITT4RT client supporting the 3gpp_overlay attribute to configure a sphere-relative overlay shall set parameter type = '1' and additionally include the parameter sph_rel_overlay_config defined as follows:

sph_rel_overlay_config =
 Overlay_azimuth,
 Overlay_elevation,
 Overlay_tilt,
 Overlay_azimuth_range,
 Overlay_elevation_range,
 Overlay_rot_yaw,
 Overlay_rot_pitch,
 Overlay_rot_roll,
 region_depth_minus1,
 timeline_change_flag

Overlay_azimuth: Specifies the azimuth angle of the centre of the overlay region on the unit sphere in units of 2−16 degrees relative to the global coordinate axes.
Overlay_elevation: Specifies the elevation angle of the centre of the overlay region on the unit sphere in units of 2−16 degrees relative to the global coordinate axes.
Overlay_tilt: Specifies the tilt angle of the offered overlay region, in units of 2−16 degrees, relative to the global coordinate axes.
Overlay_azimuth_range: Specifies the azimuth range of the region corresponding to the 2D plane on which the overlay is rendered through the centre point of the overlay region in units of 2−16 degrees.
Overlay_elevation_range: Specifies the elevation range of the offered region corresponding to the 2D plane on which the overlay is rendered through the centre point of the overlay region in units of 2−16 degrees.
Overlay_rot_yaw, Overlay_rot_pitch, and Overlay_rot_roll: specify the rotation of the 2D plane on which the overlay is rendered. Prior to rendering the 2D plane, it may be rotated as specified by overlay_rot_yaw, overlay_rot_pitch and overlay_rot_yaw and placed on a certain distance as specified by region_depth_minus1. The rotations are relative to the coordinate system as specified in clause 5.1 of ISO/IEC 23090-2 in which the origin of the coordinate system is in the centre of the overlay region, the X axis is towards the origin of the global coordinate axes, the Y axis is towards the point on the plane that corresponds to cAzimuth1 in Figure 7-4 of ISO/IEC 23090-2, and the Z axis is towards the point on the plane that corresponds to cElevation2 in Figure 7-4 of ISO/IEC 23090-2. overlay_rot_yaw expresses a rotation around the Z axis, overlay_rot_pitch rotates around the Y axis, and overlay_rot_roll rotates around the X axis. Rotations are extrinsic, i.e., around X, Y, and Z fixed reference axes. The angles increase clockwise when looking from the origin towards the positive end of an axis. The rotations are applied starting from overlay_rot_yaw, followed by overlay_rot_pitch, and ending with overlay_rot_roll.
region_depth_minus1: indicates the depth (z-value) of the region on which the overlay is to be rendered. The depth value is the norm of the normal vector of the overlay region. region_depth_minus1 + 1 specifies the depth value relative to a unit sphere in units of 2−16.
timeline_change_flag equal to '1': specifies that the overlay content playback shall pause if the overlay is not in the user's current viewport, and when the overlay is back in the user's viewport the overlay content playback shall resume with the global presentation timeline of the content. The content in the intermediate interval is skipped. timeline_change_flag equal to '0' specifies that the overlay content playback shall pause if the overlay is not in the user's current viewport, and when the overlay is back in the user's viewport the overlay content playback resumes from the paused sample. This prevents loss of any content due to the overlay being away from the user's current viewport.

Y.6.4.3.3 Viewport-relative Overlay Configuration p. 476

An ITT4RT client supporting the 3gpp_overlay attribute to configure a viewport-relative overlay shall set parameter type = '0' and additionally include the parameter vp_rel_overlay_config defined as follows:

vp_rel_overlay_config =
[Overlay_rect_left_percent, Overlay_rect_top_percent, Overlay_rect_width_percent,
Overlay_rect_height_percent, Relative_disparity_flag, Disparity, media_alignment, layering_order, 
opacity, overlay_priority]

Overlay_rect_left_percent: Specifies the x-coordinate of the left corner of the rectangular region of the overlay to be rendered on the viewport in per cents relative to the width of the viewport. The values are indicated in units of 2^-16 in the range of 0 (indicating 0%), inclusive, up to but excluding 65536 (that indicates 100%).
Overlay_rect_top_percent: Specifies the y-coordinate of the top corner of the rectangular region of the overlay to be rendered on the viewport in per cents relative to the height of the viewport. The values are indicated in units of 2^-16 in the range of 0 (indicating 0%), inclusive, up to but excluding 65536 (that indicates 100%).
Overlay_rect_width_percent: Specifies the width of the rectangular region of the overlay to be rendered on the viewport in per cents relative to the width of the viewport. The values are indicated in units of 2^-16 in the range of 0 (indicating 0%), inclusive, up to but excluding 65536 (that indicates 100%).
Overlay_rect_height_percent: Specifies the height of the rectangular region of the overlay to be rendered on the viewport in per cents relative to the height of the viewport. The values are indicated in units of 2^-16 in the range of 0 (indicating 0%), inclusive, up to but excluding 65536 (that indicates 100%).

NOTE 1:
The size of overlay region over the viewport changes according to the viewport resolution and aspect ratio. However, the aspect ratio of the overlaid media is not intended to be changed.
Relative_disparity_flag: Indicates whether the disparity is provided as Disparity_in_percent which is a percentage value of the width of the display window for one view (when the value is equal to 1) or as Disparity_in_pixels which is a number of pixels (when the value is equal to 0).
Disparity_in_percent: Specifies the disparity, in units of 2^-16, as a fraction of the width of the display window for one view. The value may be negative, in which case the displacement direction is reversed. This value is used to displace the region to the left on the left eye view and to the right on the right eye view. This applies for the case when there is a monoscopic overlay and stereoscopic background visual media.
Disparity_in_pixels: Indicates the disparity in pixels. The value may be negative, in which case the displacement direction is reversed. This value is used to displace the region to the left on the left eye view and to the right on the right eye view. This applies for the case when there is a monoscopic overlay and stereoscopic background visual media.
Media_alignment: Specifies the default intended scaling of the overlay source depending on the dimensions of the specified rectangular region and the intended placement of the scaled overlay source relative to the specified rectangular region.
Layering_order: Indicates the default layering order among the overlays that are relative to the viewport, and separately among each set of overlays that have the same depth. Viewport-relative overlays are overlaid on top of the viewport in descending order of layering_order, i.e., an overlay with a smaller layering_order value shall be in front of an overlay with a greater layering_order value. The layering order for overlays of the 360-degree video should be decided by the ITT4RT-Tx client.
Opacity: Indicates an integer value that specifies the default opacity that is to be applied for the overlay and assigned by the ITT4RT-Tx client. Value 0 is fully transparent, and value 100 is fully opaque with a linear weighting between the two extremes. Values greater than 100 are reserved.
Overlay_priority: Indicates which overlay should be prioritized in the case the ITT4RT-Rx client does not have enough decoding capacity to decode all overlays. A lower overlay_priority indicates higher priority. The value of overlay_priority, when present, shall be equal to 0 for overlays that are essential for displaying. More than one overlay may have the same overlay_priority and an ITT4RT-Rx client that does not support all overlays with the same priority may choose any subset of these.

Y.6.4.3.4 Overlay info parameter p. 477

The parameter overlay_info is a bit field consisting of flags describing the type of overlay interactivity recommended:

overlay_info= 'overlay_info='b4b3b2b1b0

b0 = '0': changing the position of the overlay is not recommended, overlay position is fixed
b0 = '1': changing the position of the overlay is allowed, ITT4RT-Rx client may change position
b1 = '0': switching the overlay on/off is not recommended
b1 = '1': switching the overlay on/off is allowed, ITT4RT-Rx client may switch overlay off
b2 = '0': rotating the overlay is not recommended
b2 = '1': rotating the overlay is allowed
b3 = '0': resizing the overlay is not recommended
b3 = '1': resizing the overlay is allowed
b4 = '0': changing the opacity of the overlay is not recommended
b4 = '1': changing the opacity of the overlay is allowed

Y.6.4.3.5 Additional Overlay Configuration p. 477

An ITT4RT client supporting the 3gpp_overlay attribute to configure a sphere-relative overlay or viewport-relative overlay may include the following additional parameter for overlay support:

overlay_overlap_flag: Indicates if the ITT4RT-Rx client is allowed to overlap overlays from the ITT4RT-Tx client. If set to 1, the ITT4RT-Rx client may overlap overlays shared by the ITT4RT-Tx client.

Y.6.4.4 Captured Content Replacement p. 477

To prevent the degradation of presentation material (e.g., slides, screen share, video, notes) that may be captured from a display (screen or projector) with a 360-degree camera, the captured content in the 360-degree video can be replaced with the original presentation material. Such replacement implies decoding the content (360-degree video and presentation material), identifying the position of the presentation material in the 360-degree video, replacing the captured presentation content at the display coordinates in the 360-degree video, and finally encoding the new 360-degree video (i.e., with the same encoding parameters as the original 360-degree video). The replacement could either be performed in the ITT4RT-Tx client in terminal which is sending the 360-degree video or in the ITT4RT-MRF.

When replacement is to be performed, the availability of the original presentation content must be signalled by the source of the content to the client performing the replacement (that is, the ITT4RT-Tx client in terminal or the ITT4RT-MRF) using the SDP attribute "a=content:slides" (RFC 4796) which may include different content, for example slides, screen share, video, notes. The client performing the replacement shall determine an appropriate configuration for performing the content replacement in the 360-degree video, unless overlay parameters are given by the source of the original presentation content (e.g., configuration in terms of sphere-relative overlay coordinates as defined in Annex Y.6.4.3.2).

When the SDP negotiation is initiated by the ITT4RT-Tx client in terminal, the ITT4RT-Tx client in terminal shall include the attribute "a=3gpp_360video_replacement" in its SDP offer to indicate that the content captured in the 360-degree video can be replaced. If the ITT4RT-MRF supports content replacement and receives an SDP offer with the attribute "a=3gpp_360video_replacement", then the ITT4RT-MRF shall include the attribute "a=3gpp_360video_replacement" in its SDP answer and shall perform content replacement.

If the ITT4RT-Tx client in terminal includes the attribute "a=3gpp_360video_replacement" in its SDP offer but does not receive the attribute in the SDP answer (that is, replacement is not supported in the ITT4RT-MRF) then the ITT4RT-Tx client in terminal may send the original presentation content using a different process than ITT4RT-MRF replacement (e.g., the presentation can be sent as an overlay as defined in Annex Y.6.4, or inserted into the 360-degree video by the ITT4RT-Tx client in terminal as described above).

If the ITT4RT-MRF does not receive the attribute "a=3gpp_360video_replacement" in an SDP offer, it shall not perform any replacement and will not include the attribute in its SDP answer.

When replacement is to be performed by the ITT4RT-MRF and the SDP negotiation is initiated by the ITT4RT-MRF, the offer sent by the ITT4RT-MRF to the ITT4RT-Tx client in terminal shall include the attribute "a=3gpp_360video_replacement". If the ITT4RT-Tx client in terminal accepts the offer by the MRF to perform replacement, the ITT4RT-Tx client in terminal shall include the attribute "a=3gpp_360video_replacement" in the SDP answer and the ITT4RT-MRF shall perform content replacement.

If the ITT4RT-MRF does not receive the attribute "a=3gpp_360video_replacement" in the SDP answer of the ITT4RT-Tx in terminal (i.e., the content captured in the 360-degree video cannot be replaced), the ITT4RT-MRF shall not perform any replacement.

If the ITT4RT-MRF does not support content replacement, it shall not include the attribute "a=3gpp_360video_replacement" in an SDP offer, it will not perform any replacement, and the ITT4RT-Tx client in terminal may send the original presentation content using a different process (e.g., the presentation can be sent as an overlay as defined in Annex Y.6.4, or inserted into the 360-degree video by the ITT4RT-Tx client in terminal as described above). In the case that the ITT4RT-MRF does not send the attribute "a=3gpp_360video_replacement" in an offer, the ITT4RT-Tx client in terminal shall not send the attribute "a=3gpp_360video_replacement" in an answer.

After an accepted offer/answer between ITT4RT-Tx in terminal and ITT4RT-MRF with both offer and answer including the attribute "a=3gpp_360video_replacement", the ITT4RT-MRF shall perform content replacement once the original presentation content is available from the source of the content and the replacement configuration is determined.

If the replacement configuration of the content is analysed and determined by the ITT4RT-Tx in terminal, the client shall include the configuration as sphere-relative overlay coordinates (defined in Annex Y.6.4.3.2) in the SDP offer/answer while negotiating the stream with the ITT4RT-MRF. If the sphere-relative overlay coordinates are not signalled in the SDP offer/answer by the ITT4RT-Tx, the ITT4RT-MRF shall analyse and determine an appropriate configuration for performing the content replacement in the 360-degree video.

The ABNF syntax for the replacement attribute is as follows:

att-field = "3gpp_360video_replacement"
att-value = [sph_rel_overlay_config]