tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Glossaries     Architecture     IMS     UICC    |    search     info

RFC 6184

 
 
 

RTP Payload Format for H.264 Video

Part 2 of 4, p. 10 to 39
Prev RFC Part       Next RFC Part

 


prevText      Top      Up      ToC       Page 10 
5.  RTP Payload Format

5.1.  RTP Header Usage

   The format of the RTP header is specified in RFC 3550 [5] and
   reprinted in Figure 1 for convenience.  This payload format uses the
   fields of the header in a manner consistent with that specification.

   When one NAL unit is encapsulated per RTP packet, the RECOMMENDED RTP
   payload format is specified in Section 5.6.  The RTP payload (and the
   settings for some RTP header bits) for aggregation packets and
   fragmentation units are specified in Sections 5.7.2 and 5.8,
   respectively.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |V=2|P|X|  CC   |M|     PT      |       sequence number         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                           timestamp                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           synchronization source (SSRC) identifier            |
      +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
      |            contributing source (CSRC) identifiers             |
      |                             ....                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 1.  RTP header according to RFC 3550

   The RTP header information to be set according to this RTP payload
   format is set as follows:

   Marker bit (M): 1 bit
      Set for the very last packet of the access unit indicated by the
      RTP timestamp, in line with the normal use of the M bit in video

Top      Up      ToC       Page 11 
      formats, to allow an efficient playout buffer handling.  For
      aggregation packets (STAP and MTAP), the marker bit in the RTP
      header MUST be set to the value that the marker bit of the last
      NAL unit of the aggregation packet would have been if it were
      transported in its own RTP packet.  Decoders MAY use this bit as
      an early indication of the last packet of an access unit but MUST
      NOT rely on this property.

         Informative note: Only one M bit is associated with an
         aggregation packet carrying multiple NAL units.  Thus, if a
         gateway has re-packetized an aggregation packet into several
         packets, it cannot reliably set the M bit of those packets.

   Payload type (PT): 7 bits
      The assignment of an RTP payload type for this new packet format
      is outside the scope of this document and will not be specified
      here.  The assignment of a payload type has to be performed either
      through the profile used or in a dynamic way.

   Sequence number (SN): 16 bits
      Set and used in accordance with RFC 3550.  For the single NALU and
      non-interleaved packetization mode, the sequence number is used to
      determine decoding order for the NALU.

   Timestamp: 32 bits
      The RTP timestamp is set to the sampling timestamp of the content.
      A 90 kHz clock rate MUST be used.

      If the NAL unit has no timing properties of its own (e.g.,
      parameter set and SEI NAL units), the RTP timestamp is set to the
      RTP timestamp of the primary coded picture of the access unit in
      which the NAL unit is included, according to Section 7.4.1.2 of
      [1].

      The setting of the RTP timestamp for MTAPs is defined in Section
      5.7.2.

      Receivers SHOULD ignore any picture timing SEI messages included
      in access units that have only one display timestamp.  Instead,
      receivers SHOULD use the RTP timestamp for synchronizing the
      display process.

      If one access unit has more than one display timestamp carried in
      a picture timing SEI message, then the information in the SEI
      message SHOULD be treated as relative to the RTP timestamp, with
      the earliest event occurring at the time given by the RTP
      timestamp and subsequent events later, as given by the difference
      in picture time values carried in the picture timing SEI message.

Top      Up      ToC       Page 12 
      Let tSEI1, tSEI2, ..., tSEIn be the display timestamps carried in
      the SEI message of an access unit, where tSEI1 is the earliest of
      all such timestamps.  Let tmadjst() be a function that adjusts the
      SEI messages time scale to a 90-kHz time scale.  Let TS be the RTP
      timestamp.  Then, the display time for the event associated with
      tSEI1 is TS.  The display time for the event with tSEIx, where x
      is [2..n], is TS + tmadjst (tSEIx - tSEI1).

         Informative note: Displaying coded frames as fields is needed
         commonly in an operation known as 3:2 pulldown, in which film
         content that consists of coded frames is displayed on a display
         using interlaced scanning.  The picture timing SEI message
         enables carriage of multiple timestamps for the same coded
         picture, and therefore the 3:2 pulldown process is perfectly
         controlled.  The picture timing SEI message mechanism is
         necessary because only one timestamp per coded frame can be
         conveyed in the RTP timestamp.

5.2.  Payload Structures

   The payload format defines three different basic payload structures.
   A receiver can identify the payload structure by the first byte of
   the RTP packet payload, which co-serves as the RTP payload header
   and, in some cases, as the first byte of the payload.  This byte is
   always structured as a NAL unit header.  The NAL unit type field
   indicates which structure is present.  The possible structures are as
   follows.

   Single NAL Unit Packet: Contains only a single NAL unit in the
   payload.  The NAL header type field is equal to the original NAL unit
   type, i.e., in the range of 1 to 23, inclusive.  Specified in Section
   5.6.

   Aggregation Packet: Packet type used to aggregate multiple NAL units
   into a single RTP payload.  This packet exists in four versions, the
   Single-Time Aggregation Packet type A (STAP-A), the Single-Time
   Aggregation Packet type B (STAP-B), Multi-Time Aggregation Packet
   (MTAP) with 16-bit offset (MTAP16), and Multi-Time Aggregation Packet
   (MTAP) with 24-bit offset (MTAP24).  The NAL unit type numbers
   assigned for STAP-A, STAP-B, MTAP16, and MTAP24 are 24, 25, 26, and
   27, respectively.  Specified in Section 5.7.

   Fragmentation Unit: Used to fragment a single NAL unit over multiple
   RTP packets.  Exists with two versions, FU-A and FU-B, identified
   with the NAL unit type numbers 28 and 29, respectively.  Specified in
   Section 5.8.

Top      Up      ToC       Page 13 
      Informative note: This specification does not limit the size of
      NAL units encapsulated in single NAL unit packets and
      fragmentation units.  The maximum size of a NAL unit encapsulated
      in any aggregation packet is 65535 bytes.

   Table 1 summarizes NAL unit types and the corresponding RTP packet
   types when each of these NAL units is directly used as a packet
   payload, and where the types are described in this memo.

      Table 1.  Summary of NAL unit types and the corresponding packet
                types

      NAL Unit  Packet    Packet Type Name               Section
      Type      Type
      -------------------------------------------------------------
      0        reserved                                     -
      1-23     NAL unit  Single NAL unit packet             5.6
      24       STAP-A    Single-time aggregation packet     5.7.1
      25       STAP-B    Single-time aggregation packet     5.7.1
      26       MTAP16    Multi-time aggregation packet      5.7.2
      27       MTAP24    Multi-time aggregation packet      5.7.2
      28       FU-A      Fragmentation unit                 5.8
      29       FU-B      Fragmentation unit                 5.8
      30-31    reserved                                     -

5.3.  NAL Unit Header Usage

   The structure and semantics of the NAL unit header were introduced in
   Section 1.3.  For convenience, the format of the NAL unit header is
   reprinted below:

      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |F|NRI|  Type   |
      +---------------+

   This section specifies the semantics of F and NRI according to this
   specification.

   F:    1 bit
         forbidden_zero_bit.  A value of 0 indicates that the NAL unit
         type octet and payload should not contain bit errors or other
         syntax violations.  A value of 1 indicates that the NAL unit
         type octet and payload may contain bit errors or other syntax
         violations.

Top      Up      ToC       Page 14 
         MANEs SHOULD set the F bit to indicate detected bit errors in
         the NAL unit.  The H.264 specification requires that the F bit
         be equal to 0.  When the F bit is set, the decoder is advised
         that bit errors or any other syntax violations may be present
         in the payload or in the NAL unit type octet.  The simplest
         decoder reaction to a NAL unit in which the F bit is equal to 1
         is to discard such a NAL unit and to conceal the lost data in
         the discarded NAL unit.

   NRI:  2 bits
         nal_ref_idc.  The semantics of value 00 and a non-zero value
         remain unchanged from the H.264 specification.  In other words,
         a value of 00 indicates that the content of the NAL unit is not
         used to reconstruct reference pictures for inter picture
         prediction.  Such NAL units can be discarded without risking
         the integrity of the reference pictures.  Values greater than
         00 indicate that the decoding of the NAL unit is required to
         maintain the integrity of the reference pictures.

         In addition to the specification above, according to this RTP
         payload specification, values of NRI indicate the relative
         transport priority, as determined by the encoder.  MANEs can
         use this information to protect more important NAL units better
         than they do less important NAL units.  The highest transport
         priority is 11, followed by 10, and then by 01; finally, 00 is
         the lowest.

            Informative note: Any non-zero value of NRI is handled
            identically in H.264 decoders.  Therefore, receivers need
            not manipulate the value of NRI when passing NAL units to
            the decoder.

         An H.264 encoder MUST set the value of NRI according to the
         H.264 specification (Subclause 7.4.1) when the value of
         nal_unit_type is in the range of 1 to 12, inclusive.  In
         particular, the H.264 specification requires that the value of
         NRI SHALL be equal to 0 for all NAL units having nal_unit_type
         equal to 6, 9, 10, 11, or 12.

         For NAL units having nal_unit_type equal to 7 or 8 (indicating
         a sequence parameter set or a picture parameter set,
         respectively), an H.264 encoder SHOULD set the value of NRI to
         11 (in binary format).  For coded slice NAL units of a primary
         coded picture having nal_unit_type equal to 5 (indicating a
         coded slice belonging to an IDR picture), an H.264 encoder
         SHOULD set the value of NRI to 11 (in binary format).

Top      Up      ToC       Page 15 
         For a mapping of the remaining nal_unit_types to NRI values,
         the following example MAY be used and has been shown to be
         efficient in a certain environment [14].  Other mappings MAY
         also be desirable, depending on the application and the H.264
         profile in use.

            Informative note: Data partitioning is not available in
            certain profiles, e.g., in the Main or Baseline profiles.
            Consequently, the NAL unit types 2, 3, and 4 can occur only
            if the video bitstream conforms to a profile in which data
            partitioning is allowed and not in streams that conform to
            the Main or Baseline profiles.

      Table 2.  Example of NRI values for coded slices and coded slice
                data partitions of primary coded reference pictures

       NAL Unit Type     Content of NAL Unit              NRI (binary)
       ----------------------------------------------------------------
        1              non-IDR coded slice                         10
        2              Coded slice data partition A                10
        3              Coded slice data partition B                01
        4              Coded slice data partition C                01

            Informative note: As mentioned before, the NRI value of non-
            reference pictures is 00 as mandated by H.264.

         An H.264 encoder SHOULD set the value of NRI for coded slice
         and coded slice data partition NAL units of redundant coded
         reference pictures equal to 01 (in binary format).

         Definitions of the values for NRI for NAL unit types 24 to 29,
         inclusive, are given in Sections 5.7 and 5.8 of this memo.

         No recommendation for the value of NRI is given for NAL units
         having nal_unit_type in the range of 13 to 23, inclusive,
         because these values are reserved for ITU-T and ISO/IEC.  No
         recommendation for the value of NRI is given for NAL units
         having nal_unit_type equal to 0 or in the range of 30 to 31,
         inclusive, as the semantics of these values are not specified
         in this memo.

Top      Up      ToC       Page 16 
5.4.  Packetization Modes

   This memo specifies three cases of packetization modes:

   o  Single NAL unit mode

   o  Non-interleaved mode

   o  Interleaved mode

   The single NAL unit mode is targeted for conversational systems that
   comply with ITU-T Recommendation H.241 [3]  (see Section 12.1).  The
   non-interleaved mode is targeted for conversational systems that may
   not comply with ITU-T Recommendation H.241.  In the non-interleaved
   mode, NAL units are transmitted in NAL unit decoding order.  The
   interleaved mode is targeted for systems that do not require very low
   end-to-end latency.  The interleaved mode allows transmission of NAL
   units out of NAL unit decoding order.

   The packetization mode in use MAY be signaled by the value of the
   OPTIONAL packetization-mode media type parameter.  The used
   packetization mode governs which NAL unit types are allowed in RTP
   payloads.  Table 3 summarizes the allowed packet payload types for
   each packetization mode.  Packetization modes are explained in more
   detail in Section 6.

      Table 3.  Summary of allowed NAL unit types for each packetization
                mode (yes = allowed, no = disallowed, ig = ignore)

      Payload Packet    Single NAL    Non-Interleaved    Interleaved
      Type    Type      Unit Mode           Mode             Mode
      -------------------------------------------------------------
      0      reserved      ig               ig               ig
      1-23   NAL unit     yes              yes               no
      24     STAP-A        no              yes               no
      25     STAP-B        no               no              yes
      26     MTAP16        no               no              yes
      27     MTAP24        no               no              yes
      28     FU-A          no              yes              yes
      29     FU-B          no               no              yes
      30-31  reserved      ig               ig               ig

   Some NAL unit or payload type values (indicated as reserved in Table
   3) are reserved for future extensions.  NAL units of those types
   SHOULD NOT be sent by a sender (direct as packet payloads, as
   aggregation units in aggregation packets, or as fragmented units in
   FU packets) and MUST be ignored by a receiver.  For example, the
   payload types 1-23, with the associated packet type "NAL unit", are

Top      Up      ToC       Page 17 
   allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode" but
   disallowed in "Interleaved Mode".  However, NAL units of NAL unit
   types 1-23 can be used in "Interleaved Mode" as aggregation units in
   STAP-B, MTAP16, and MTAP24 packets as well as fragmented units in FU-
   A and FU-B packets.  Similarly, NAL units of NAL unit types 1-23 can
   also be used in the "Non-Interleaved Mode" as aggregation units in
   STAP-A packets or fragmented units in FU-A packets, in addition to
   being directly used as packet payloads.

5.5.  Decoding Order Number (DON)

   In the interleaved packetization mode, the transmission order of NAL
   units is allowed to differ from the decoding order of the NAL units.
   Decoding order number (DON) is a field in the payload structure or a
   derived variable that indicates the NAL unit decoding order.
   Rationale and examples of use cases for transmission out of decoding
   order and for the use of DON are given in Section 13.

   The coupling of transmission and decoding order is controlled by the
   OPTIONAL sprop-interleaving-depth media type parameter as follows.
   When the value of the OPTIONAL sprop-interleaving-depth media type
   parameter is equal to 0 (explicitly or per default), the transmission
   order of NAL units MUST conform to the NAL unit decoding order.  When
   the value of the OPTIONAL sprop-interleaving-depth media type
   parameter is greater than 0:

   o  the order of NAL units in an MTAP16 and an MTAP24 is not required
      to be the NAL unit decoding order, and

   o  the order of NAL units generated by de-packetizing STAP-Bs, MTAPs,
      and FUs in two consecutive packets is not required to be the NAL
      unit decoding order.

   The RTP payload structures for a single NAL unit packet, an STAP-A,
   and an FU-A do not include DON.  STAP-B and FU-B structures include
   DON, and the structure of MTAPs enables derivation of DON, as
   specified in Section 5.7.2.

      Informative note: When an FU-A occurs in interleaved mode, it
      always follows an FU-B, which sets its DON.

      Informative note: If a transmitter wants to encapsulate a single
      NAL unit per packet and transmit packets out of their decoding
      order, STAP-B packet type can be used.

   In the single NAL unit packetization mode, the transmission order of
   NAL units, determined by the RTP sequence number, MUST be the same as
   their NAL unit decoding order.  In the non-interleaved packetization

Top      Up      ToC       Page 18 
   mode, the transmission order of NAL units in single NAL unit packets,
   STAP-As, and FU-As MUST be the same as their NAL unit decoding order.
   The NAL units within an STAP MUST appear in the NAL unit decoding
   order.  Thus, the decoding order is first provided through the
   implicit order within an STAP and then provided through the RTP
   sequence number for the order between STAPs, FUs, and single NAL unit
   packets.

   The signaling of the value of DON for NAL units carried in STAP-B,
   MTAP, and a series of fragmentation units starting with an FU-B is
   specified in Sections 5.7.1, 5.7.2, and 5.8, respectively.  The DON
   value of the first NAL unit in transmission order MAY be set to any
   value.  Values of DON are in the range of 0 to 65535, inclusive.
   After reaching the maximum value, the value of DON wraps around to 0.

   The decoding order of two NAL units contained in any STAP-B, MTAP, or
   a series of fragmentation units starting with an FU-B is determined
   as follows.  Let DON(i) be the decoding order number of the NAL unit
   having index i in the transmission order.  Function don_diff(m,n) is
   specified as follows:

      If DON(m) == DON(n), don_diff(m,n) = 0

      If (DON(m) < DON(n) and DON(n) - DON(m) < 32768),
      don_diff(m,n) = DON(n) - DON(m)

      If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768),
      don_diff(m,n) = 65536 - DON(m) + DON(n)

      If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768),
      don_diff(m,n) = - (DON(m) + 65536 - DON(n))

      If (DON(m) > DON(n) and DON(m) - DON(n) < 32768),
      don_diff(m,n) = - (DON(m) - DON(n))

   A positive value of don_diff(m,n) indicates that the NAL unit having
   transmission order index n follows, in decoding order, the NAL unit
   having transmission order index m.  When don_diff(m,n) is equal to 0,
   the NAL unit decoding order of the two NAL units can be in either
   order.  A negative value of don_diff(m,n) indicates that the NAL unit
   having transmission order index n precedes, in decoding order, the
   NAL unit having transmission order index m.

   Values of DON-related fields (DON, DONB, and DOND; see Section 5.7)
   MUST be such that the decoding order determined by the values of DON,
   as specified above, conforms to the NAL unit decoding order.

Top      Up      ToC       Page 19 
   If the order of two NAL units in NAL unit decoding order is switched
   and the new order does not conform to the NAL unit decoding order,
   the NAL units MUST NOT have the same value of DON.  If the order of
   two consecutive NAL units in the NAL unit stream is switched and the
   new order still conforms to the NAL unit decoding order, the NAL
   units MAY have the same value of DON.  For example, when arbitrary
   slice order is allowed by the video coding profile in use, all the
   coded slice NAL units of a coded picture are allowed to have the same
   value of DON.  Consequently, NAL units having the same value of DON
   can be decoded in any order, and two NAL units having a different
   value of DON should be passed to the decoder in the order specified
   above.  When two consecutive NAL units in the NAL unit decoding order
   have a different value of DON, the value of DON for the second NAL
   unit in decoding order SHOULD be the value of DON for the first,
   incremented by one.

   An example of the de-packetization process to recover the NAL unit
   decoding order is given in Section 7.

      Informative note: Receivers should not expect that the absolute
      difference of values of DON for two consecutive NAL units in the
      NAL unit decoding order will be equal to one, even in error-free
      transmission.  An increment by one is not required, as at the time
      of associating values of DON to NAL units, it may not be known
      whether all NAL units are delivered to the receiver.  For example,
      a gateway may not forward coded slice NAL units of non-reference
      pictures or SEI NAL units when there is a shortage of bitrate in
      the network to which the packets are forwarded.  In another
      example, a live broadcast is interrupted by pre-encoded content,
      such as commercials, from time to time.  The first intra picture
      of a pre-encoded clip is transmitted in advance to ensure that it
      is readily available in the receiver.  When transmitting the first
      intra picture, the originator does not exactly know how many NAL
      units will be encoded before the first intra picture of the pre-
      encoded clip follows in decoding order.  Thus, the values of DON
      for the NAL units of the first intra picture of the pre-encoded
      clip have to be estimated when they are transmitted, and gaps in
      values of DON may occur.

5.6.  Single NAL Unit Packet

   The single NAL unit packet defined here MUST contain only one NAL
   unit of the types defined in [1].  This means that neither an
   aggregation packet nor a fragmentation unit can be used within a
   single NAL unit packet.  A NAL unit stream composed by de-packetizing
   single NAL unit packets in RTP sequence number order MUST conform to
   the NAL unit decoding order.  The structure of the single NAL unit
   packet is shown in Figure 2.

Top      Up      ToC       Page 20 
      Informative note: The first byte of a NAL unit co-serves as the
      RTP payload header.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |F|NRI|  Type   |                                               |
    +-+-+-+-+-+-+-+-+                                               |
    |                                                               |
    |               Bytes 2..n of a single NAL unit                 |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 2.  RTP payload format for single NAL unit packet

5.7.  Aggregation Packets

   Aggregation packets are the NAL unit aggregation scheme of this
   payload specification.  The scheme is introduced to reflect the
   dramatically different MTU sizes of two key target networks: wireline
   IP networks (with an MTU size that is often limited by the Ethernet
   MTU size, roughly 1500 bytes) and IP-based or non-IP-based (e.g.,
   ITU-T H.324/M) wireless communication systems with preferred
   transmission unit sizes of 254 bytes or less.  To prevent media
   transcoding between the two worlds, and to avoid undesirable
   packetization overhead, a NAL unit aggregation scheme is introduced.

   Two types of aggregation packets are defined by this specification:

   o  Single-time aggregation packet (STAP): aggregates NAL units with
      identical NALU-times.  Two types of STAPs are defined, one without
      DON (STAP-A) and another including DON (STAP-B).

   o  Multi-time aggregation packet (MTAP): aggregates NAL units with
      potentially differing NALU-times.  Two different MTAPs are
      defined, differing in the length of the NAL unit timestamp offset.

   Each NAL unit to be carried in an aggregation packet is encapsulated
   in an aggregation unit.  Please see below for the four different
   aggregation units and their characteristics.

Top      Up      ToC       Page 21 
   The structure of the RTP payload format for aggregation packets is
   presented in Figure 3.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |F|NRI|  Type   |                                               |
    +-+-+-+-+-+-+-+-+                                               |
    |                                                               |
    |             one or more aggregation units                     |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 3.  RTP payload format for aggregation packets

   MTAPs and STAPs share the following packetization rules:

   o  The RTP timestamp MUST be set to the earliest of the NALU-times of
      all the NAL units to be aggregated.

   o  The type field of the NAL unit type octet MUST be set to the
      appropriate value, as indicated in Table 4.

   o  The F bit MUST be cleared if all F bits of the aggregated NAL
      units are zero; otherwise, it MUST be set.

   o  The value of NRI MUST be the maximum of all the NAL units carried
      in the aggregation packet.

                 Table 4.  Type field for STAPs and MTAPs

      Type   Packet    Timestamp offset   DON-related fields
                       field length       (DON, DONB, DOND)
                       (in bits)          present
      --------------------------------------------------------
      24     STAP-A       0                 no
      25     STAP-B       0                 yes
      26     MTAP16      16                 yes
      27     MTAP24      24                 yes

   The marker bit in the RTP header is set to the value that the marker
   bit of the last NAL unit of the aggregated packet would have if it
   were transported in its own RTP packet.

Top      Up      ToC       Page 22 
   The payload of an aggregation packet consists of one or more
   aggregation units.  See Sections 5.7.1 and 5.7.2 for the four
   different types of aggregation units.  An aggregation packet can
   carry as many aggregation units as necessary; however, the total
   amount of data in an aggregation packet obviously MUST fit into an IP
   packet, and the size SHOULD be chosen so that the resulting IP packet
   is smaller than the MTU size.  An aggregation packet MUST NOT contain
   fragmentation units, as specified in Section 5.8.  Aggregation
   packets MUST NOT be nested; that is, an aggregation packet MUST NOT
   contain another aggregation packet.

5.7.1.  Single-Time Aggregation Packet (STAP)

   A single-time aggregation packet (STAP) SHOULD be used whenever NAL
   units are aggregated that all share the same NALU-time.  The payload
   of an STAP-A does not include DON and consists of at least one
   single-time aggregation unit, as presented in Figure 4.  The payload
   of an STAP-B consists of a 16-bit unsigned decoding order number
   (DON) (in network byte order) followed by at least one single-time
   aggregation unit, as presented in Figure 5.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                    :                                               |
    +-+-+-+-+-+-+-+-+                                               |
    |                                                               |
    |                single-time aggregation units                  |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 4.  Payload format for STAP-A

Top      Up      ToC       Page 23 
     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                    :  decoding order number (DON)  |               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               |
    |                                                               |
    |                single-time aggregation units                  |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 5.  Payload format for STAP-B

   The DON field specifies the value of DON for the first NAL unit in an
   STAP-B in transmission order.  For each successive NAL unit in
   appearance order in an STAP-B, the value of DON is equal to (the
   value of DON of the previous NAL unit in the STAP-B + 1) % 65536, in
   which '%' stands for the modulo operation.

   A single-time aggregation unit consists of 16-bit unsigned size
   information (in network byte order) that indicates the size of the
   following NAL unit in bytes (excluding these two octets, but
   including the NAL unit type octet of the NAL unit), followed by the
   NAL unit itself, including its NAL unit type byte.  A single-time
   aggregation unit is byte aligned within the RTP payload, but it may
   not be aligned on a 32-bit word boundary.  Figure 6 presents the
   structure of the single-time aggregation unit.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                    :        NAL unit size          |               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               |
    |                                                               |
    |                           NAL unit                            |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 6.  Structure for single-time aggregation unit

Top      Up      ToC       Page 24 
   Figure 7 presents an example of an RTP packet that contains an STAP-
   A.  The STAP contains two single-time aggregation units, labeled as 1
   and 2 in the figure.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          RTP Header                           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |STAP-A NAL HDR |         NALU 1 Size           | NALU 1 HDR    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                         NALU 1 Data                           |
    :                                                               :
    +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |               | NALU 2 Size                   | NALU 2 HDR    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                         NALU 2 Data                           |
    :                                                               :
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 7.  An example of an RTP packet including an STAP-A
               containing two single-time aggregation units

Top      Up      ToC       Page 25 
   Figure 8 presents an example of an RTP packet that contains an STAP-
   B.  The STAP contains two single-time aggregation units, labeled as 1
   and 2 in the figure.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          RTP Header                           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |STAP-B NAL HDR | DON                           | NALU 1 Size   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | NALU 1 Size   | NALU 1 HDR    | NALU 1 Data                   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
    :                                                               :
    +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |               | NALU 2 Size                   | NALU 2 HDR    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                       NALU 2 Data                             |
    :                                                               :
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 8.  An example of an RTP packet including an STAP-B
               containing two single-time aggregation units

5.7.2.  Multi-Time Aggregation Packets (MTAPs)

   The NAL unit payload of MTAPs consists of a 16-bit unsigned decoding
   order number base (DONB) (in network byte order) and one or more
   multi-time aggregation units, as presented in Figure 9.  DONB MUST
   contain the value of DON for the first NAL unit in the NAL unit
   decoding order among the NAL units of the MTAP.

      Informative note: The first NAL unit in the NAL unit decoding
      order is not necessarily the first NAL unit in the order in which
      the NAL units are encapsulated in an MTAP.

Top      Up      ToC       Page 26 
     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                    :  decoding order number base   |               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               |
    |                                                               |
    |                 multi-time aggregation units                  |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 9.  NAL unit payload format for MTAPs

   Two different multi-time aggregation units are defined in this
   specification.  Both of them consist of 16 bits of unsigned size
   information of the following NAL unit (in network byte order), an
   8-bit unsigned decoding order number difference (DOND), and n bits
   (in network byte order) of timestamp offset (TS offset) for this NAL
   unit, whereby n can be 16 or 24.  The choice between the different
   MTAP types (MTAP16 and MTAP24) is application dependent: the larger
   the timestamp offset is, the higher the flexibility of the MTAP, but
   the overhead is also higher.

   The structure of the multi-time aggregation units for MTAP16 and
   MTAP24 are presented in Figures 10 and 11, respectively.  The
   starting or ending position of an aggregation unit within a packet is
   not required to be on a 32-bit word boundary.  The DON of the NAL
   unit contained in a multi-time aggregation unit is equal to (DONB +
   DOND) % 65536, in which % denotes the modulo operation.  This memo
   does not specify how the NAL units within an MTAP are ordered, but,
   in most cases, NAL unit decoding order SHOULD be used.

   The timestamp offset field MUST be set to a value equal to the value
   of the following formula: if the NALU-time is larger than or equal to
   the RTP timestamp of the packet, then the timestamp offset equals
   (the NALU-time of the NAL unit - the RTP timestamp of the packet).
   If the NALU-time is smaller than the RTP timestamp of the packet,
   then the timestamp offset is equal to the NALU-time + (2^32 - the RTP
   timestamp of the packet).

Top      Up      ToC       Page 27 
     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    :        NAL unit size          |      DOND     |  TS offset    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  TS offset    |                                               |
    +-+-+-+-+-+-+-+-+              NAL unit                         |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 10.  Multi-time aggregation unit for MTAP16


     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    :        NAL unit size         |      DOND     |  TS offset    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         TS offset             |                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
    |                              NAL unit                         |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 11.  Multi-time aggregation unit for MTAP24

   For the "earliest" multi-time aggregation unit in an MTAP, the
   timestamp offset MUST be zero.  Hence, the RTP timestamp of the MTAP
   itself is identical to the earliest NALU-time.

      Informative note: The "earliest" multi-time aggregation unit is
      the one that would have the smallest extended RTP timestamp among
      all the aggregation units of an MTAP if the NAL units contained in
      the aggregation units were encapsulated in single NAL unit
      packets.  An extended timestamp is a timestamp that has more than
      32 bits and is capable of counting the wraparound of the timestamp
      field, thus enabling one to determine the smallest value if the
      timestamp wraps.  Such an "earliest" aggregation unit may not be
      the first one in the order in which the aggregation units are
      encapsulated in an MTAP.  The "earliest" NAL unit need not be the
      same as the first NAL unit in the NAL unit decoding order either.

   Figure 12 presents an example of an RTP packet that contains a multi-
   time aggregation packet of type MTAP16 that contains two multi-time
   aggregation units, labeled as 1 and 2 in the figure.

Top      Up      ToC       Page 28 
     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          RTP Header                           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |MTAP16 NAL HDR |  decoding order number base   | NALU 1 Size   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  NALU 1 Size  |  NALU 1 DOND  |       NALU 1 TS offset        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  NALU 1 HDR   |  NALU 1 DATA                                  |
    +-+-+-+-+-+-+-+-+                                               +
    :                                                               :
    +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |               | NALU 2 SIZE                   |  NALU 2 DOND  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |       NALU 2 TS offset        |  NALU 2 HDR   |  NALU 2 DATA  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               |
    :                                                               :
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 12.  An RTP packet including a multi-time aggregation
                packet of type MTAP16 containing two multi-time
                aggregation units

Top      Up      ToC       Page 29 
   Figure 13 presents an example of an RTP packet that contains a multi-
   time aggregation packet of type MTAP24 that contains two multi-time
   aggregation units, labeled as 1 and 2 in the figure.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          RTP Header                           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |MTAP24 NAL HDR |  decoding order number base   | NALU 1 Size   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  NALU 1 Size  |  NALU 1 DOND  |       NALU 1 TS offs          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |NALU 1 TS offs |  NALU 1 HDR   |  NALU 1 DATA                  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
    :                                                               :
    +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |               | NALU 2 SIZE                   |  NALU 2 DOND  |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |       NALU 2 TS offset                        |  NALU 2 HDR   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  NALU 2 DATA                                                  |
    :                                                               :
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 13.  An RTP packet including a multi-time aggregation
                packet of type MTAP24 containing two multi-time
                aggregation units

5.8.  Fragmentation Units (FUs)

   This payload type allows fragmenting a NAL unit into several RTP
   packets.  Doing so on the application layer instead of relying on
   lower-layer fragmentation (e.g., by IP) has the following advantages:

   o  The payload format is capable of transporting NAL units bigger
      than 64 kbytes over an IPv4 network that may be present in pre-
      recorded video, particularly in High-Definition formats (there is
      a limit of the number of slices per picture, which results in a
      limit of NAL units per picture, which may result in big NAL
      units).

   o  The fragmentation mechanism allows fragmenting a single NAL unit
      and applying generic forward error correction as described in
      Section 12.5.

Top      Up      ToC       Page 30 
   Fragmentation is defined only for a single NAL unit and not for any
   aggregation packets.  A fragment of a NAL unit consists of an integer
   number of consecutive octets of that NAL unit.  Each octet of the NAL
   unit MUST be part of exactly one fragment of that NAL unit.
   Fragments of the same NAL unit MUST be sent in consecutive order with
   ascending RTP sequence numbers (with no other RTP packets within the
   same RTP packet stream being sent between the first and last
   fragment).  Similarly, a NAL unit MUST be reassembled in RTP sequence
   number order.

   When a NAL unit is fragmented and conveyed within fragmentation units
   (FUs), it is referred to as a fragmented NAL unit.  STAPs and MTAPs
   MUST NOT be fragmented.  FUs MUST NOT be nested; that is, an FU MUST
   NOT contain another FU.

   The RTP timestamp of an RTP packet carrying an FU is set to the NALU-
   time of the fragmented NAL unit.

   Figure 14 presents the RTP payload format for FU-As.  An FU-A
   consists of a fragmentation unit indicator of one octet, a
   fragmentation unit header of one octet, and a fragmentation unit
   payload.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | FU indicator  |   FU header   |                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
    |                                                               |
    |                         FU payload                            |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 14.  RTP payload format for FU-A

   Figure 15 presents the RTP payload format for FU-Bs.  An FU-B
   consists of a fragmentation unit indicator of one octet, a
   fragmentation unit header of one octet, a decoding order number (DON)
   (in network byte order), and a fragmentation unit payload.  In other
   words, the structure of FU-B is the same as the structure of FU-A,
   except for the additional DON field.

Top      Up      ToC       Page 31 
     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | FU indicator  |   FU header   |               DON             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
    |                                                               |
    |                         FU payload                            |
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               :...OPTIONAL RTP padding        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 15.  RTP payload format for FU-B

   NAL unit type FU-B MUST be used in the interleaved packetization mode
   for the first fragmentation unit of a fragmented NAL unit.  NAL unit
   type FU-B MUST NOT be used in any other case.  In other words, in the
   interleaved packetization mode, each NALU that is fragmented has an
   FU-B as the first fragment, followed by one or more FU-A fragments.

   The FU indicator octet has the following format:

       +---------------+
       |0|1|2|3|4|5|6|7|
       +-+-+-+-+-+-+-+-+
       |F|NRI|  Type   |
       +---------------+

   Values equal to 28 and 29 in the type field of the FU indicator octet
   identify an FU-A and an FU-B, respectively.  The use of the F bit is
   described in Section 5.3.  The value of the NRI field MUST be set
   according to the value of the NRI field in the fragmented NAL unit.

   The FU header has the following format:

      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |S|E|R|  Type   |
      +---------------+

   S:     1 bit
          When set to one, the Start bit indicates the start of a
          fragmented NAL unit.  When the following FU payload is not the
          start of a fragmented NAL unit payload, the Start bit is set
          to zero.

Top      Up      ToC       Page 32 
   E:     1 bit
          When set to one, the End bit indicates the end of a fragmented
          NAL unit, i.e., the last byte of the payload is also the last
          byte of the fragmented NAL unit.  When the following FU
          payload is not the last fragment of a fragmented NAL unit, the
          End bit is set to zero.

   R:     1 bit
          The Reserved bit MUST be equal to 0 and MUST be ignored by the
          receiver.

   Type:  5 bits
          The NAL unit payload type as defined in Table 7-1 of [1].

   The value of DON in FU-Bs is selected as described in Section 5.5.

      Informative note: The DON field in FU-Bs allows gateways to
      fragment NAL units to FU-Bs without organizing the incoming NAL
      units to the NAL unit decoding order.

   A fragmented NAL unit MUST NOT be transmitted in one FU; that is, the
   Start bit and End bit MUST NOT both be set to one in the same FU
   header.

   The FU payload consists of fragments of the payload of the fragmented
   NAL unit so that if the fragmentation unit payloads of consecutive
   FUs are sequentially concatenated, the payload of the fragmented NAL
   unit can be reconstructed.  The NAL unit type octet of the fragmented
   NAL unit is not included as such in the fragmentation unit payload,
   but rather the information of the NAL unit type octet of the
   fragmented NAL unit is conveyed in the F and NRI fields of the FU
   indicator octet of the fragmentation unit and in the type field of
   the FU header.  An FU payload MAY have any number of octets and MAY
   be empty.

      Informative note: Empty FUs are allowed to reduce the latency of a
      certain class of senders in nearly lossless environments.  These
      senders can be characterized in that they packetize NALU fragments
      before the NALU is completely generated and, hence, before the
      NALU size is known.  If zero-length NALU fragments were not
      allowed, the sender would have to generate at least one bit of
      data of the following fragment before the current fragment could
      be sent.  Due to the characteristics of H.264, where sometimes
      several macroblocks occupy zero bits, this is undesirable and can
      add delay.  However, the (potential) use of zero-length NALU
      fragments should be carefully weighed against the increased risk
      of the loss of at least a part of the NALU because of the
      additional packets employed for its transmission.

Top      Up      ToC       Page 33 
   If a fragmentation unit is lost, the receiver SHOULD discard all
   following fragmentation units in transmission order corresponding to
   the same fragmented NAL unit.

   A receiver in an endpoint or in a MANE MAY aggregate the first n-1
   fragments of a NAL unit to an (incomplete) NAL unit, even if fragment
   n of that NAL unit is not received.  In this case, the
   forbidden_zero_bit of the NAL unit MUST be set to one to indicate a
   syntax violation.

6.  Packetization Rules

   The packetization modes are introduced in Section 5.2.  The
   packetization rules common to more than one of the packetization
   modes are specified in Section 6.1.  The packetization rules for the
   single NAL unit mode, the non-interleaved mode, and the interleaved
   mode are specified in Sections 6.2, 6.3, and 6.4, respectively.

6.1.  Common Packetization Rules

   All senders MUST enforce the following packetization rules,
   regardless of the packetization mode in use:

   o  Coded slice NAL units or coded slice data partition NAL units
      belonging to the same coded picture (and thus sharing the same RTP
      timestamp value) MAY be sent in any order; however, for delay-
      critical systems, they SHOULD be sent in their original decoding
      order to minimize the delay.  Note that the decoding order is the
      order of the NAL units in the bitstream.

   o  Parameter sets are handled in accordance with the rules and
      recommendations given in Section 8.4.

   o  MANEs MUST NOT duplicate any NAL unit except for sequence or
      picture parameter set NAL units, as neither this memo nor the
      H.264 specification provides means to identify duplicated NAL
      units.  Sequence and picture parameter set NAL units MAY be
      duplicated to make their correct reception more probable, but any
      such duplication MUST NOT affect the contents of any active
      sequence or picture parameter set.  Duplication SHOULD be
      performed on the application layer and not by duplicating RTP
      packets (with identical sequence numbers).

Top      Up      ToC       Page 34 
   Senders using the non-interleaved mode and the interleaved mode MUST
   enforce the following packetization rule:

   o  In an RTP translator, MANEs MAY convert single NAL unit packets
      into one aggregation packet, convert an aggregation packet into
      several single NAL unit packets, or mix both concepts.  The RTP
      translator SHOULD take into account at least the following
      parameters: path MTU size, unequal protection mechanisms (e.g.,
      through packet-based FEC according to RFC 5109 [18], especially
      for sequence and picture parameter set NAL units and coded slice
      data partition A NAL units), bearable latency of the system, and
      buffering capabilities of the receiver.

         Informative note: An RTP translator is required to handle RTP
         Control Protocol (RTCP) as per RFC 3550.

6.2.  Single NAL Unit Mode

   This mode is in use when the value of the OPTIONAL packetization-mode
   media type parameter is equal to 0 or the packetization-mode is not
   present.  All receivers MUST support this mode.  It is primarily
   intended for low-delay applications that are compatible with systems
   using ITU-T Recommendation H.241 [3] (see Section 12.1).  Only single
   NAL unit packets MAY be used in this mode.  STAPs, MTAPs, and FUs
   MUST NOT be used.  The transmission order of single NAL unit packets
   MUST comply with the NAL unit decoding order.

6.3.  Non-Interleaved Mode

   This mode is in use when the value of the OPTIONAL packetization-mode
   media type parameter is equal to 1.  This mode SHOULD be supported.
   It is primarily intended for low-delay applications.  Only single NAL
   unit packets, STAP-As, and FU-As MAY be used in this mode.  STAP-Bs,
   MTAPs, and FU-Bs MUST NOT be used.  The transmission order of NAL
   units MUST comply with the NAL unit decoding order.

6.4.  Interleaved Mode

   This mode is in use when the value of the OPTIONAL packetization-mode
   media type parameter is equal to 2.  Some receivers MAY support this
   mode.  STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used.  STAP-As and
   single NAL unit packets MUST NOT be used.  The transmission order of
   packets and NAL units is constrained as specified in Section 5.5.

Top      Up      ToC       Page 35 
7.  De-Packetization Process

   The de-packetization process is implementation dependent.  Therefore,
   the following description should be seen as an example of a suitable
   implementation.  Other schemes may also be used as long as the output
   for the same input is the same as the process described below.  The
   same output means that the resulting NAL units and their order are
   identical.  Optimizations relative to the described algorithms are
   likely possible.  Section 7.1 presents the de-packetization process
   for the single NAL unit and non-interleaved packetization modes,
   whereas Section 7.2 describes the process for the interleaved mode.
   Section 7.3 includes additional de-packetization guidelines for
   intelligent receivers.

   All normal RTP mechanisms related to buffer management apply.  In
   particular, duplicated or outdated RTP packets (as indicated by the
   RTP sequence number and the RTP timestamp) are removed.  To determine
   the exact time for decoding, factors such as a possible intentional
   delay to allow for proper inter-stream synchronization must be
   factored in.

7.1.  Single NAL Unit and Non-Interleaved Mode

   The receiver includes a receiver buffer to compensate for
   transmission delay jitter.  The receiver stores incoming packets in
   reception order into the receiver buffer.  Packets are de-packetized
   in RTP sequence number order.  If a de-packetized packet is a single
   NAL unit packet, the NAL unit contained in the packet is passed
   directly to the decoder.  If a de-packetized packet is an STAP-A, the
   NAL units contained in the packet are passed to the decoder in the
   order in which they are encapsulated in the packet.  For all the FU-A
   packets containing fragments of a single NAL unit, the de-packetized
   fragments are concatenated in their sending order to recover the NAL
   unit, which is then passed to the decoder.

      Informative note: If the decoder supports arbitrary slice order,
      coded slices of a picture can be passed to the decoder in any
      order, regardless of their reception and transmission order.

7.2.  Interleaved Mode

   The general concept behind these de-packetization rules is to reorder
   NAL units from transmission order to the NAL unit decoding order.

   The receiver includes a receiver buffer, which is used to compensate
   for transmission delay jitter and to reorder NAL units from
   transmission order to the NAL unit decoding order.  In this section,
   the receiver operation is described under the assumption that there

Top      Up      ToC       Page 36 
   is no transmission delay jitter.  To differentiate the receiver
   buffer from a practical receiver buffer that is also used for
   compensation of transmission delay jitter, the receiver buffer is
   hereafter called the de-interleaving buffer in this section.
   Receivers SHOULD also prepare for transmission delay jitter, i.e.,
   either reserve separate buffers for transmission delay jitter
   buffering and de-interleaving buffering or use a receiver buffer for
   both transmission delay jitter and de-interleaving.  Moreover,
   receivers SHOULD take transmission delay jitter into account in the
   buffering operation, e.g., by additional initial buffering before
   starting of decoding and playback.

   This section is organized as follows: Subsection 7.2.1 presents how
   to calculate the size of the de-interleaving buffer.  Subsection
   7.2.2 specifies the receiver process on how to organize received NAL
   units to the NAL unit decoding order.

7.2.1.  Size of the De-Interleaving Buffer

   In either Offer/Answer or declarative Session Description Protocol
   (SDP) usage, the sprop-deint-buf-req media type parameter signals the
   requirement for the de-interleaving buffer size.  Therefore, it is
   RECOMMENDED to set the de-interleaving buffer size, in terms of
   number of bytes, equal to or greater than the value of the sprop-
   deint-buf-req media type parameter.

   When the SDP Offer/Answer model or any other capability exchange
   procedure is used in session setup, the properties of the received
   stream SHOULD be such that the receiver capabilities are not
   exceeded.  In the SDP Offer/Answer model, the receiver can indicate
   its capabilities to allocate a de-interleaving buffer with the deint-
   buf-cap media type parameter.  See Section 8.1 for further
   information on the deint-buf-cap and sprop-deint-buf-req media type
   parameters and Section 8.2.2 for further information on their use in
   the SDP Offer/Answer model.

7.2.2.  De-Interleaving Process

   There are two buffering states in the receiver: initial buffering and
   buffering while playing.  Initial buffering occurs when the RTP
   session is initialized.  After initial buffering, decoding and
   playback are started, and the buffering-while-playing mode is used.

   Regardless of the buffering state, the receiver stores incoming NAL
   units, in reception order, in the de-interleaving buffer as follows.
   NAL units of aggregation packets are stored in the de-interleaving
   buffer individually.  The value of DON is calculated and stored for
   each NAL unit.

Top      Up      ToC       Page 37 
   The receiver operation is described below with the help of the
   following functions and constants:

   o  Function AbsDON is specified in Section 8.1.

   o  Function don_diff is specified in Section 5.5.

   o  Constant N is the value of the OPTIONAL sprop-interleaving-depth
      media type parameter (see Section 8.1) incremented by 1.

   Initial buffering lasts until one of the following conditions is
   fulfilled:

   o  There are N or more VCL NAL units in the de-interleaving buffer.

   o  If sprop-max-don-diff is present, don_diff(m,n) is greater than
      the value of sprop-max-don-diff, in which n corresponds to the NAL
      unit having the greatest value of AbsDON among the received NAL
      units and m corresponds to the NAL unit having the smallest value
      of AbsDON among the received NAL units.

   o  Initial buffering has lasted for the duration equal to or greater
      than the value of the OPTIONAL sprop-init-buf-time media type
      parameter.

   The NAL units to be removed from the de-interleaving buffer are
   determined as follows:

   o  If the de-interleaving buffer contains at least N VCL NAL units,
      NAL units are removed from the de-interleaving buffer and passed
      to the decoder in the order specified below until the buffer
      contains N-1 VCL NAL units.

   o  If sprop-max-don-diff is present, all NAL units m for which
      don_diff(m,n) is greater than sprop-max-don-diff are removed from
      the de-interleaving buffer and passed to the decoder in the order
      specified below.  Herein, n corresponds to the NAL unit having the
      greatest value of AbsDON among the NAL units in the de-
      interleaving buffer.

Top      Up      ToC       Page 38 
   The order in which NAL units are passed to the decoder is specified
   as follows:

   o  Let PDON be a variable that is initialized to 0 at the beginning
      of the RTP session.

   o  For each NAL unit associated with a value of DON, a DON distance
      is calculated as follows.  If the value of DON of the NAL unit is
      larger than the value of PDON, the DON distance is equal to DON -
      PDON.  Otherwise, the DON distance is equal to 65535 - PDON + DON
      + 1.

   o  NAL units are delivered to the decoder in ascending order of DON
      distance.  If several NAL units share the same value of DON
      distance, they can be passed to the decoder in any order.

   o  When a desired number of NAL units have been passed to the
      decoder, the value of PDON is set to the value of DON for the last
      NAL unit passed to the decoder.

7.3.  Additional De-Packetization Guidelines

   The following additional de-packetization rules may be used to
   implement an operational H.264 de-packetizer:

   o  Intelligent RTP receivers (e.g., in gateways) may identify lost
      coded slice data partitions A (DPAs).  If a lost DPA is detected,
      after taking into account possible retransmission and FEC, a
      gateway may decide not to send the corresponding coded slice data
      partitions B and C, as their information is meaningless for H.264
      decoders.  In this way, a MANE can reduce network load by
      discarding useless packets without parsing a complex bitstream.

   o  Intelligent RTP receivers (e.g., in gateways) may identify lost
      FUs.  If a lost FU is found, a gateway may decide not to send the
      following FUs of the same fragmented NAL unit, as their
      information is meaningless for H.264 decoders.  In this way, a
      MANE can reduce network load by discarding useless packets without
      parsing a complex bitstream.

   o  Intelligent receivers having to discard packets or NALUs should
      first discard all packets/NALUs in which the value of the NRI
      field of the NAL unit type octet is equal to 0.  This will
      minimize the impact on user experience and keep the reference
      pictures intact.  If more packets have to be discarded, then

Top      Up      ToC       Page 39 
      packets with a numerically lower NRI value should be discarded
      before packets with a numerically higher NRI value.  However,
      discarding any packets with an NRI bigger than 0 very likely leads
      to decoder drift and SHOULD be avoided.



(page 39 continued on part 3)

Next RFC Part