RFC 6313
Export of Structured Data in IP Flow Information Export (IPFIX)
4. Linkage with the IPFIX Information Model
As in the IPFIX protocol specification [RFC5101], the new Information Elements specified in Section 4.3 MUST be sent in canonical format in network-byte order (also known as the big-endian byte ordering).4.1. New Abstract Data Types
This document specifies three new abstract data types, as described below.4.1.1. basicList
The type "basicList" represents a list of zero or more instances of any Information Element, primarily used for single-valued data types. Examples include a list of port numbers, a list of interface indexes, a list of AS in a BGP AS-PATH, etc.4.1.2. subTemplateList
The type "subTemplateList" represents a list of zero or more instances of a structured data type, where the data type of each list element is the same and corresponds with a single Template Record. Examples include a structured data type composed of multiple pairs of ("MPLS label stack entry position", "MPLS label stack value"), a structured data type composed of performance metrics, and a structured data type composed of multiple pairs of IP address, etc.4.1.3. subTemplateMultiList
The type "subTemplateMultiList" represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different Template definitions. Examples include a structured data type composed of
multiple access-list entries, where entries can be composed of different criteria types.4.2. New Data Type Semantic
This document specifies a new data type semantic, in addition to the ones specified in Section 3.2 of the IPFIX information model [RFC5102], as described below.4.2.1. List
A list represents an arbitrary-length sequence of zero or more structured data Information Elements, either composed of regular Information Elements or composed of data conforming to a Template Record.4.3. New Information Elements
This document specifies three new Information Elements, as described below.4.3.1. basicList
A basicList specifies a generic Information Element with a basicList abstract data type as defined in Section 4.1.1 and list semantics as defined in Section 4.2.1. Examples include a list of port numbers, a list of interface indexes, etc.4.3.2. subTemplateList
A subTemplateList specifies a generic Information Element with a subTemplateList abstract data type as defined in Section 4.1.2 and list semantics as defined in Section 4.2.1.4.3.3. subTemplateMultiList
A subTemplateMultiList specifies a generic Information Element with a subTemplateMultiList abstract data type as defined in Section 4.1.3 and list semantics as defined in Section 4.2.1.4.4. New Structured Data Type Semantics
Structured data type semantics are provided in order to express the relationship among multiple list elements in a Structured Data Information Element. These structured data type semantics require a new IPFIX subregistry, as specified in the "IANA Considerations" section. The semantics are specified in the following subsections.
4.4.1. undefined
The "undefined" structured data type semantic specifies that the semantic of list elements is not specified and that, if a semantic exists, then it is up to the Collecting Process to draw its own conclusions. The "undefined" structured data type semantic, which is the default value, is used when no other structured data type semantic applies. For example, a mediator that wants to translate IPFIX [RFC5101] into the export of structured data according to the specifications in this document doesn't know what the semantic is; it can only guess, as the IPFIX specifications [RFC5101] does not contain any semantic. Therefore, the mediator should use the "undefined" semantic.4.4.2. noneOf
The "noneOf" structured data type semantic specifies that none of the elements are actual properties of the Data Record. For example, a mediator might want to report to a Collector that a specific Flow is suspicious, but that it checked already that this Flow does not belong to the attack type 1, attack type 2, or attack type 3. So this Flow might need some further inspection. In such a case, the mediator would report the Flow Record with a basicList composed of (attack type 1, attack type 2, attack type 3) and the respective structured data type semantic of "noneOf". Another example is a router that monitors some specific BGP AS-PATHs and reports if a Flow belongs to any of them. If the router wants to export that a Flow does not belong to any of the monitored BGP AS- PATHs, the router reports a Data Record with a basicList composed of (BGP AS-PATH 1, BGP AS-PATH 2, BGP AS-PATH 3) and the respective structured data type semantic of "noneOf".4.4.3. exactlyOneOf
The "exactlyOneOf" structured data type semantic specifies that only a single element from the structured data is an actual property of the Data Record. This is equivalent to a logical XOR operation. For example, if a Flow record contains a basicList of outgoing interfaces with the "exactlyOneOf" semantic, then it implies that the reported Flow only egressed from a single interface, although the Flow Record lists all of the possible outgoing interfaces. This is a typical example of a per destination load-balancing.
Another example is a mediator that must aggregate Data Records from
different Observation Points and report an aggregated Observation
Point. However, the different Observation Points can be specified by
different Information Element types depending on the Exporter. For
example:
Exporter1 Observation Point is characterized by the
exporterIPv4Address, so a specific Exporter can be represented.
Exporter2 Observation Point is characterized by the
exporterIPv4Address and a basicList of ingressInterface, so the
Exporting Process can express that the observations were made on a
series of input interfaces.
Exporter3 Observation Point is characterized by the
exporterIPv4Address and a specific lineCardId, so the Exporting
Process can express that the observation was made on a specific
linecard.
If the mediator models the three different types of Observation
Points with the three Template Records below:
Template Record 1: exporterIPv4Address
Template Record 2: exporterIPv4Address, basicList of
ingressInterface
Template Record 3: exporterIPv4Address, lineCardId
then it can represent the aggregated Observation Point with a
subTemplateMultiList and the semantic "exactlyOneOf". The aggregated
Observation Point is modeled with the Data Records corresponding to
either Template Record 1, Template Record 2, or Template Record 3 but
not more than one of these. This implies that the Flow was observed
at exactly one of the Observation Points reported.
4.4.4. oneOrMoreOf
The "oneOrMoreOf" structured data type semantic specifies that one or
more elements from the list in the structured data are actual
properties of the Data Record. This is equivalent to a logical OR
operation.
Consider an example where a mediator must report an aggregated Flow
(e.g., by aggregating IP addresses from IP prefixes), with an
aggregated Observation Point. However, the different Observation
Points can be specified by different Information Element types as
described in Section 4.4.2.
If the mediator models the three different types of Observation
Points with the three Template Records below:
Template Record 1: exporterIPv4Address
Template Record 2: exporterIPv4Address, basicList of
ingressInterface
Template Record 3: exporterIPv4Address, lineCardId
then it can represent the aggregated Observation Point with a
subTemplateMultiList and the semantic "oneOrMoreOf". The aggregated
Observation Point is modeled with the Data Records corresponding to
either Template Record 1, Template Record 2, or Template Record 3.
This implies that the Flow was observed on at least one of the
Observation Points reported, and potentially on multiple Observation
Points.
4.4.5. allOf
The "allOf" structured data type semantic specifies that all of the
list elements from the structured data are actual properties of the
Data Record.
For example, if a Record contains a basicList of outgoing interfaces
with the "allOf" semantic, then the observed Flow is typically a
multicast Flow where each packet in the Flow has been replicated to
each outgoing interface in the basicList.
4.4.6. ordered
The "ordered" structured data type semantic specifies that elements
from the list in the structured data are ordered.
For example, an Exporter might want to export the AS10 AS20 AS30 AS40
BGP AS-PATH. In such a case, the Exporter would report a basicList
composed of (AS10, AS20, AS30, AS40) and the respective structured
data type semantic of "ordered".
4.5. Encoding of IPFIX Data Types
The following subsections define the encoding of the abstract data
types defined in Section 4.1. These data types may be encoded using
either fixed- or variable-length Information Elements, as discussed
in Section 5.1. Like in the IPFIX specifications [RFC5101], all
lengths are specified in octets.
4.5.1. basicList
The basicList Information Element defined in Section 4.3.1 represents a list of zero or more instances of an Information Element and is encoded as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Semantic |0| Field ID | Element... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ...Length | basicList Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1: basicList Encoding Semantic The Semantic field indicates the relationship among the different Information Element values within this Structured Data Information Element. Refer to IANA's "IPFIX Structured Data Types Semantics" registry. Field ID Field ID is the Information Element identifier of the Information Element(s) contained in the list. Element Length Per Section 7 of [RFC5101], the Element Length field indicates the length, in octets, of each list element specified by Field ID, or contains the value 0xFFFF if the length is encoded as a variable- length Information Element at the start of the basicList Content. Effectively, the Element Length field is part of the header, so even in the case of a zero-element list, it MUST NOT be omitted. basicList Content A Collecting Process decodes list elements from the basicList Content until no further data remains. A field count is not included but can be derived when the Information Element is decoded.
Note that in the diagram above, Field ID is shown with the Enterprise bit (most significant bit) set to 0. Instead, if the Enterprise bit is set to 1, a four-byte Enterprise Number MUST be encoded immediately after the Element Length as shown below. See the "Field Specifier Format" section in the IPFIX protocol [RFC5101] for additional information. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Semantic |1| Field ID | Element... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ...Length | Enterprise Number ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | basicList Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2: basicList Encoding with Enterprise Number Also, note that if a basicList has zero elements, the encoded data contains the Semantic field, Field ID, the Element Length field, and the four-byte Enterprise Number (if present), while the basicList Content is empty. If the basicList is encoded as a variable-length Information Element in less than 255 octets, it MAY be encoded with the Length field per Section 7 of [RFC5101] as shown in Figure 3. However, the three-byte length encoding, as shown in Figure 4, is RECOMMENDED (see Section 5.1). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Length (< 255)| Semantic |0| Field ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Element Length | basicList Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3: Variable-Length basicList Encoding (Length < 255 Octets)
If the basicList is encoded as a variable-length Information Element in 255 or more octets, it MUST be encoded with the Length field per Section 7 of [RFC5101] as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 255 | Length (0 to 65535) | Semantic | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| Field ID | Element Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | basicList Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4: Variable-Length basicList Encoding (Length 0 to 65535 Octets)4.5.2. subTemplateList
The subTemplateList Information Element represents a list of zero or more Data Records corresponding to a specific Template. Because the Template Record referenced by a subTemplateList Information Element can itself contain other subTemplateList Information Elements, and because these Template Record references are part of the Information Elements content in the Data Record, it is possible to represent complex hierarchical data structures. The following diagram shows how a subTemplateList Information Element is encoded within a Data Record: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Semantic | Template ID | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | subTemplateList Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5: subTemplateList Encoding
Semantic
The Semantic field indicates the relationship among the different
Data Records within this Structured Data Information Element.
Template ID
The Template ID field contains the ID of the Template used to
encode and decode the subTemplateList Content.
subTemplateList Content
subTemplateList Content consists of zero or more instances of Data
Records corresponding to the Template ID specified in the Template
ID field. A Collecting Process decodes the subTemplateList
Content until no further data remains. A record count is not
included but can be derived when the subTemplateList is decoded.
Encoding and decoding are performed recursively if the specified
Template itself contains Structured Data Information Elements as
described here.
Note that, if a subTemplateList has zero elements, the encoded data
contains only the Semantic field and the Template ID field, while the
subTemplateList Content is empty.
If the subTemplateList is encoded as a variable-length Information
Element in less than 255 octets, it MAY be encoded with the Length
field per Section 7 of [RFC5101] as shown in Figure 6. However, the
three-byte length encoding, as shown in Figure 7, is RECOMMENDED (see
Section 5.1).
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (< 255)| Semantic | Template ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| subTemplateList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6: Variable-Length subTemplateList Encoding
(Length < 255 Octets)
If the subTemplateList is encoded as a variable-length Information Element in 255 or more octets, it MUST be encoded with the Length field per Section 7 of [RFC5101] as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 255 | Length (0 to 65535) | Semantic | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID | subTemplateList Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7: Variable-Length subTemplateList Encoding (Length 0 to 65535 Octets)4.5.3. subTemplateMultiList
Whereas each element in a subTemplateList Information Element corresponds to a single Template, it is sometimes useful for a list to contain elements corresponding to different Templates. To support this case, each top-level element in a subTemplateMultiList Information Element carries a Template ID, Length, and zero or more Data Records corresponding to the Template ID. The following diagram shows how a subTemplateMultiList Information Element is encoded within a Data Record. Note that the encoding following the Semantic field is consistent with the Set Header specified in [RFC5101]. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Semantic | Template ID X |Data Records...| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... Length X | Data Record X.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record X.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record X.L Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Template ID Y |Data Records...| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Length Y | Data Record Y.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Y.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Y.M Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Template ID Z |Data Records...| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... Length Z | Data Record Z.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Z.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Z.N Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+ Figure 8: subTemplateMultiList Encoding Semantic The Semantic field indicates the top-level relationship among the series of Data Records corresponding to the different Template Records within this Structured Data Information Element. Template ID Unlike the subTemplateList Information Element, each element of the subTemplateMultiList contains a Template ID that specifies the encoding of the following Data Records.
Data Records Length
This is the total length of the Data Records encoding for the
Template ID previously specified, including the two bytes for the
Template ID and the two bytes for the Data Records Length field
itself.
Data Record X.M
The Data Record X.M consists of the Mth Data Record of the
Template Record X. A Collecting Process decodes the Data Records
according to Template Record X until no further data remains,
according to the Data Records Length X. Further Template IDs and
Data Records may then be decoded according to the overall
subTemplateMultiList length. A record count is not included but
can be derived when the Element Content is decoded. Encoding and
decoding are performed recursively if the specified Template
itself contains Structured Data Information Elements as described
here.
In the exceptional case of zero instances in the
subTemplateMultiList, no data is encoded, only the Semantic field and
Template ID field(s), and the Data Record Length field is set to
zero.
If the subTemplateMultiList is encoded as a variable-length
Information Element in less than 255 octets, it MAY be encoded with
the Length field per Section 7 of [RFC5101] as shown in Figure 9.
However, the three-byte length encoding, as shown in Figure 10, is
RECOMMENDED (see Section 5.1).
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (< 255)| Semantic | Template ID X |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Records Length X | Data Record X.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record X.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record X.L Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Template ID Y | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Records Length Y | Data Record Y.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Y.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Y.M Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Template ID Z | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Records Length Z | Data Record Z.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Z.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Data Record Z.N Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9: Variable-Length subTemplateMultiList Encoding (Length < 255 Octets) If the subTemplateMultiList is encoded as a variable-length Information Element in 255 or more octets, it MUST be encoded with the Length field per Section 7 of [RFC5101] as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 255 | Length (0 to 65535) | Semantic | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID X | Data Records Length X | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record X.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record X.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record X.L Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID Y | Data Records Length Y | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record Y.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record Y.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record Y.M Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID Z | Data Records Length Z | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record Z.1 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record Z.2 Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Record Z.N Content ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10: Variable-Length subTemplateMultiList Encoding (Length 0 to 65535 Octets)5. Structured Data Format
5.1. Length Encoding Considerations
The new Structured Data Information Elements represent a list that potentially carries complex hierarchical and repeated data.
When the encoding of a Structured Data Information Element has a fixed length (because, for example, it contains the same number of fixed-length elements, or if the permutations of elements in the list always produces the same total length), the element length can be encoded in the corresponding Template Record. However, when representing variable-length data, hierarchical data, and repeated data with variable element counts, where the number and length of elements can vary from record to record, we RECOMMEND that the Information Elements are encoded using the variable-length encoding described in Section 7 of [RFC5101], with the length carried before the Structured Data Information Element encoding. Because of the complex and repeated nature of the data, it is potentially difficult for the Exporting Process to efficiently know in advance the exact encoding size. In this case, the Exporting Process may encode the available data starting at a fixed offset and fill in the final length afterwards. Therefore, the three-byte length encoding is RECOMMENDED for variable-length Information Elements in all Template Records containing a Structured Data Information Element, even if the encoded length can be less than 255 bytes, because the starting offset of the data is known in advance. When encoding such data, an Exporting Process MUST take care to not exceed the maximum allowed IPFIX message length of 65535 bytes as specified in [RFC5101].5.2. Recursive Structured Data
It is possible to define recursive relationships between IPFIX structured data instances, for example, when representing a tree structure. The simplest case of this might be a basicList, where each element is itself a basicList, or a subTemplateList where one of the fields of the referenced Template is itself a subTemplateList referencing the same Template. Also, the Exporting Process MUST take care when encoding recursively-defined structured data not to exceed the maximum allowed length of an IPFIX Message (as noted in Length Encoding Considerations).5.3. Structured Data Information Elements Applicability in Options Template Sets
Structured Data Information Elements MAY be used in Options Template Sets. As an example, consider a mediation function that must aggregate Data Records from multiple Observation Point types:
Router 1, (interface 1)
Router 2, (linecard A)
Router 3, (linecard B)
Router 4, (linecard C, interface 2)
In order to encode the PSAMP Selection Sequence Report Interpretation
[RFC5476], the mediation function must express this combination of
Observation Points as a single new Observation Point. Recall from
[RFC5476] that the PSAMP Selection Sequence Report Interpretation
consists of the following fields:
Scope: selectionSequenceId
Non-Scope: one Information Element mapping the Observation Point
selectorId (one or more)
Without structured data, there is clearly no way to express the
complex aggregated Observation Point as "one Information Element
mapping the Observation Point". However, the desired result may be
easily achieved using the structured data types. Refer to Section
9.5. for an encoding example related to this case study.
Regarding the scope in the Options Template Record, the IPFIX
specification [RFC5101] mentions that "the IPFIX protocol doesn't
prevent the use of any Information Elements for scope". Therefore, a
Structured Data Information Element MAY be used as scope in an
Options Template Set.
Extending the previous example, the mediation function could export a
given name for this complex aggregated Observation Point:
Scope: Aggregated Observation Point (structured data)
Non-Scope: a new Information Element containing the name
5.4. Usage Guidelines for Equivalent Data Representations
Because basicList, subTemplateList, and subTemplateMultiList are all
lists, in several cases, there is more than one way to represent what
is effectively the same data structure. However, in some cases, one
approach has an advantage over the other, e.g., more compact, uses
fewer resources, and is therefore preferred over an alternate
representation.
A subTemplateList can represent the same simple list of single-valued
Information Elements as a basicList, if the Template referenced by
the subTemplateList contains only one single-valued Information
Element. Although the encoding is more compact than a basicList by
two bytes, using a subTemplateList, in this case, requires a new
Template per Information Element. The basicList requires no additional Template and is therefore RECOMMENDED in this case. Although a subTemplateMultiList with one Element can represent the contents of a subTemplateList, the subTemplateMultiList carries two additional bytes (Element Length). It is also potentially useful to a Collecting Process to know in advance that a subTemplateList directly indicates that list element types are consistent. The subTemplateList Information Element is therefore RECOMMENDED in this case. The Semantic field in a subTemplateMultiList indicates the top-level relationship among the series of Data Records corresponding to the different Template Records, within this Structured Data Information Element. If a semantic is required to describe the relationship among the different Data Records corresponding to a single Template ID within the subTemplateMultiList, then an encoding based on a basicList of subTemplateLists should be used; refer to Section 5.6 for more information. Alternatively, if a semantic is required to describe the relationship among all Data Records within a subTemplateMultiList (regardless of the Template Record), an encoding based on a subTemplateMultiList with one Data Record corresponding to a single Template ID can be used. Note that the referenced Information Element(s) in the Structured Data Information Elements can be taken from the IPFIX information model [RFC5102], the PSAMP information model [RFC5477], any of the Information Elements defined in the IANA IPFIX registry [IANA-IPFIX], or enterprise-specific Information Elements. If a Template Record contains a subTemplateList as the only field, a Set encoding as specified in the IPFIX protocol specifications [RFC5101] should be considered, unless: - A relationship among multiple list elements must be exported, in which case, the semantic from the IPFIX Structured Data Information Element can convey this relationship. - The Exporting Process wants to convey the number of elements in the list, even in the special cases of zero or one element in the list. Indeed, the case of an empty list cannot be represented with the IPFIX protocol specifications [RFC5101]. In the case of a single element list, the Template Record specified in the IPFIX protocol specification [RFC5101] could be used. However, on the top of the Template Record with the subTemplateList to export multiple list elements, this supplementary Template would impose some extra
management, both on the Exporting Process and on the Collecting
Process, which might have to correlate the information from two
Template Records.
Similarly, if a Template Record contains a subTemplateMultiList as
the only field, an IPFIX Message as described in the IPFIX protocol
specification [RFC5101] should be considered, unless:
- A relationship among top-level list elements must be exported, in
which case, the semantic from the IPFIX Structured Data Information
Element can convey this relationship.
- The Exporting Process wants to convey the number of Data Records
corresponding to every Template in the subTemplateMultiList.
5.5. Padding
The Exporting Process MAY insert some padding octets in structured
data field values in a Data Record by including the 'paddingOctets'
Information Element as described in [RFC5101], Section 3.3.1. The
paddingOctets Information Element can be included in a Template
Record referenced by a structured data Information Element for this
purpose.
5.6. Semantic
Semantic interpretations of received Data Records at or beyond the
Collecting Process remain explicitly undefined, unless that data is
transmitted using this extension with explicit structured data type
semantic information.
It is not the Exporter's role to check the validity of the semantic
representation of Data Records.
More complex semantics can be expressed as a combination of the
Semantic Data Information Elements specified in this document.
For example, the export of the AS10 AS20 AS30 AS40 {AS50,AS60} BGP
AS-PATH would be reported as a basicList of two elements, each
element being a basicList of BGP AS, with the top-level structured
data type semantic of "ordered". The first element would contain a
basicList composed of (AS10,AS20,AS30,AS40) and the respective
structured data type semantic of "ordered", while the second element
would contain a basicList composed of (AS50, AS60) and the respective
structured data type semantic of "exactlyOneOf". A high-level Data
Record diagram would be represented as:
BGP AS-PATH = (basicList, ordered,
(basicList, ordered, AS10,AS20,AS30,AS40),
(basicList, exactlyOneOf, AS50, AS60)
)
If a semantic is required to describe the relationship among the
different Data Records corresponding to a single Template ID within
the subTemplateMultiList, then an encoding based on a basicList of
subTemplateLists should be used, as shown in the next case study.
Case study 1:
In this example, an Exporter monitoring security attacks must export
a list of security events consisting of attackers and targets. For
the sake of the example, assume that the Collector can differentiate
the attacker (which is expressed using source fields) from the target
(which is expressed using destination fields). Imagine that
attackers A1 or A2 may attack targets T1 and T2.
The first case uses a subTemplateMultiList composed of two Template
Records, one representing the attacker and one representing the
target, each of them containing an IP address and a port.
Attacker Template Record = (src IP address, src port)
Target Template Record = (dst IP address, dst port)
A high-level Data Record diagram would be represented as:
Alert = (subTemplateMultiList, allOf,
(Attacker Template Record, A1, A2),
(Target Template Record, T1, T2)
)
The Collecting Process can only conclude that the list of attackers
(A1, A2) and the list of targets (T1, T2) are present, without
knowing the relationship amongst attackers and targets. The
Exporting Process would have to explicitly call out the relationship
amongst attackers and targets as the top-level semantic offered by
the subTemplateMultiList isn't sufficient.
The only proper encoding for the previous semantic (i.e., attacker A1
or A2 may attack target T1 and T2) uses a basicList of
subTemplateLists and is represented as follows:
Attacker Template Record = (src IP address, src port)
Target Template Record = (dst IP address, dst port)
Alert = (basicList, allOf,
(subTemplateList, exactlyOneOf, attacker A1, A2)
(subTemplateList, allOf, target T1, T2)
)
Case study 2:
In this example, an Exporter monitoring security attacks must export
a list of attackers and targets. For the sake of the example, assume
that the Collector can differentiate the attacker (which is expressed
using source fields) from the target (which is expressed using
destination fields). Imagine that attacker A1 or A2 is attacking
target T1, while attacker A3 is attacking targets T2 and T3. The
first case uses a subTemplateMultiList that contains Data Records
corresponding to two Template Records, one representing the attacker
and one representing the target, each of them containing an IP
address and a port.
Attacker Template Record = (src IP address, src port)
Target Template Record = (dst IP address, dst port)
A high-level Data Record diagram would be represented as:
Alert = (subTemplateMultiList, allOf,
(Attacker Template Record, A1, A2, A3),
(Target Template Record, T1, T2, T3)
)
The Collecting Process can only conclude that the list of attackers
(A1, A2, A3), and the list of targets (T1, T2, T3) are present,
without knowing the relationship amongst attackers and targets.
The second case could use a Data Record definition composed of the
following:
Alert = (subTemplateMultiList, allOf,
(Attacker Template Record, A1, A2),
(Target Template Record, T1),
(Attacker Template Record, A3),
(Target Template Record, T2, T3)
)
With the above representation, the Collecting Process can infer that
the alert consists of the list of attackers (A1, A2), target (T1),
attacker (A3), and list of targets (T2, T3). From the sequence in
which attackers and targets are encoded, the Collector can possibly
deduce that some relationship exists among (A1, A2, T1) and (A2, T1,
T2) but cannot understand what it is exactly. So, there is a need
for the Exporting Process to explicitly define the relationship
between the attackers, and targets and the top-level semantic of the
subTemplateMultiList is not sufficient.
The only proper encoding for the previous semantic (i.e., attacker A1
or A2 attacks target T1, attacker A3 attacks targets T2 and T3) uses
a basicList of subTemplateLists and is represented as follows:
Participant P1 =
(basicList, allOf,
(subTemplateList, exactlyOneOf, attacker A1, A2)
(subTemplateList, undefined, target T1)
)
Participant P2 =
(basicList, allOf,
(subTemplateList, undefined, attacker A3,
(subTemplateList, allOf, targets T2, T3)
)
The security alert is represented as a subTemplateList of
participants.
Alert =
(subTemplateList, allOf, Participant P1, Participant P2)
Note that, in the particular case of a single element in a Structured
Data Information Element, the Semantic field is actually not very
useful since it specifies the relationship among multiple elements.
Any choice of allOf, exactlyOneOf, or OneOrMoreOf would provide the
same result semantically. Therefore, in case of a single element in
a Structured Data Information Element, the default "undefined"
semantic SHOULD be used.
(page 33 continued on part 3)
