Tech-invite3GPPspaceIETF RFCsSIP
929190898887868584838281807978777675747372717069686766656463626160595857565554535251504948474645444342414039383736353433323130292827262524232221201918171615141312111009080706050403020100
in Index   Prev   Next

RFC 6313

Export of Structured Data in IP Flow Information Export (IPFIX)

Pages: 71
Proposed Standard
Errata
Updates:  5102
Part 1 of 4 – Pages 1 to 12
None   None   Next

Top   ToC   RFC6313 - Page 1
Internet Engineering Task Force (IETF)                         B. Claise
Request for Comments: 6313                                 G. Dhandapani
Updates: 5102                                                  P. Aitken
Category: Standards Track                                       S. Yates
ISSN: 2070-1721                                      Cisco Systems, Inc.
                                                               July 2011


    Export of Structured Data in IP Flow Information Export (IPFIX)

Abstract

This document specifies an extension to the IP Flow Information Export (IPFIX) protocol specification in RFC 5101 and the IPFIX information model specified in RFC 5102 to support hierarchical structured data and lists (sequences) of Information Elements in data records. This extension allows definition of complex data structures such as variable-length lists and specification of hierarchical containment relationships between Templates. Finally, the semantics are provided in order to express the relationship among multiple list elements in a structured data record. Status of This Memo This is an Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc6313. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of
Top   ToC   RFC6313 - Page 2
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

1. Overview ........................................................5 1.1. IPFIX Documents Overview ...................................5 1.2. Relationship between IPFIX and PSAMP .......................6 2. Introduction ....................................................6 2.1. The IPFIX Track ............................................7 2.2. The IPFIX Limitations ......................................8 2.3. Structured Data Use Cases ..................................8 2.4. Specifications Summary ....................................11 3. Terminology ....................................................11 3.1. New Terminology ...........................................12 3.2. Conventions Used in This Document .........................12 4. Linkage with the IPFIX Information Model .......................12 4.1. New Abstract Data Types ...................................12 4.1.1. basicList ..........................................12 4.1.2. subTemplateList ....................................12 4.1.3. subTemplateMultiList ...............................12 4.2. New Data Type Semantic ....................................13 4.2.1. List ...............................................13 4.3. New Information Elements ..................................13 4.3.1. basicList ..........................................13 4.3.2. subTemplateList ....................................13 4.3.3. subTemplateMultiList ...............................13 4.4. New Structured Data Type Semantics ........................13 4.4.1. undefined ..........................................14 4.4.2. noneOf .............................................14 4.4.3. exactlyOneOf .......................................14 4.4.4. oneOrMoreOf ........................................15 4.4.5. allOf ..............................................16 4.4.6. ordered ............................................16 4.5. Encoding of IPFIX Data Types ..............................16 4.5.1. basicList ..........................................17 4.5.2. subTemplateList ....................................19 4.5.3. subTemplateMultiList ...............................21 5. Structured Data Format .........................................25 5.1. Length Encoding Considerations ............................25 5.2. Recursive Structured Data .................................26 5.3. Structured Data Information Elements Applicability in Options Template Sets ..................................26 5.4. Usage Guidelines for Equivalent Data Representations ......27 5.5. Padding ...................................................29 5.6. Semantic ..................................................29 6. Template Management ............................................33 7. The Collecting Process's Side ..................................33
Top   ToC   RFC6313 - Page 3
   8. Defining New Information Elements Based on the New
      Abstract Data Types ............................................34
   9. Structured Data Encoding Examples ..............................34
      9.1. Encoding a Multicast Data Record with basicList ...........35
      9.2. Encoding a Load-Balanced Data Record with a basicList .....37
      9.3. Encoding subTemplateList ..................................38
      9.4. Encoding subTemplateMultiList .............................41
      9.5. Encoding an Options Template Set Using Structured Data ....46
   10. Relationship with the Other IPFIX Documents ...................51
      10.1. Relationship with Reducing Redundancy ....................51
           10.1.1. Encoding Structured Data Element Using
                   Common Properties .................................51
           10.1.2. Encoding Common Properties Elements with
                   Structured Data Information Element ...............51
      10.2. Relationship with Guidelines for IPFIX Testing ...........53
      10.3. Relationship with IPFIX Mediation Function ...............54
   11. IANA Considerations ...........................................54
      11.1. New Abstract Data Types ..................................54
           11.1.1. basicList .........................................54
           11.1.2. subTemplateList ...................................54
           11.1.3. subTemplateMultiList ..............................55
      11.2. New Data Type Semantics ..................................55
           11.2.1. list ..............................................55
      11.3. New Information Elements .................................55
           11.3.1. basicList .........................................55
           11.3.2. subTemplateList ...................................56
           11.3.3. subTemplateMultiList ..............................56
      11.4. New Structured Data Semantics ............................56
           11.4.1. undefined .........................................56
           11.4.2. noneOf ............................................57
           11.4.3. exactlyOneOf ......................................57
           11.4.4. oneOrMoreOf .......................................57
           11.4.5. allOf .............................................57
           11.4.6. ordered ...........................................58
   12. Security Considerations .......................................58
   13. References ....................................................58
      13.1. Normative References .....................................58
      13.2. Informative References ...................................58
   14. Acknowledgements ..............................................59
   Appendix A. Additions to XML Specification of IPFIX
               Information Elements and Abstract Data Types ..........60
   Appendix B. Encoding IPS Alert Using Structured Data
               Information Elements ..................................65
Top   ToC   RFC6313 - Page 4

Table of Figures

Figure 1: basicList Encoding ......................................17 Figure 2: basicList Encoding with Enterprise Number ...............18 Figure 3: Variable-Length basicList Encoding (Length < 255 Octets) 18 Figure 4: Variable-Length basicList Encoding (Length 0 to 65535 Octets) .................................................19 Figure 5: subTemplateList Encoding ................................19 Figure 6: Variable-Length subTemplateList Encoding (Length < 255 Octets) ...................................20 Figure 7: Variable-Length subTemplateList Encoding (Length 0 to 65535 Octets) ..............................21 Figure 8: subTemplateMultiList Encoding ...........................21 Figure 9: Variable-Length subTemplateMultiList Encoding (Length < 255 Octets) ...................................23 Figure 10: Variable-Length subTemplateMultiList Encoding (Length 0 to 65535 Octets) ..............................24 Figure 11: Encoding basicList, Template Record .....................35 Figure 12: Encoding basicList, Data Record, Semantic allOf .........36 Figure 13: Encoding basicList, Data Record with Variable-Length Elements, Semantic allOf ................................37 Figure 14: Encoding basicList, Data Record, Semantic exactlyOneOf ..38 Figure 15: Encoding subTemplateList, Template for One-Way Delay Metrics .................................................39 Figure 16: Encoding subTemplateList, Template Record ...............40 Figure 17: Encoding subTemplateList, Data Set ......................40 Figure 18: Encoding subTemplateMultiList, Template for Filtering Attributes ..............................................44 Figure 19: Encoding subTemplateMultiList, Template for Sampling Attributes ..............................................44 Figure 20: Encoding subTemplateMultiList, Template for Flow Record .45 Figure 21: Encoding subTemplateMultiList, Data Set .................45 Figure 22: PSAMP SSRI to Be encoded ................................48 Figure 23: Options Template Record for PSAMP SSRI Using subTemplateMultiList ....................................48 Figure 24: PSAMP SSRI, Template Record for interface ...............49 Figure 25: PSAMP SSRI, Template Record for linecard ................49 Figure 26: PSAMP SSRI, Template Record for linecard and interface ..49 Figure 27: Example of a PSAMP SSRI Data Record, Encoded Using a subTemplateMultiList ...................................50 Figure 28: Common and Specific Properties Exported Together [RFC5473] ..............................................51 Figure 29: Common and Specific Properties Exported Separately According to [RFC5473] .................................52 Figure 30: Common and Specific Properties Exported with Structured Data Information Element ...............................52 Figure 31: Encoding IPS Alert, Template for Target ................67 Figure 32: Encoding IPS Alert, Template for Attacker ..............68
Top   ToC   RFC6313 - Page 5
  Figure 33: Encoding IPS Alert, Template for Participant ...........68
  Figure 34: Encoding IPS Alert, Template for IPS Alert .............69
  Figure 35: Encoding IPS Alert, Data Set ...........................69

1. Overview

1.1. IPFIX Documents Overview

The IPFIX protocol [RFC5101] provides network administrators with access to IP Flow information. The architecture for the export of measured IP Flow information out of an IPFIX Exporting Process to a Collecting Process is defined in the IPFIX architecture [RFC5470], per the requirements defined in RFC 3917 [RFC3917]. The IPFIX architecture [RFC5470] specifies how IPFIX Data Records and Templates are carried via a congestion-aware transport protocol from IPFIX Exporting Processes to IPFIX Collecting Processes. IPFIX has a formal description of IPFIX Information Elements, their name, type, and additional semantic information, as specified in the IPFIX information model [RFC5102]. In order to gain a level of confidence in the IPFIX implementation, probe the conformity and robustness, and allow interoperability, the guidelines for IPFIX testing [RFC5471] present a list of tests for implementers of compliant Exporting Processes and Collecting Processes. The Bidirectional Flow Export [RFC5103] specifies a method for exporting bidirectional flow (biflow) information using the IP Flow Information Export (IPFIX) protocol, representing each biflow using a single Flow Record. "Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports" [RFC5473] specifies a bandwidth-saving method for exporting Flow or packet information, by separating information common to several Flow Records from information specific to an individual Flow Record: common Flow information is exported only once.
Top   ToC   RFC6313 - Page 6

1.2. Relationship between IPFIX and PSAMP

The specification in this document applies to the IPFIX protocol specifications [RFC5101]. All specifications from [RFC5101] apply unless specified otherwise in this document. The Packet Sampling (PSAMP) protocol [RFC5476] specifies the export of packet information from a PSAMP Exporting Process to a PSAMP Collecting Process. Like IPFIX, PSAMP has a formal description of its information elements, their name, type, and additional semantic information. The PSAMP information model is defined in [RFC5477]. As the PSAMP protocol specifications [RFC5476] are based on the IPFIX protocol specifications, the specifications in this document are also valid for the PSAMP protocol. Indeed, the major difference between IPFIX and PSAMP is that the IPFIX protocol exports Flow Records while the PSAMP protocol exports Packet Reports. From a pure export point of view, IPFIX will not distinguish a Flow Record composed of several packets aggregated together from a Flow Record composed of a single packet. So the PSAMP export can be seen as a special IPFIX Flow Record containing information about a single packet.

2. Introduction

While collecting the interface counters every five minutes has proven to be useful in the past, more and more granular information is required from network elements for a series of applications: performance assurance, capacity planning, security, billing, or simply monitoring. However, the amount of information has become so large that, when dealing with highly granular information such as Flow information, a push mechanism (as opposed to a pull mechanism, such as Simple Network Management Protocol (SNMP)) is the only solution for routers whose primary function is to route packets. Indeed, polling short-lived Flows via SNMP is not an option: high-end routers can support hundreds of thousands of Flows simultaneously. Furthermore, in order to reduce the export bandwidth requirements, the network elements have to integrate mediation functions to aggregate the collected information, both in space (typically, from different linecards or different Exporters) and in time. Typically, it would be beneficial if access routers could export Flow Records, composed of the counters before and after an optimization mechanism on the egress interface, instead of exporting two Flow Records with identical tuple information.
Top   ToC   RFC6313 - Page 7
   In terms of aggregation in time, let us imagine that, for performance
   assurance, the network management application must receive the
   performance metrics associated with a specific Flow, every
   millisecond.  Since the performance metrics will be constantly
   changing, there is a new dimension to the Flow definition: we are not
   dealing anymore with a single Flow lasting a few seconds or a few
   minutes, but with a multitude of one millisecond sub-flows for which
   the performance metrics are reported.

   Which current protocol is suitable for these requirements: push
   mechanism, highly granular information, and huge number of similar
   records? IPFIX, as specified in RFC 5101 would give part of the
   solution.

2.1. The IPFIX Track

The IPFIX working group has specified a protocol to export Flow information [RFC5101]. This protocol is designed to export information about IP traffic Flows and related measurement data, where a Flow is defined by a set of key attributes (e.g., source and destination IP address, source and destination port). The IPFIX protocol specification [RFC5101] specifies that traffic measurements for Flows are exported using a TLV (type, length, value) format. The information is exported using a Template Record that is sent once to export the {type, length} pairs that define the data format for the Information Elements in a Flow. The Data Records specify values for each Flow. Based on the requirements for IP Flow Information Export (IPFIX) [RFC3917], the IPFIX protocol has been optimized to export Flow- related information. However, thanks to its Template mechanism, the IPFIX protocol can export any type of information, as long as the relevant Information Element is specified in the IPFIX information model [RFC5102], registered with IANA [IANA-IPFIX], or specified as an enterprise-specific Information Element. For each Information Element, the IPFIX information model [RFC5102] defines a numeric identifier, an abstract data type, an encoding mechanism for the data type, and any semantic constraints. Only basic, single-valued data types, e.g., numbers, strings, and network addresses, are currently supported.
Top   ToC   RFC6313 - Page 8

2.2. The IPFIX Limitations

The IPFIX protocol specification [RFC5101] does not support the encoding of hierarchical structured data and arbitrary-length lists (sequences) of Information Elements as fields within a Template Record. As it is currently specified, a Data Record is a "flat" list of single-valued attributes. However, it is a common data modeling requirement to compose complex hierarchies of data types, with multiple occurrences, e.g., 0..* cardinality allowed for instances of each Information Element in the hierarchy. A typical example is the MPLS label stack entries model. An early NetFlow implementation used two Information Elements to represent the MPLS label stack entry: a "label stack entry position" followed by a "label stack value". However, several drawbacks were discovered. Firstly, the Information Elements in the Template Record had to be imposed so that the position would always precede the value. However, some encoding optimizations are based on the permutation of Information Element order. Secondly, a new semantic intelligence, not described in the information model, had to be hard-coded in the Collecting Process: the label value at the position "X" in the stack is contained in the "label stack value" Information Element following by a "label stack entry position" Information Element containing the value "X". Therefore, this model was abandoned. The selected solution in the IPFIX information model [RFC5102] is a long series of Information Elements: mplsTopLabelStackSection, mplsLabelStackSection2, mplsLabelStackSection3, mplsLabelStackSection4, mplsLabelStackSection5, mplsLabelStackSection6, mplsLabelStackSection7, mplsLabelStackSection8, mplsLabelStackSection9, mplsLabelStackSection10. While this model removes any ambiguity, it overloads the IPFIX information model with repetitive information. Furthermore, if mplsLabelStackSection11 is required, IANA [IANA-IPFIX] will not be able to assign the new Information Element next to the other ones in the registry, which might cause some confusion.

2.3. Structured Data Use Cases

Clearly, the MPLS label stack entries issue can best be solved by using a real structured data type composed of ("label stack entry position", "label stack value") pairs, potentially repeated multiple times in Flow Records, since this would be the most efficient from an information model point of view.
Top   ToC   RFC6313 - Page 9
   Some more examples enter the same category: how to encode the list of
   output interfaces in a multicast Flow, how to encode the list of BGP
   Autonomous Systems (AS) in a BGP Flow, how to encode the BGP
   communities in a BGP Flow, etc.

   The one-way delay passive measurement, which is described in the
   IPFIX applicability [RFC5472], is yet another example that would
   benefit from a structured data encoding.  Assuming synchronized
   clocks, the Collector can deduce the one-way delay between two
   Observation Points from the following two Information Elements,
   collected from two different Observation Points:

       - Packet arrival time: observationTimeMicroseconds [RFC5477]
       - Packet ID: digestHashValue [RFC5477]

   In practice, this implies that many pairs of
   (observationTimeMicroseconds, digestHashValue) must be exported for
   each Observation Point, even if Hash-Based Filtering [RFC5475] is
   used.  On top of that information, if the requirement is to
   understand the one-way delay per application type, the 5-tuple
   (source IP address, destination IP address, protocol, source port,
   destination port) would need to be added to every Flow Record.
   Instead of exporting this repetitive 5-tuple, as part of every single
   Flow Record a Flow Record composed of a structured data type such as
   the following would save a lot of bandwidth:

      5-tuple
                { observationTimeMicroseconds 1, digestHashValue 1 }
                { observationTimeMicroseconds 2, digestHashValue 2 }
                { observationTimeMicroseconds 3, digestHashValue 3 }
                { ...  , ... }
Top   ToC   RFC6313 - Page 10
   As a last example, here is a more complex case of hierarchical
   structured data encoding.  Consider the example scenario of an IPS
   (Intrusion Prevention System) alert data structure containing
   multiple participants, where each participant contains multiple
   attackers and multiple targets, with each target potentially composed
   of multiple applications, as depicted below:

      alert
          signatureId
          protocolIdentifier
          riskRating
          participant 1
              attacker 1
                  sourceIPv4Address
                  applicationId
              ...
              attacker N
                  sourceIPv4Address
                  applicationId
              target 1
                  destinationIPv4Address
                  applicationId 1
                  ...
                  applicationId n
              ...
              target N
                  destinationIPv4Address
                  applicationId 1
                  ...
                  applicationId n
          participant 2
              ...

   To export this information in IPFIX, the data would need to be
   flattened (thus, losing the hierarchical relationships) and a new
   IPFIX Template created for each alert, according to the number of
   applicationId elements in each target, the number of targets and
   attackers in each participant, and the number of participants in each
   alert.  Clearly, each Template will be unique to each alert, and a
   large amount of CPU, memory, and export bandwidth will be wasted
   creating, exporting, maintaining, and withdrawing the Templates.  See
   Appendix B for a specific example related to this case study.
Top   ToC   RFC6313 - Page 11

2.4. Specifications Summary

This document specifies an IPFIX extension to support hierarchical structured data and variable-length lists by defining three new Information Elements and three corresponding new abstract data types called basicList, subTemplateList, and subTemplateMultiList. These are defined in Sections 4.1 and 4.3. The three Structured Data Information Elements carry some semantic information so that the Collecting Process can understand the relationship between the different list elements. The semantic in the Structured Data Information Elements is provided in order to express the relationship among the multiple top-level list elements. As an example, if a list is composed of the elements (A,B,C), the semantic expresses the relationship among A, B, and C, regardless of whether A, B, and C are individual elements or a list of elements. It is important to note that whereas the Information Elements and abstract data types defined in the IPFIX information model [RFC5102] represent single values, these new abstract data types are structural in nature and primarily contain references to other Information Elements and to Templates. By referencing other Information Elements and Templates from an Information Element's data content, it is possible to define complex data structures such as variable-length lists and to specify hierarchical containment relationships between Templates. Therefore, this document prefers the more generic "Data Record" term to the "Flow Record" term. This document specifies three new abstract data types, which are basic blocks to represent structured data. However, this document does not comment on all possible combinations of basicList, subTemplateList, and subTemplateMultiList. Neither does it limit the possible combinations.

3. Terminology

IPFIX-specific terminology used in this document is defined in Section 2 of the IPFIX protocol specification [RFC5101] and Section 3 of the PSAMP protocol specification [RFC5476]. As in [RFC5101], these IPFIX-specific terms have the first letter of a word capitalized when used in this document.
Top   ToC   RFC6313 - Page 12

3.1. New Terminology

Structured Data Information Element One of the Information Elements supporting structured data, i.e., the basicList, subTemplateList, or subTemplateMultiList Information Elements specified in Section 4.3.

3.2. Conventions Used in This Document

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].


(page 12 continued on part 2)

Next Section