Tech-invite3GPPspaceIETF RFCsSIP
929190898887868584838281807978777675747372717069686766656463626160595857565554535251504948474645444342414039383736353433323130292827262524232221201918171615141312111009080706050403020100
in Index   Prev   Next

RFC 8456

Benchmarking Methodology for Software-Defined Networking (SDN) Controller Performance

Pages: 64
Informational
Part 3 of 4 – Pages 31 to 50
First   Prev   Next

Top   ToC   RFC8456 - Page 31   prevText

5.3. Security

5.3.1. Exception Handling

Objective: Determine the effects of handling error packets and notifications on performance tests. The impact MUST be measured for the following performance tests: 1. Path Provisioning Rate 2. Path Provisioning Time 3. Network Topology Change Detection Time Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. This test MUST be performed after obtaining the baseline measurement results for the performance tests listed above. 2. Ensure that the invalid messages are not dropped by the intermediate devices connecting the controller and Network Devices. Procedure: 1. Perform the above-listed performance tests, and send 1% of the messages from the Asynchronous Message Processing Rate test (Section 5.1.3) as invalid messages from the connected Network Devices emulated at the forwarding-plane test emulator. 2. Perform the above-listed performance tests, and send 2% of the messages from the Asynchronous Message Processing Rate test (Section 5.1.3) as invalid messages from the connected Network Devices emulated at the forwarding-plane test emulator. Note: Invalid messages can be frames with incorrect protocol fields or any form of failure notifications sent towards the controller.
Top   ToC   RFC8456 - Page 32
   Measurements:

      Measurements MUST be done as per the equation defined in the
      "Measurements" section of the corresponding test listed under
      "Objective".

   Reporting Format:

      The Exception Handling results MUST be reported in tabular format,
      with a column for each of the below parameters and row for each of
      the above-listed performance tests:

         -  Without Exceptions

         -  With 1% Exceptions

         -  With 2% Exceptions

5.3.2. Handling Denial-of-Service Attacks

Objective: Determine the effects of handling DoS attacks on performance and scalability tests. The impact MUST be measured for the following tests: 1. Path Provisioning Rate 2. Path Provisioning Time 3. Network Topology Change Detection Time 4. Network Discovery Size Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisite: This test MUST be performed after obtaining the baseline measurement results for the performance tests listed above.
Top   ToC   RFC8456 - Page 33
   Procedure:

      Perform the above-listed tests, and launch a DoS attack towards
      the controller while the trial is running.

      Note: DoS attacks can be launched on one of the following
      interfaces:

         1. Northbound (e.g., query for flow entries continuously on the
            northbound interface)

         2. Management (e.g., Ping requests to the controller's
            management interface)

         3. Southbound (e.g., TCP SYN messages on the southbound
            interface)

   Measurements:

      Measurements MUST be done as per the equation defined in the
      "Measurements" section of the corresponding test listed under
      "Objective".

   Reporting Format:

      The results regarding the handling of DoS attacks MUST be reported
      in tabular format, with a column for each of the below parameters
      and a row for each of the above-listed tests.

         -  Without any attacks

         -  With attacks

      The report should also specify the nature of the attack and the
      interface in question.
Top   ToC   RFC8456 - Page 34

5.4. Reliability

5.4.1. Controller Failover Time

Objective: Measure the time taken to switch from an active controller to the backup controller when the controllers work in redundancy mode and the active controller fails, defined as the interval starting when the active controller is brought down and ending with the first rediscovery message received from the new controller at its southbound interface. Reference Test Setup: This test SHOULD use the test setup illustrated in Section 3.2 of this document. Prerequisites: 1. Master controller election MUST be completed. 2. Nodes are connected to the controller cluster per the implemented redundancy mode (e.g., active-standby). 3. The controller cluster should have successfully completed the network topology discovery. 4. The Network Device MUST send all new flows to the controller when it receives them from the test traffic generator. 5. The controller should have learned the location of the destination (D1) at test traffic generator TP2.
Top   ToC   RFC8456 - Page 35
   Procedure:

      1. Send unidirectional traffic continuously with incremental
         sequence numbers and source addresses from test traffic
         generator TP1 at the rate at which the controller can process
         the traffic without any drops.

      2. Ensure that there are no packet drops observed at test traffic
         generator TP2.

      3. Bring down the active controller.

      4. Stop the trial when the first frame after the failover
         operation is received on test traffic generator TP2.

      5. Record the time at which the last valid frame was received (T1)
         at test traffic generator TP2 before the sequence error and the
         time at which the first valid frame was received (T2) after the
         sequence error at test traffic generator TP2.

   Measurements:

      Controller Failover Time = (T2 - T1)

      Packet Loss = Number of missing packet sequences

   Reporting Format:

      The Controller Failover Time results MUST be tabulated with the
      following information:

         -  Number of cluster nodes

         -  Redundancy mode

         -  Controller Failover Time

         -  Packet Loss

         -  Cluster keep-alive interval
Top   ToC   RFC8456 - Page 36

5.4.2. Network Re-provisioning Time

Objective: Measure the time taken by the controller to reroute traffic when there is a failure in existing traffic paths, defined as the interval starting with the first failure notification message received by the controller and ending with the last flow re-provisioning message sent by the controller at its southbound interface. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. A network with a specified number of nodes and redundant paths MUST be deployed. 2. The controller MUST know the location of test traffic generators TP1 and TP2. 3. Ensure that the controller does not pre-provision the alternate path in the emulated Network Devices at the forwarding-plane test emulator. Procedure: 1. Send bidirectional traffic continuously with a unique sequence number from test traffic generators TP1 and TP2. 2. Bring down a link or switch in the traffic path. 3. Stop the trial after receiving the first frame after network reconvergence. 4. Record the time of the last received frame prior to the frame loss at test traffic generator TP2 (TP2-Tlfr) and the time of the first frame received after the frame loss at test traffic generator TP2 (TP2-Tffr). There must be a gap in sequence numbers of these frames. 5. Record the time of the last received frame prior to the frame loss at test traffic generator TP1 (TP1-Tlfr) and the time of the first frame received after the frame loss at test traffic generator TP1 (TP1-Tffr).
Top   ToC   RFC8456 - Page 37
   Measurements:

      Forward Direction Path Re-provisioning Time (FDRT)
                                                 = (TP2-Tffr - TP2-Tlfr)

      Reverse Direction Path Re-provisioning Time (RDRT)
                                                 = (TP1-Tffr - TP1-Tlfr)

      Network Re-provisioning Time = (FDRT + RDRT)/2

      Forward Direction Packet Loss = Number of missing sequence frames
         at test traffic generator TP1

      Reverse Direction Packet Loss = Number of missing sequence frames
         at test traffic generator TP2

   Reporting Format:

      The Network Re-provisioning Time results MUST be tabulated with
      the following information:

         -  Number of nodes in the primary path

         -  Number of nodes in the alternate path

         -  Network Re-provisioning Time

         -  Forward Direction Packet Loss

         -  Reverse Direction Packet Loss

6. IANA Considerations

This document has no IANA actions.
Top   ToC   RFC8456 - Page 38

7. Security Considerations

The benchmarking tests described in this document are limited to the performance characterization of controllers in a lab environment with isolated networks. The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the controller. Special capabilities SHOULD NOT exist in the controller specifically for benchmarking purposes. Any implications for network security arising from the controller SHOULD be identical in the lab and in production networks.

8. References

8.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>. [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>. [RFC8455] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., and S. Banks, "Terminology for Benchmarking Software-Defined Networking (SDN) Controller Performance", RFC 8455, DOI 10.17487/RFC8455, October 2018, <https://www.rfc-editor.org/info/rfc8455>.

8.2. Informative References

[OpenFlow-Spec] ONF, "OpenFlow Switch Specification" Version 1.4.0 (Wire Protocol 0x05), October 2013, <https://www.opennetworking.org/wp-content/ uploads/2014/10/openflow-spec-v1.4.0.pdf>.
Top   ToC   RFC8456 - Page 39

Appendix A. Benchmarking Methodology Using OpenFlow Controllers

This section gives an overview of the OpenFlow protocol [OpenFlow-Spec] and provides a test methodology for benchmarking SDN Controllers supporting the OpenFlow southbound protocol. The OpenFlow protocol is used as an example to illustrate the methodologies defined in this document.

A.1. Protocol Overview

OpenFlow [OpenFlow-Spec] is an open standard protocol defined by the Open Networking Foundation (ONF) and used for programming the forwarding plane of network switches or routers via a centralized controller.

A.2. Messages Overview

The OpenFlow protocol supports three message types -- namely, controller-to-switch, asynchronous, and symmetric. Controller-to-switch messages are initiated by the controller and used to directly manage or inspect the state of the switch. These messages allow controllers to query/configure the switch ("features" messages, configuration messages), collect information from a switch (Read-State messages), send packets on a specified port of a switch (OFPT_PACKET_OUT messages), and modify the switch forwarding plane and state (Modify-State messages, Role-Request messages, etc.). Asynchronous messages are generated by the switch without a controller soliciting them. These messages allow switches to update controllers to denote an arrival of a new flow (OFPT_PACKET_IN messages), switch state changes ("flow-removed" messages, port-status messages), and errors (Error messages). Symmetric messages are generated in either direction without solicitation. These messages allow switches and controllers to set up a connection (Hello messages), verify liveness (Echo messages), and offer additional functionalities (Experimenter messages).

A.3. Connection Overview

The OpenFlow channel is used to exchange OpenFlow messages between an OpenFlow switch and an OpenFlow controller. The OpenFlow channel connection can be set up using plain TCP or TLS. By default, a switch establishes a single connection with the SDN Controller. A switch may establish multiple parallel connections to a single controller (auxiliary connection) or multiple controllers to handle controller failures and load balancing.
Top   ToC   RFC8456 - Page 40

A.4. Performance Benchmarking Tests

A.4.1. Network Topology Discovery Time

Procedure: Network Devices OpenFlow SDN Controller Application | | | | |<Initialize controller | | |app., NB and SB interfaces>| | | | |<Deploy network with | | | given no. of OF switches> | | | | | | OFPT_HELLO Exchange | | |<-------------------------->| | | | | | OFPT_PACKET_OUT with LLDP| | | to all switches| | (Tm1)|<---------------------------| | | | | | OFPT_PACKET_IN with LLDP| | | rcvd from Switch 1| | |--------------------------->| | | | | | OFPT_PACKET_IN with LLDP| | | rcvd from Switch 2| | |--------------------------->| | | . | | | . | | | | | | OFPT_PACKET_IN with LLDP| | | rcvd from Switch n| | (Tmn)|--------------------------->| | | | | | | <Wait for the expiry of| | | the Trial Duration (Td)>| | | | | | Query the controller for| | | discovered n/w topo. (Di)| | |<--------------------------| | | | | | <Compare the discovered| | | n/w topology and the| | | offered n/w topology>| | | |
Top   ToC   RFC8456 - Page 41
   Legend:

      NB: Northbound
      SB: Southbound
      OF: OpenFlow
      OFP: OpenFlow Protocol
      LLDP: Link-Layer Discovery Protocol
      Tm1: Time of reception of first LLDP message from controller
      Tmn: Time of last LLDP message sent to controller

   Discussion:

      The Network Topology Discovery Time can be obtained by calculating
      the time difference between the first OFPT_PACKET_OUT with an LLDP
      message received from the controller (Tm1) and the last
      OFPT_PACKET_IN with an LLDP message sent to the controller (Tmn)
      when the comparison is successful.
Top   ToC   RFC8456 - Page 42

A.4.2. Asynchronous Message Processing Time

Procedure: Network Devices OpenFlow SDN Controller Application | | | |OFPT_PACKET_IN with single | | |OFP match header | | (T0)|--------------------------->| | | | | |OFPT_PACKET_OUT with single | | | OFP action header | | (R0)|<---------------------------| | | . | | | . | | | . | | | | | |OFPT_PACKET_IN with single | | |OFP match header | | (Tn)|--------------------------->| | | | | |OFPT_PACKET_OUT with single | | | OFP action header | | (Rn)|<---------------------------| | | | | |<Wait for the expiry of the | | |Trial Duration> | | | | | |<Record the number of | | |OFPT_PACKET_INs/ | | |OFPT_PACKET_OUTs | | |exchanged (Nrx)> | | | | | Legend: T0,T1, ..Tn: transmit timestamps of OFPT_PACKET_IN messages R0,R1, ..Rn: receive timestamps of OFPT_PACKET_OUT messages Nrx: Number of successful OFPT_PACKET_IN/OFPT_PACKET_OUT message exchanges Discussion: The Asynchronous Message Processing Time will be obtained by calculating the sum of ((R0 - T0),(R1 - T1)..(Rn - Tn))/Nrx.
Top   ToC   RFC8456 - Page 43

A.4.3. Asynchronous Message Processing Rate

Procedure: Network Devices OpenFlow SDN Controller Application | | | |OFPT_PACKET_IN with single | | |OFP match header | | |--------------------------->| | | | | |OFPT_PACKET_OUT with single | | | OFP action header | | |<---------------------------| | | | | | . | | | . | | | . | | | | | |OFPT_PACKET_IN with single | | |OFP match header | | |--------------------------->| | | | | |OFPT_PACKET_OUT with single | | | OFP action header | | |<---------------------------| | | | | |<Repeat the steps until | | |the expiry of the | | |Trial Duration> | | | | | |<Record the number of OFP | | (Ntx1)|match headers sent> | | | | | |<Record the number of OFP | | (Nrx1)|action headers rcvd> | | | | | Note: The Ntx1 on initial trials should be greater than Nrx1. Repeat the trials until the Nrxn for two consecutive trials equals (+/-P%).
Top   ToC   RFC8456 - Page 44
   Discussion:

      Using a single procedure, this test will measure two benchmarks:

         1. The Maximum Asynchronous Message Processing Rate will be
            obtained by calculating the maximum OFPT_PACKET_OUTs (Nrxn)
            received from the controller(s) across n trials.

         2. The Loss-Free Asynchronous Message Processing Rate will be
            obtained by calculating the maximum OFPT_PACKET_OUTs
            received from the controller(s) when the Loss Ratio equals
            zero.  The Loss Ratio is obtained by calculating
            1 - Nrxn/Ntxn.

A.4.4. Reactive Path Provisioning Time

Procedure: Test Traffic Test Traffic Network Devices OpenFlow Generator TP1 Generator TP2 Controller | | | | | |G-ARP (D1) | | | |--------------------->| | | | | | | | |OFPT_PACKET_IN(D1) | | | |-------------------->| | | | | |Traffic (S1,D1) | | (Tsf1)|----------------------------------->| | | | | | | | | | | | | | | | |OFPT_PACKET_IN(S1,D1)| | | |-------------------->| | | | | | | | FLOW_MOD(D1) | | | |<--------------------| | | | | | |Traffic (S1,D1) | | | (Tdf1)|<---------------------| | | | | |
Top   ToC   RFC8456 - Page 45
   Legend:

      G-ARP: Gratuitous ARP message
      Tsf1: Time of first frame sent from TP1
      Tdf1: Time of first frame received from TP2

   Discussion:

      The Reactive Path Provisioning Time can be obtained by finding the
      time difference between the transmit and receive times of the
      traffic (Tsf1 - Tdf1).
Top   ToC   RFC8456 - Page 46

A.4.5. Proactive Path Provisioning Time

Procedure: Test Traffic Test Traffic Network Devices OpenFlow SDN Generator TP1 Generator TP2 Controller Application | | | | | | | | | | | | | |<Install flow| | | | | for S1,D1> | | |G-ARP (D1) | | | | |-------------->| | | | | | | | | | |OFPT_PACKET_IN(D1)| | | | |----------------->| | | | | | | |Traffic (S1,D1) | | | (Tsf1)|--------------------------->| | | | | | | | | | | FLOW_MOD(D1) | | | | |<-----------------| | | | | | | | |Traffic (S1,D1)| | | | (Tdf1)|<--------------| | | | | | | | Legend: G-ARP: Gratuitous ARP message Tsf1: Time of first frame sent from TP1 Tdf1: Time of first frame received from TP2 Discussion: The Proactive Path Provisioning Time can be obtained by finding the time difference between the transmit and receive times of the traffic (Tsf1 - Tdf1).
Top   ToC   RFC8456 - Page 47

A.4.6. Reactive Path Provisioning Rate

Procedure: Test Traffic Test Traffic Network Devices OpenFlow Generator TP1 Generator TP2 Controller | | | | | | | | | | | | | |G-ARP (D1..Dn) | | | |--------------------| | | | | | | | |OFPT_PACKET_IN(D1..Dn)| | | |--------------------->| | | | | |Traffic (S1..Sn,D1..Dn) | | |--------------------------------->| | | | | | | | |OFPT_PACKET_IN(S1..Sn,| | | | D1..Dn)| | | |--------------------->| | | | | | | | FLOW_MOD(S1) | | | |<---------------------| | | | | | | | FLOW_MOD(D1) | | | |<---------------------| | | | | | | | FLOW_MOD(S2) | | | |<---------------------| | | | | | | | FLOW_MOD(D2) | | | |<---------------------| | | | . | | | | . | | | | | | | | FLOW_MOD(Sn) | | | |<---------------------| | | | | | | | FLOW_MOD(Dn) | | | |<---------------------| | | | | | | Traffic (S1..Sn, | | | | D1..Dn)| | | |<-------------------| | | | | | | | | |
Top   ToC   RFC8456 - Page 48
   Legend:

      G-ARP: Gratuitous ARP message
      D1..Dn: Destination Endpoint 1, Destination Endpoint 2 ...,
              Destination Endpoint n
      S1..Sn: Source Endpoint 1, Source Endpoint 2 ...,
              Source Endpoint n

   Discussion:

      The Reactive Path Provisioning Rate can be obtained by finding the
      total number of frames received at test traffic generator TP2
      after the Trial Duration.
Top   ToC   RFC8456 - Page 49

A.4.7. Proactive Path Provisioning Rate

Procedure: Test Traffic Test Traffic Network Devices OpenFlow SDN Generator TP1 Generator TP2 Controller Application | | | | | | |G-ARP (D1..Dn) | | | | |--------------->| | | | | | | | | | |OFPT_PACKET_IN | | | | | (D1..Dn)| | | | |---------------->| | | | | | | |Traffic (S1..Sn,D1..Dn) | | | (Tsf1)|---------------------------->| | | | | | | | | | | |<Install flow| | | | | for S1,D1> | | | | | | | | | | . | | | | |<Install flow| | | | | for Sn,Dn> | | | | | | | | | FLOW_MOD(S1) | | | | |<----------------| | | | | | | | | | FLOW_MOD(D1) | | | | |<----------------| | | | | | | | | | . | | | | | FLOW_MOD(Sn) | | | | |<----------------| | | | | | | | | | FLOW_MOD(Dn) | | | | |<----------------| | | | | | | | |Traffic (S1..Sn,| | | | | D1..Dn)| | | | (Tdf1)|<---------------| | | | | | | |
Top   ToC   RFC8456 - Page 50
   Legend:

      G-ARP: Gratuitous ARP message
      D1..Dn: Destination Endpoint 1, Destination Endpoint 2 ...,
              Destination Endpoint n
      S1..Sn: Source Endpoint 1, Source Endpoint 2 ...,
              Source Endpoint n

   Discussion:

      The Proactive Path Provisioning Rate can be obtained by finding
      the total number of frames received at test traffic generator TP2
      after the Trial Duration.

A.4.8. Network Topology Change Detection Time

Procedure: Network Devices OpenFlow SDN Controller Application | | | | | <Bring down a link in | | | Switch S1>| | | | T0 |PORT_STATUS with link down | | | from S1 | | |--------------------------->| | | | | |First OFPT_PACKET_OUT with | | |LLDP to OF switch | | T1 |<---------------------------| | | | | | | <Record time of first| | | OFPT_PACKET_OUT with| | | LLDP T1>| | | | Discussion: The Network Topology Change Detection Time can be obtained by finding the difference between the time that OpenFlow Switch S1 sends the PORT_STATUS message (T0) and the time that the OpenFlow controller sends the first topology rediscovery message (T1) to OpenFlow switches.


(next page on part 4)

Next Section