Tech-invite3GPPspaceIETFspace
96959493929190898887868584838281807978777675747372717069686766656463626160595857565554535251504948474645444342414039383736353433323130292827262524232221201918171615141312111009080706050403020100
in Index   Prev   Next

RFC 8456

Benchmarking Methodology for Software-Defined Networking (SDN) Controller Performance

Pages: 64
Informational
Part 2 of 4 – Pages 11 to 30
First   Prev   Next

Top   ToC   RFC8456 - Page 11   prevText

5. Benchmarking Tests

5.1. Performance

5.1.1. Network Topology Discovery Time

Objective: Measure the time taken by the controller(s) to determine the complete network topology, defined as the interval starting with the first discovery message from the controller(s) at its southbound interface and ending with all features of the static topology determined. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller MUST support network discovery. 2. The tester should be able to retrieve the discovered topology information through either the controller's management interface or northbound interface to determine if the discovery was successful and complete. 3. Ensure that the controller's topology rediscovery timeout has been set to the maximum value, to avoid initiation of the rediscovery process in the middle of the test. Procedure: 1. Ensure that the controller is operational and that its network applications, northbound interface, and southbound interface are up and running. 2. Establish the network connections between the controller and the Network Devices. 3. Record the time for the first discovery message (Tm1) received from the controller at the forwarding-plane test emulator interface (I1).
Top   ToC   RFC8456 - Page 12
      4. Query the controller every t seconds (the RECOMMENDED value for
         t is 3) to obtain the discovered network topology information
         through the northbound interface or the management interface,
         and compare it with the deployed network topology information.

      5. Stop the trial when the discovered topology information matches
         the deployed network topology or when the discovered topology
         information returns the same details for three consecutive
         queries.

      6. Record the time for the last discovery message (Tmn) sent to
         the controller from the forwarding-plane test emulator
         interface (I1) when the trial completes successfully (e.g.,
         when the topology matches).

   Measurements:

      Topology Discovery Time (DT1) = Tmn - Tm1

                                              DT1 + DT2 + DT3 .. DTn
      Average Topology Discovery Time (TDm) = -----------------------
                                                   Total Trials

                                               SUM[SQUAREOF(DTi - TDm)]
      Topology Discovery Time Variance (TDv) = ------------------------
                                                   Total Trials - 1

   Reporting Format:

      The Topology Discovery Time results MUST be reported in tabular
      format, with a row for each successful iteration.  The last row of
      the table indicates the Topology Discovery Time variance, and the
      previous row indicates the Average Topology Discovery Time.

      If this test is repeated with a varying number of nodes over the
      same topology, the results SHOULD be reported in the form of a
      graph.  The X coordinate SHOULD be the number of nodes (N), and
      the Y coordinate SHOULD be the Average Topology Discovery Time.
Top   ToC   RFC8456 - Page 13

5.1.2. Asynchronous Message Processing Time

Objective: Measure the time taken by the controller(s) to process an asynchronous message, defined as the interval starting with an asynchronous message from a Network Device after the discovery of all the devices by the controller(s) and ending with a response message from the controller(s) at its southbound interface. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisite: The controller MUST have successfully completed the network topology discovery for the connected Network Devices. Procedure: 1. Generate asynchronous messages from every connected Network Device to the SDN Controller, one at a time in series from the forwarding-plane test emulator for the Trial Duration. 2. Record every request transmit time (T1) and the corresponding response received time (R1) at the forwarding-plane test emulator interface (I1) for every successful message exchange. Measurements: Asynchronous Message Processing Time (APT1) = SUM{Ri} - SUM{Ti} ----------------------- Nrx Where Nrx is the total number of successful messages exchanged. Average Asynchronous Message Processing Time = APT1 + APT2 + APT3 .. APTn -------------------------- Total Trials
Top   ToC   RFC8456 - Page 14
     Asynchronous Message Processing Time Variance (TAMv) =
                                              SUM[SQUAREOF(APTi - TAMm)]
                                              --------------------------
                                                    Total Trials - 1

        Where TAMm is the Average Asynchronous Message Processing Time.

   Reporting Format:

      The Asynchronous Message Processing Time results MUST be reported
      in tabular format, with a row for each iteration.  The last row of
      the table indicates the Asynchronous Message Processing Time
      variance, and the previous row indicates the Average Asynchronous
      Message Processing Time.

      The report SHOULD capture the following information, in addition
      to the configuration parameters captured per Section 4.8:

         -  Successful messages exchanged (Nrx)

         -  Percentage of unsuccessful messages exchanged, computed
            using the formula ((1 - Nrx/Ntx) * 100), where Ntx is the
            total number of messages transmitted to the controller

      If this test is repeated with a varying number of nodes with the
      same topology, the results SHOULD be reported in the form of a
      graph.  The X coordinate SHOULD be the number of nodes (N), and
      the Y coordinate SHOULD be the Average Asynchronous Message
      Processing Time.

5.1.3. Asynchronous Message Processing Rate

Objective: Measure the number of responses to asynchronous messages (a new flow arrival notification message, link down, etc.) for which the controller(s) performed processing and replied with a valid and productive (non-trivial) response message. Using a single procedure, this test will measure the following two benchmarks on the Asynchronous Message Processing Rate (see Section 2.3.1.3 of [RFC8455]): 1. Maximum Asynchronous Message Processing Rate 2. Loss-Free Asynchronous Message Processing Rate
Top   ToC   RFC8456 - Page 15
      Here, two benchmarks are determined through a series of trials
      where the number of messages sent to the controller(s) and the
      responses received from the controller(s) are counted over the
      Trial Duration.  The message response rate and the Message Loss
      Ratio are calculated for each trial.

   Reference Test Setup:

      This test SHOULD use one of the test setups illustrated in
      Section 3.1 or Section 3.2 of this document.

   Prerequisites:

      1. The controller(s) MUST have successfully completed the network
         topology discovery for the connected Network Devices.

      2. Choose and record the Trial Duration (Td), the sending rate
         STEP size, the tolerance on equality for two consecutive trials
         (P%), and the maximum possible message-sending rate (Ntx1/Td).

   Procedure:

      1. Generate asynchronous messages continuously at the maximum
         possible rate on the established connections from all the
         emulated/simulated Network Devices for the given Trial
         Duration (Td).

      2. Record the total number of responses received (Nrx1) from the
         controller as well as the number of messages sent (Ntx1) to the
         controller within the Trial Duration (Td).

      3. Calculate the Asynchronous Message Processing Rate (APR1) and
         the Message Loss Ratio (Lr1).  Ensure that the controller(s)
         has returned to normal operation.

      4. Repeat the trial by reducing the asynchronous message-sending
         rate used in the last trial by the STEP size.

      5. Continue repeating the trials and reducing the sending rate
         until both the maximum value of Nrxn (number of responses
         received from the controller) and the Nrxn corresponding to a
         Loss Ratio of zero have been found.
Top   ToC   RFC8456 - Page 16
      6. The trials corresponding to the benchmark levels MUST be
         repeated using the same asynchronous message rates until the
         responses received from the controller are equal (+/-P%) for
         two consecutive trials.

      7. Record the number of responses received (Nrxn) from the
         controller as well as the number of messages sent (Ntxn) to the
         controller in the last trial.

   Measurements:

                                                    Nrxn
      Asynchronous Message Processing Rate (APRn) = -----
                                                     Td

      Maximum Asynchronous Message Processing Rate = MAX(APRn) for all n

                                                  Nrxn
      Asynchronous Message Loss Ratio (Lrn) = 1 - -----
                                                  Ntxn

      Loss-Free Asynchronous Message Processing Rate = MAX(APRn)
         given Lrn = 0

   Reporting Format:

      The Asynchronous Message Processing Rate results MUST be reported
      in tabular format, with a row for each trial.

      The table should report the following information, in addition to
      the configuration parameters captured per Section 4.8, with
      columns:

         -  Offered rate (Ntxn/Td)

         -  Asynchronous Message Processing Rate (APRn)

         -  Loss Ratio (Lr)

         -  Benchmark at this iteration (blank for none, Maximum
            Asynchronous Message Processing Rate, Loss-Free Asynchronous
            Message Processing Rate)

      The results MAY be presented in the form of a graph.  The X axis
      SHOULD be the offered rate, and dual Y axes would represent the
      Asynchronous Message Processing Rate and the Loss Ratio,
      respectively.
Top   ToC   RFC8456 - Page 17
      If this test is repeated with a varying number of nodes over the
      same topology, the results SHOULD be reported in the form of a
      graph.  The X axis SHOULD be the number of nodes (N), and the
      Y axis SHOULD be the Asynchronous Message Processing Rate.  Both
      the Maximum Asynchronous Message Processing Rate and the Loss-Free
      Asynchronous Message Processing Rate should be plotted for each N.

5.1.4. Reactive Path Provisioning Time

Objective: Measure the time taken by the controller to set up a path reactively between source and destination nodes, defined as the interval starting with the first flow provisioning request message received by the controller(s) at its southbound interface and ending with the last flow provisioning response message sent from the controller(s) at its southbound interface. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. The number of Network Devices in the path is a parameter of the test that may be varied from two to the maximum discovery size in repetitions of this test. Prerequisites: 1. The controller MUST contain the network topology information for the deployed network topology. 2. The controller should know the location of the destination endpoint for which the path has to be provisioned. This can be achieved through dynamic learning or static provisioning. 3. Ensure that the default action for "flow miss" in the Network Device is configured to "send to controller". 4. Ensure that each Network Device in a path requires the controller to make the forwarding decision while paving the entire path.
Top   ToC   RFC8456 - Page 18
   Procedure:

      1. Send a single traffic stream from test traffic generator TP1 to
         test traffic generator TP2.

      2. Record the time of the first flow provisioning request message
         sent to the controller (Tsf1) from the Network Device at the
         forwarding-plane test emulator interface (I1).

      3. Wait for the arrival of the first traffic frame at the endpoint
         (i.e., test traffic generator TP2) or the expiry of the Trial
         Duration (Td).

      4. Record the time of the last flow provisioning response message
         received from the controller (Tdf1) to the Network Device at
         the forwarding-plane test emulator interface (I1).

   Measurements:

      Reactive Path Provisioning Time (RPT1) = Tdf1 - Tsf1

      Average Reactive Path Provisioning Time =
                                              RPT1 + RPT2 + RPT3 .. RPTn
                                              --------------------------
                                                      Total Trials

      Reactive Path Provisioning Time Variance (TRPv) =
                                              SUM[SQUAREOF(RPTi - TRPm)]
                                              --------------------------
                                                     Total Trials - 1

         Where TRPm is the Average Reactive Path Provisioning Time.

   Reporting Format:

      The Reactive Path Provisioning Time results MUST be reported in
      tabular format, with a row for each iteration.  The last row of
      the table indicates the Reactive Path Provisioning Time variance,
      and the previous row indicates the Average Reactive Path
      Provisioning Time.

      The report should capture the following information, in addition
      to the configuration parameters captured per Section 4.8:

         -  Number of Network Devices in the path
Top   ToC   RFC8456 - Page 19

5.1.5. Proactive Path Provisioning Time

Objective: Measure the time taken by the controller to set up a path proactively between source and destination nodes, defined as the interval starting with the first proactive flow provisioned in the controller(s) at its northbound interface and ending with the last flow provisioning response message sent from the controller(s) at its southbound interface. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller MUST contain the network topology information for the deployed network topology. 2. The controller should know the location of the destination endpoint for which the path has to be provisioned. This can be achieved through dynamic learning or static provisioning. 3. Ensure that the default action for "flow miss" in the Network Device is "drop". Procedure: 1. Send a single traffic stream from test traffic generator TP1 to test traffic generator TP2. 2. Install the flow entries so that the traffic travels from test traffic generator TP1 until it reaches test traffic generator TP2 through the controller's northbound interface or management interface. 3. Wait for the arrival of the first traffic frame at test traffic generator TP2 or the expiry of the Trial Duration (Td). 4. Record the time when the proactive flow is provisioned in the controller (Tsf1) at the management-plane test emulator interface (I2). 5. Record the time of the last flow provisioning message received from the controller (Tdf1) at the forwarding-plane test emulator interface (I1).
Top   ToC   RFC8456 - Page 20
   Measurements:

      Proactive Flow Provisioning Time (PPT1) = Tdf1 - Tsf1

      Average Proactive Path Provisioning Time =
                                              PPT1 + PPT2 + PPT3 .. PPTn
                                              --------------------------
                                                      Total Trials

      Proactive Path Provisioning Time Variance (TPPv) =
                                              SUM[SQUAREOF(PPTi - TPPm)]
                                              --------------------------
                                                   Total Trials - 1

         Where TPPm is the Average Proactive Path Provisioning Time.

   Reporting Format:

      The Proactive Path Provisioning Time results MUST be reported in
      tabular format, with a row for each iteration.  The last row of
      the table indicates the Proactive Path Provisioning Time variance,
      and the previous row indicates the Average Proactive Path
      Provisioning Time.

      The report should capture the following information, in addition
      to the configuration parameters captured per Section 4.8:

         -  Number of Network Devices in the path
Top   ToC   RFC8456 - Page 21

5.1.6. Reactive Path Provisioning Rate

Objective: Measure the maximum number of independent paths a controller can concurrently establish per second between source and destination nodes reactively, defined as the number of paths provisioned per second by the controller(s) at its southbound interface for the flow provisioning requests received for path provisioning at its southbound interface between the start of the test and the expiry of the given Trial Duration. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller MUST contain the network topology information for the deployed network topology. 2. The controller should know the location of destination addresses for which the paths have to be provisioned. This can be achieved through dynamic learning or static provisioning. 3. Ensure that the default action for "flow miss" in the Network Device is configured to "send to controller". 4. Ensure that each Network Device in a path requires the controller to make the forwarding decision while provisioning the entire path. Procedure: 1. Send traffic with unique source and destination addresses from test traffic generator TP1. 2. Record the total number of unique traffic frames (Ndf) received at test traffic generator TP2 within the Trial Duration (Td).
Top   ToC   RFC8456 - Page 22
   Measurements:

                                                Ndf
      Reactive Path Provisioning Rate (RPR1) = ------
                                                Td

      Average Reactive Path Provisioning Rate =
                                              RPR1 + RPR2 + RPR3 .. RPRn
                                              --------------------------
                                                     Total Trials

      Reactive Path Provisioning Rate Variance (RPPv) =
                                              SUM[SQUAREOF(RPRi - RPPm)]
                                              --------------------------
                                                    Total Trials - 1

         Where RPPm is the Average Reactive Path Provisioning Rate.

   Reporting Format:

      The Reactive Path Provisioning Rate results MUST be reported in
      tabular format, with a row for each iteration.  The last row of
      the table indicates the Reactive Path Provisioning Rate variance,
      and the previous row indicates the Average Reactive Path
      Provisioning Rate.

      The report should capture the following information, in addition
      to the configuration parameters captured per Section 4.8:

         -  Number of Network Devices in the path

         -  Offered rate
Top   ToC   RFC8456 - Page 23

5.1.7. Proactive Path Provisioning Rate

Objective: Measure the maximum number of independent paths a controller can concurrently establish per second between source and destination nodes proactively, defined as the number of paths provisioned per second by the controller(s) at its southbound interface for the paths requested in its northbound interface between the start of the test and the expiry of the given Trial Duration. The measurement is based on data-plane observations of successful path activation. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller MUST contain the network topology information for the deployed network topology. 2. The controller should know the location of destination addresses for which the paths have to be provisioned. This can be achieved through dynamic learning or static provisioning. 3. Ensure that the default action for "flow miss" in the Network Device is "drop". Procedure: 1. Send traffic continuously with unique source and destination addresses from test traffic generator TP1. 2. Install corresponding flow entries so that the traffic travels from simulated sources at test traffic generator TP1 until it reaches the simulated destinations at test traffic generator TP2 through the controller's northbound interface or management interface. 3. Record the total number of unique traffic frames (Ndf) received at test traffic generator TP2 within the Trial Duration (Td).
Top   ToC   RFC8456 - Page 24
   Measurements:

                                                 Ndf
      Proactive Path Provisioning Rate (PPR1) = ------
                                                 Td

      Average Proactive Path Provisioning Rate =
                                              PPR1 + PPR2 + PPR3 .. PPRn
                                              --------------------------
                                                      Total Trials

      Proactive Path Provisioning Rate Variance (PPPv) =
                                              SUM[SQUAREOF(PPRi - PPPm)]
                                              -------------------------
                                                   Total Trials - 1

         Where PPPm is the Average Proactive Path Provisioning Rate.

   Reporting Format:

      The Proactive Path Provisioning Rate results MUST be reported in
      tabular format, with a row for each iteration.  The last row of
      the table indicates the Proactive Path Provisioning Rate variance,
      and the previous row indicates the Average Proactive Path
      Provisioning Rate.

      The report should capture the following information, in addition
      to the configuration parameters captured per Section 4.8:

         -  Number of Network Devices in the path

         -  Offered rate
Top   ToC   RFC8456 - Page 25

5.1.8. Network Topology Change Detection Time

Objective: Measure the amount of time taken by the controller to detect any changes in the network topology, defined as the interval starting with the notification message received by the controller(s) at its southbound interface and ending with the first topology rediscovery message sent from the controller(s) at its southbound interface. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller MUST have successfully discovered the network topology information for the deployed network topology. 2. The periodic network discovery operation should be configured to twice the Trial Duration (Td) value. Procedure: 1. Trigger a topology change event by bringing down an active Network Device in the topology. 2. Record the time when the first topology change notification is sent to the controller (Tcn) at the forwarding-plane test emulator interface (I1). 3. Stop the trial when the controller sends the first topology rediscovery message to the Network Device or the expiry of the Trial Duration (Td). 4. Record the time when the first topology rediscovery message is received from the controller (Tcd) at the forwarding-plane test emulator interface (I1).
Top   ToC   RFC8456 - Page 26
   Measurements:

      Network Topology Change Detection Time (TDT1) = Tcd - Tcn

      Average Network Topology Change Detection Time =
                                              TDT1 + TDT2 + TDT3 .. TDTn
                                              --------------------------
                                                      Total Trials

      Network Topology Change Detection Time Variance (NTDv) =
                                              SUM[SQUAREOF(TDTi - NTDm)]
                                              --------------------------
                                                    Total Trials - 1

         Where NTDm is the Average Network Topology Change
            Detection Time.

   Reporting Format:

      The Network Topology Change Detection Time results MUST be
      reported in tabular format, with a row for each iteration.  The
      last row of the table indicates the Network Topology Change
      Detection Time variance, and the previous row indicates the
      Average Network Topology Change Detection Time.

5.2. Scalability

5.2.1. Control Sessions Capacity

Objective: Measure the maximum number of control sessions the controller can maintain, defined as the number of sessions that the controller can accept from Network Devices, starting with the first control session and ending with the last control session that the controller(s) accepts at its southbound interface. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: None
Top   ToC   RFC8456 - Page 27
   Procedure:

      1. Establish control connections with the controller from every
         Network Device emulated in the forwarding-plane test emulator.

      2. Stop the trial when the controller starts dropping the control
         connections.

      3. Record the number of successful connections established (CCn)
         with the controller at the forwarding-plane test emulator.

   Measurement:

      Control Sessions Capacity = CCn

   Reporting Format:

      The Control Sessions Capacity results MUST be reported in addition
      to the configuration parameters captured per Section 4.8.

5.2.2. Network Discovery Size

Objective: Measure the network size (number of nodes, links, and hosts) that a controller can discover, defined as the size of a network that the controller(s) can discover, starting with a network topology provided by the user for discovery and ending with the number of nodes, links, and hosts that the controller(s) were able to successfully discover. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller MUST support automatic network discovery. 2. The tester should be able to retrieve the discovered topology information through either the controller's management interface or northbound interface.
Top   ToC   RFC8456 - Page 28
   Procedure:

      1. Establish the network connections between the controller and
         the network nodes.

      2. Query the controller every t seconds (the RECOMMENDED value for
         t is 30) to obtain the discovered network topology information
         through the northbound interface or the management interface.

      3. Stop the trial when the discovered network topology information
         remains the same as that of the last two query responses.

      4. Compare the obtained network topology information with the
         deployed network topology information.

      5. If the comparison is successful, increase the number of nodes
         by 1 and repeat the trial.
         If the comparison is unsuccessful, decrease the number of nodes
         by 1 and repeat the trial.

      6. Continue the trial until the comparison (step 5) is successful.

      7. Record the number of nodes for the last trial run (Ns) where
         the topology comparison was successful.

   Measurement:

       Network Discovery Size = Ns

   Reporting Format:

      The Network Discovery Size results MUST be reported in addition to
      the configuration parameters captured per Section 4.8.
Top   ToC   RFC8456 - Page 29

5.2.3. Forwarding Table Capacity

Objective: Measure the maximum number of flow entries a controller can manage in its Forwarding Table. Reference Test Setup: This test SHOULD use one of the test setups illustrated in Section 3.1 or Section 3.2 of this document. Prerequisites: 1. The controller's Forwarding Table should be empty. 2. "Flow idle time" MUST be set to a higher or infinite value. 3. The controller MUST have successfully completed network topology discovery. 4. The tester should be able to retrieve the Forwarding Table information through either the controller's management interface or northbound interface. Procedures: o Reactive Flow Provisioning Mode: 1. Send bidirectional traffic continuously with unique source and destination addresses from test traffic generators TP1 and TP2 at the Asynchronous Message Processing Rate of the controller. 2. Query the controller at a regular interval (e.g., every 5 seconds) for the number of learned flow entries from its northbound interface. 3. Stop the trial when the retrieved value is constant for three consecutive iterations, and record the value received from the last query (Nrp).
Top   ToC   RFC8456 - Page 30
      o  Proactive Flow Provisioning Mode:

         1. Install unique flows continuously through the controller's
            northbound interface or management interface until a failure
            response is received from the controller.

         2. Record the total number of successful responses (Nrp).

         Note:

         Some controller designs for Proactive Flow Provisioning mode
         may require the switch to send flow setup requests in order to
         generate flow setup responses.  In such cases, it is
         recommended to generate bidirectional traffic for the
         provisioned flows.

   Measurements:

      Proactive Flow Provisioning Mode:

         Max Flow Entries = Total number of flows provisioned (Nrp)

      Reactive Flow Provisioning Mode:

         Max Flow Entries = Total number of learned flow entries (Nrp)

      Forwarding Table Capacity = Max Flow Entries

   Reporting Format:

      The Forwarding Table Capacity results MUST be tabulated with the
      following information, in addition to the configuration parameters
      captured per Section 4.8:

         -  Provisioning Type (Proactive/Reactive)


(next page on part 3)

Next Section