tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Glossaries     Architecture     IMS     UICC    |    search     info

RFC 5810

 
 
 

Forwarding and Control Element Separation (ForCES) Protocol Specification

Part 2 of 5, p. 10 to 31
Prev RFC Part       Next RFC Part

 


prevText      Top      Up      ToC       Page 10 
4.  Overview

   The reader is referred to the framework document [RFC3746], and in
   particular, Sections 3 and 4, for an architectural overview and an
   explanation of how the ForCES protocol fits in.  There may be some
   content overlap between the framework document and this section in
   order to provide clarity.  This document is authoritative on the
   protocol, whereas [RFC3746] is authoritative on the architecture.

Top      Up      ToC       Page 11 
4.1.  Protocol Framework

   Figure 1 below is reproduced from the framework document for clarity.
   It shows an NE with two CEs and two FEs.

                            ---------------------------------------
                            | ForCES Network Element              |
     --------------   Fc    | --------------      --------------  |
     | CE Manager |---------+-|     CE 1   |------|    CE 2    |  |
     --------------         | |            |  Fr  |            |  |
           |                | --------------      --------------  |
           | Fl             |         |  |    Fp       /          |
           |                |       Fp|  |----------| /           |
           |                |         |             |/            |
           |                |         |             |             |
           |                |         |     Fp     /|----|        |
           |                |         |  /--------/      |        |
     --------------     Ff  | --------------      --------------  |
     | FE Manager |---------+-|     FE 1   |  Fi  |     FE 2   |  |
     --------------         | |            |------|            |  |
                            | --------------      --------------  |
                            |   |  |  |  |          |  |  |  |    |
                            ----+--+--+--+----------+--+--+--+-----
                                |  |  |  |          |  |  |  |
                                |  |  |  |          |  |  |  |
                                  Fi/f                   Fi/f

          Fp: CE-FE interface
          Fi: FE-FE interface
          Fr: CE-CE interface
          Fc: Interface between the CE manager and a CE
          Ff: Interface between the FE manager and an FE
          Fl: Interface between the CE manager and the FE manager
          Fi/f: FE external interface

                  Figure 1: ForCES Architectural Diagram

   The ForCES protocol domain is found in the Fp reference points.  The
   Protocol Element configuration reference points, Fc and Ff, also play
   a role in the booting up of the ForCES protocol.  The protocol
   element configuration (indicated by reference points Fc, Ff, and Fl
   in [RFC3746]) is out of scope of the ForCES protocol but is touched
   on in this document in discussion of FEM and CEM since it is an
   integral part of the protocol pre-association phase.

Top      Up      ToC       Page 12 
   Figure 2 below shows further breakdown of the Fp interfaces by means
   of the example of an MPLS QoS-enabled Network Element.

         -------------------------------------------------
         |       |       |       |       |       |       |
         |OSPF   |RIP    |BGP    |RSVP   |LDP    |. . .  |
         |       |       |       |       |       |       |
         -------------------------------------------------    CE
         |               ForCES Interface                |
         -------------------------------------------------
                                 ^   ^
                                 |   |
                         ForCES  |   |data
                         control |   |packets
                         messages|   |(e.g., routing packets)
                                 |   |
                                 v   v
         -------------------------------------------------
         |               ForCES Interface                |
         -------------------------------------------------    FE
         |       |       |       |       |       |       |
         |LPM Fwd|Meter  |Shaper |MPLS   |Classi-|. . .  |
         |       |       |       |       |fier   |       |
         -------------------------------------------------

                 Figure 2: Examples of CE and FE Functions

   The ForCES interface shown in Figure 2 constitutes two pieces: the PL
   and the TML.

Top      Up      ToC       Page 13 
   This is depicted in Figure 3 below.

         +------------------------------------------------
         |               CE PL                           |
         +------------------------------------------------
         |              CE TML                           |
         +------------------------------------------------
                                   ^
                                   |
                      ForCES       |   (i.e.,  ForCES data + control
                      PL           |    packets )
                      messages     |
                      over         |
                      specific     |
                      TML          |
                      encaps       |
                      and          |
                      transport    |
                                   |
                                   v
         +------------------------------------------------
         |              FE TML                           |
         +------------------------------------------------
         |               FE PL                           |
         +------------------------------------------------

                        Figure 3: ForCES Interface

   The PL is in fact the ForCES protocol.  Its semantics and message
   layout are defined in this document.  The TML layer is necessary to
   connect two ForCES PLs as shown in Figure 3 above.  The TML is out of
   scope for this document but is within scope of ForCES.  This document
   defines requirements the PL needs the TML to meet.

   Both the PL and the TML are standardized by the IETF.  While only one
   PL is defined, different TMLs are expected to be standardized.  To
   interoperate, the TML at the CE and FE are expected to conform to the
   same definition.

   On transmit, the PL delivers its messages to the TML.  The local TML
   delivers the message to the destination TML.  On receive, the TML
   delivers the message to its destination PL.

4.1.1.  The PL

   The PL is common to all implementations of ForCES and is standardized
   by the IETF as defined in this document.  The PL is responsible for
   associating an FE or CE to an NE.  It is also responsible for tearing

Top      Up      ToC       Page 14 
   down such associations.  An FE uses the PL to transmit various
   subscribed-to events to the CE PL as well as to respond to various
   status requests issued from the CE PL.  The CE configures both the FE
   and associated LFBs' operational parameters using the PL.  In
   addition, the CE may send various requests to the FE to activate or
   deactivate it, reconfigure its HA parameterization, subscribe to
   specific events, etc.  More details can be found in Section 7.

4.1.2.  The TML

   The TML transports the PL messages.  The TML is where the issues of
   how to achieve transport-level reliability, congestion control,
   multicast, ordering, etc. are handled.  It is expected that more than
   one TML will be standardized.  The various possible TMLs could vary
   their implementations based on the capabilities of underlying media
   and transport.  However, since each TML is standardized,
   interoperability is guaranteed as long as both endpoints support the
   same TML.  All ForCES protocol layer implementations MUST be portable
   across all TMLs, because all TMLs MUST have the top-edge semantics
   defined in this document.

4.1.3.  The FEM/CEM Interface

   The FEM and CEM components, although valuable in the setup and
   configurations of both the PL and TML, are out of scope of the ForCES
   protocol.  The best way to think of them is as configurations/
   parameterizations for the PL and TML before they become active (or
   even at runtime based on implementation).  In the simplest case, the
   FE or CE reads a static configuration file.  RFC 3746 has a more
   detailed description on how the FEM and CEM could be used.  The pre-
   association phase, where the CEM and FEM can be used, are described
   briefly in Section 4.2.1.

   An example of typical things the FEM/CEM could configure would be
   TML-specific parameterizations such as:

   a.  How the TML connection should happen (for example, what IP
       addresses to use, transport modes, etc.)

   b.  The ID for the FE (FEID) or CE (CEID) (which would also be issued
       during the pre-association phase)

   c.  Security parameterization such as keys, etc.

   d.  Connection association parameters

Top      Up      ToC       Page 15 
   An example of connection association parameters might be:

   o  simple parameters: send up to 3 association messages every 1
      second

   o  complex parameters: send up to 4 association messages with
      increasing exponential timeout

4.2.  ForCES Protocol Phases

   ForCES, in relation to NEs, involves two phases: the pre-association
   phase where configuration/initialization/bootup of the TML and PL
   layer happens, and the post-association phase where the ForCES
   protocol operates to manipulate the parameters of the FEs.

                       CE sends Association Setup
           +---->--->------------>---->---->---->------->----+
           |                                                 Y
           ^                                                 |
           |                                                 Y
       +---+-------+                                     +-------------+
       |FE pre-    |                                     | FE post-    |
       |association|    CE sends Association Teardown    | association |
       |phase      |<------- <------<-----<------<-------+ phase       |
       |           |                                     |             |
       +-----------+                                     +-------------+
             ^                                               Y
             |                                               |
             +-<---<------<-----<------<----<---------<------+
                           FE loses association

                     Figure 4: The FE Protocol Phases

   In the mandated case, once associated, the FE may forward packets
   depending on the configuration of its specific LFBs.  An FE that is
   associated to a CE will continue sending packets until it receives an
   Association Teardown Message or until it loses association.  An
   unassociated FE MAY continue sending packets when it has a high
   availability capability.  The extra details are explained in
   Section 8 and not discussed here to allow for a clear explanation of
   the basics.

   The FE state transitions are controlled by means of the FE Object LFB
   FEState component, which is defined in [RFC5812], Section 5.1, and
   also explained in Section 7.3.2.

Top      Up      ToC       Page 16 
   The FE initializes in the FEState OperDisable.  When the FE is ready
   to process packets in the data path, it transitions itself to the
   OperEnable state.

   The CE may decide to pause the FE after it already came up as
   OperEnable.  It does this by setting the FEState to AdminDisable.
   The FE stays in the AdminDisable state until it is explicitly
   configured by the CE to transition to the OperEnable state.

   When the FE loses its association with the CE, it may go into the
   pre-association phase depending on the programmed policy.  For the FE
   to properly complete the transition to the AdminDisable state, it
   MUST stop packet forwarding and this may impact multiple LFBS.  How
   this is achieved is outside the scope of this specification.

4.2.1.  Pre-association

   The ForCES interface is configured during the pre-association phase.
   In a simple setup, the configuration is static and is typically read
   from a saved configuration file.  All the parameters for the
   association phase are well known after the pre-association phase is
   complete.  A protocol such as DHCP may be used to retrieve the
   configuration parameters instead of reading them from a static
   configuration file.  Note, this will still be considered static pre-
   association.  Dynamic configuration may also happen using the Fc, Ff,
   and Fl reference points (refer to [RFC3746]).  Vendors may use their
   own proprietary service discovery protocol to pass the parameters.
   Essentially, only guidelines are provided here and the details are
   left to the implementation.

   The following are scenarios reproduced from the framework document to
   show a pre-association example.

Top      Up      ToC       Page 17 
      <----Ff ref pt--->              <--Fc ref pt------->
      FE Manager      FE                CE Manager    CE
       |              |                 |             |
       |              |                 |             |
    (security exchange)               (security exchange)
      1|<------------>| authentication 1|<----------->|authentication
       |              |                 |             |
     (FE ID, components)              (CE ID, components)
      2|<-------------| request        2|<------------|request
       |              |                 |             |
      3|------------->| response       3|------------>|response
      (corresponding CE ID)          (corresponding FE ID)
       |              |                 |             |
       |              |                 |             |

        Figure 5: Examples of a Message Exchange over the Ff and Fc
                             Reference Points


      <-----------Fl ref pt-------------->            |

      FE Manager      FE               CE Manager     CE
       |              |                 |             |
       |              |                 |             |
      (security exchange)               |             |
      1|<------------------------------>|             |
       |              |                 |             |
      (a list of CEs and their components)            |
      2|<-------------------------------|             |
       |              |                 |             |
      (a list of FEs and their components)            |
      3|------------------------------->|             |
       |              |                 |             |
       |              |                 |             |

    Figure 6: Example of a Message Exchange over the Fl Reference Point

   Before the transition to the association phase, the FEM will have
   established contact with a CEM component.  Initialization of the
   ForCES interface will have completed, and authentication as well as
   capability discovery may be complete.  Both the FE and CE would have
   the necessary information for connecting to each other for
   configuration, accounting, identification, and authentication
   purposes.  To summarize, at the completion of this stage both sides
   have all the necessary protocol parameters such as timers, etc.  The
   Fl reference point may continue to operate during the association
   phase and may be used to force a disassociation of an FE or CE.  The
   specific interactions of the CEM and the FEM that are part of the

Top      Up      ToC       Page 18 
   pre-association phase are out of scope; for this reason, these
   details are not discussed any further in this specification.  The
   reader is referred to the framework document [RFC3746] for a slightly
   more detailed discussion.

4.2.2.  Post-association

   In this phase, the FE and CE components communicate with each other
   using the ForCES protocol (PL over TML) as defined in this document.
   There are three sub-phases:

   o  Association Setup Stage

   o  Established Stage

   o  Association Lost Stage

4.2.2.1.  Association Setup Stage

   The FE attempts to join the NE.  The FE may be rejected or accepted.
   Once granted access into the NE, capabilities exchange happens with
   the CE querying the FE.  Once the CE has the FE capability
   information, the CE can offer an initial configuration (possibly to
   restore state) and can query certain components within either an LFB
   or the FE itself.

   More details are provided in Section 4.4.

   On successful completion of this stage, the FE joins the NE and is
   moved to the Established Stage.

4.2.2.2.  Established Stage

   In this stage, the FE is continuously updated or queried.  The FE may
   also send asynchronous event notifications to the CE or synchronous
   heartbeat notifications if programmed to do so.  This stage continues
   until a termination occurs, either due to loss of connectivity or due
   to a termination initiated by either the CE or the FE.

   Refer to the section on protocol scenarios, Section 4.4, for more
   details.

4.2.2.3.  Association Lost Stage

   In this stage, both or either the CE or FE declare the other side is
   no longer associated.  The disconnection could be initiated by either
   party for administrative purposes but may also be driven by
   operational reasons such as loss of connectivity.

Top      Up      ToC       Page 19 
   A core LFB known as the FE Protocol Object (FEPO) is defined (refer
   to Appendix B and Section 7.3.1).  FEPO defines various timers that
   can be used in conjunction with a traffic-sensitive heartbeat
   mechanism (Section 4.3.3) to detect loss of connectivity.

   The loss of connectivity between TMLs does not indicate a loss of
   association between respective PL layers.  If the TML cannot repair
   the transport loss before the programmed FEPO timer thresholds
   associated with the FE is exceeded, then the association between the
   respective PL layers will be lost.

   FEPO defines several policies that can be programmed to define
   behavior upon a detected loss of association.  The FEPO's programmed
   CE failover policy (refer to Sections 8, 7.3.1, 4.3.3, and B) defines
   what takes place upon loss of association.

   For this version of the protocol (as defined in this document), the
   FE, upon re-association, MUST discard any state it has as invalid and
   retrieve new state.  This approach is motivated by a desire for
   simplicity (as opposed to efficiency).

4.3.  Protocol Mechanisms

   Various semantics are exposed to the protocol users via the PL header
   including transaction capabilities, atomicity of transactions, two-
   phase commits, batching/parallelization, high availability, and
   failover as well as command pipelines.

   The EM (Execution Mode) flag, AT (Atomic Transaction) flag, and TP
   (Transaction Phase) flag as defined in the common header
   (Section 6.1) are relevant to these mechanisms.

4.3.1.  Transactions, Atomicity, Execution, and Responses

   In the master-slave relationship, the CE instructs one or more FEs on
   how to execute operations and how to report the results.

   This section details the different modes of execution that a CE can
   order the FE(s) to perform, as defined in Section 4.3.1.1.  It also
   describes the different modes a CE can ask the FE(s) to use for
   formatting the responses after processing the operations as
   requested.  These modes relate to the transactional two-phase commit
   operations.

Top      Up      ToC       Page 20 
4.3.1.1.  Execution

   There are 3 execution modes that can be requested for a batch of
   operations spanning one or more LFB selectors (refer to
   Section 7.1.5) in one protocol message.  The EM flag defined in the
   common header (Section 6.1) selects the execution mode for a protocol
   message, as below:

   a.  execute-all-or-none

   b.  continue-execute-on-failure

   c.  execute-until-failure

4.3.1.1.1.  execute-all-or-none

   When set to this mode of execution, independent operations in a
   message MAY be targeted at one or more LFB selectors within an FE.
   All these operations are executed serially, and the FE MUST have no
   execution failure for any of the operations.  If any operation fails
   to execute, then all the previous ones that have been executed prior
   to the failure will need to be undone.  That is, there is rollback
   for this mode of operation.

   Refer to Section 4.3.1.2.2 for how this mode is used in cases of
   transactions.  In such a case, no operation is executed unless a
   commit is issued by the CE.

   Care should be taken on how this mode is used because a mis-
   configuration could result in traffic losses.  To add certainty to
   the success of an operation, one should use this mode in a
   transactional operation as described in Section 4.3.1.2.2

4.3.1.1.2.  continue-execute-on-failure

   If several independent operations are targeted at one or more LFB
   selectors, execution continues for all operations at the FE even if
   one or more operations fail.

4.3.1.1.3.  execute-until-failure

   In this mode, all operations are executed on the FE sequentially
   until the first failure.  The rest of the operations are not executed
   but operations already completed are not undone.  That is, there is
   no rollback in this mode of operation.

Top      Up      ToC       Page 21 
4.3.1.2.  Transaction and Atomicity

4.3.1.2.1.  Transaction Definition

   A transaction is defined as a collection of one or more ForCES
   operations within one or more PL messages that MUST meet the ACIDity
   properties [ACID], defined as:

   Atomicity:   In a transaction involving two or more discrete pieces
                of information, either all of the pieces are committed
                or none are.

   Consistency: A transaction either creates a new and valid state of
                data or, if any failure occurs, returns all data to the
                state it was in before the transaction was started.

   Isolation:   A transaction in process and not yet committed MUST
                remain isolated from any other transaction.

   Durability:  Committed data is saved by the system such that, even in
                the event of a failure and a system restart, the data is
                available in its correct state.

   There are cases where the CE knows exact memory and implementation
   details of the FE such as in the case of an FE-CE pair from the same
   vendor where the FE-CE pair is tightly coupled.  In such a case, the
   transactional operations may be simplified further by extra
   computation at the CE.  This view is not discussed further other than
   to mention that it is not disallowed.

   As defined above, a transaction is always atomic and MAY be

   a.  Within an FE alone
       Example: updating multiple tables that are dependent on each
       other.  If updating one fails, then any that were already updated
       MUST be undone.

   b.  Distributed across the NE
       Example: updating table(s) that are inter-dependent across
       several FEs (such as L3 forwarding-related tables).

4.3.1.2.2.  Transaction Protocol

   By use of the execution mode, as defined in Section 4.3.1.1, the
   protocol has provided a mechanism for transactional operations within
   one stand-alone message.  The 'execute-all-or-none' mode can meet the
   ACID requirements.

Top      Up      ToC       Page 22 
   For transactional operations of multiple messages within one FE or
   across FEs, a classical transactional protocol known as two-phase
   commit (2PC) [2PCREF] is supported by the protocol to achieve the
   transactional operations utilizing Config messages (Section 7.6.1).

   The COMMIT and TRCOMP operations in conjunction with the AT and the
   TP flags in the common header (Section 6.1) are provided for 2PC-
   based transactional operations spanning multiple messages.

   The AT flag, when set, indicates that this message belongs to an
   Atomic Transaction.  All messages for a transaction operation MUST
   have the AT flag set.  If not set, it means that the message is a
   stand-alone message and does not participate in any transaction
   operation that spans multiple messages.

   The TP flag indicates the Transaction Phase to which this message
   belongs.  There are 4 possible phases for a transactional operation
   known as:

      SOT (Start of Transaction)

      MOT (Middle of Transaction)

      EOT (End of Transaction)

      ABT (Abort)

   The COMMIT operation is used by the CE to signal to the FE(s) to
   commit a transaction.  When used with an ABT TP flag, the COMMIT
   operation signals the FE(s) to roll back (i.e., un-COMMIT) a
   previously committed transaction.

   The TRCOMP operation is a small addition to the classical 2PC
   approach.  TRCOMP is sent by the CE to signal to the FE(s) that the
   transaction they have COMMITed is now over.  This allows the FE(s) an
   opportunity to clear state they may have kept around to perform a
   roll back (if it became necessary).

   A transaction operation is started with a message in which the TP
   flag is set to Start of Transaction (SOT).  Multi-part messages,
   after the first one, are indicated by the Middle of Transaction (MOT)
   flag.  All messages from the CE MUST set the AlwaysACK flag
   (Section 6) to solicit responses from the FE(s).

   Before the CE issues a commit (described further below), the FE MUST
   only validate that the operation can be executed but not execute it.

Top      Up      ToC       Page 23 
      Any failure notified by an FE causes the CE to abort the
      transaction on all FEs involved in the transaction.  This is
      achieved by sending a Config message with an ABT flag and a COMMIT
      operation.

      If there are no failures by any participating FE, the transaction
      commitment phase is signaled from the CE to the FE by an End of
      Transaction (EOT) configuration message with a COMMIT operation.

   The FE MUST respond to the CE's EOT message.  There are two possible
   failure scenarios in which the CE MUST abort the transaction (as
   described above):

   a.  If any participating FE responds with a failure message in
       relation to the transaction.

   b.  If no response is received from a participating FE within a
       specified timeout.

   If all participating FEs respond with a success indicator within the
   expected time, then the CE MUST issue a TRCOMP operation to all
   participating FEs.  An FE MUST NOT respond to a TRCOMP.

   Note that a transactional operation is generically atomic; therefore,
   it requires that the execution modes of all messages in a transaction
   operation should always be kept the same and be set to 'execute-all-
   or-none'.  If the EM flag is set to other execution modes, it will
   result in a transaction failure.

   As noted above, a transaction may span multiple messages.  It is up
   to the CE to keep track of the different outstanding messages making
   up a transaction.  As an example, the correlator field could be used
   to mark transactions and a sequence field to label the different
   messages within the same atomic transaction, but this is out of scope
   and up to implementations.

4.3.1.2.3.  Recovery

   Any of the participating FEs or the CE or the associations between
   them may fail after the EOT Response message has been sent by the FE
   but before the CE has received all the responses, e.g., if the EOT
   response never reaches the CE.

   In this protocol revision, as indicated in Section 4.2.2.3, an FE
   losing an association would be required to get entirely new state
   from the newly associated CE upon a re-association.  Although this
   approach is simplistic and provides likeliness of losing data path

Top      Up      ToC       Page 24 
   traffic, it is a design choice to avoid the additional complexity of
   managing graceful restarts.  The HA mechanisms (Section 8) are
   provided to allow for a continuous operation in case of FE failures.

   Flexibility is provided on how to react when an FE loses association.
   This is dictated by the CE failover policy (refer to Section 8 and
   Section 7.3).

4.3.1.2.4.  Transaction Messaging Example

   This section illustrates an example of how a successful two-phase
   commit between a CE and an FE would look in the simple case.

         FE PL                                                  CE PL

           |                                                      |
           | (1) Config, SOT,AT, EM=All-or-None, OP= SET/DEL,etc  |
           |<-----------------------------------------------------|
           |                                                      |
           | (2) ACKnowledge                                      |
           |----------------------------------------------------->|
           |                                                      |
           | (3) Config, MOT,AT, EM=All-or-None, OP= SET/DEL,etc  |
           |<-----------------------------------------------------|
           |                                                      |
           | (4) ACKnowledge                                      |
           |----------------------------------------------------->|
           |                                                      |
           | (5) Config, MOT,AT, EM=All-or-None, OP= SET/DEL,etc  |
           |<-----------------------------------------------------|
           |                                                      |
           | (6) ACKnowledge                                      |
           |----------------------------------------------------->|
           .                                                      .
           .                                                      .
           .                                                      .
           .                                                      .
           |                                                      |
           | (N) Config, EOT,AT, EM=All-or-None, OP= COMMIT       |
           |<-----------------------------------------------------|
           |                                                      |
           | (N+1)Config-response, ACKnowledge, OP=COMMIT-RESPONSE|
           |----------------------------------------------------->|
           |                                                      |
           | (N+2) Config, OP=TRCOMP                              |
           |<-----------------------------------------------------|

                  Figure 7: Example of a Two-Phase Commit

Top      Up      ToC       Page 25 
   For the scenario illustrated above:

   o  In step 1, the CE issues a Config message with an operation of
      choice like a DEL or SET.  The transaction flags are set to
      indicate a Start of Transaction (SOT), Atomic Transaction (AT),
      and execute-all-or-none.

   o  The FE validates that it can execute the request successfully and
      then issues an acknowledgment back to the CE in step 2.

   o  In step 3, the same sort of construct as in step 1 is repeated by
      the CE with the transaction flag changed to Middle of Transaction
      (MOT).

   o  The FE validates that it can execute the request successfully and
      then issues an acknowledgment back to the CE in step 4.

   o  The CE-FE exchange continues in the same manner until all the
      operations and their parameters are transferred to the FE.  This
      happens in step (N-1).

   o  In step N, the CE issues a commit.  A commit is a Config message
      with an operation of type COMMIT.  The transaction flag is set to
      End of Transaction (EOT).  Essentially, this is an "empty" message
      asking the FE to execute all the operations it has gathered since
      the beginning of the transaction (message #1).

   o  The FE at this point executes the full transaction.  It then
      issues an acknowledgment back to the CE in step (N+1) that
      contains a COMMIT-RESPONSE.

   o  The CE in this case has the simple task of issuing a TRCOMP
      operation to the FE in step (N+2).

4.3.2.  Scalability

   It is desirable that the PL not become the bottleneck when larger
   bandwidth pipes become available.  To pick a hypothetical example in
   today's terms, if a 100-Gbps pipe is available and there is
   sufficient work, then the PL should be able to take advantage of this
   and use all of the 100-Gbps pipe.  Two mechanisms have been provided
   to achieve this.  The first one is batching and the second one is a
   command pipeline.

Top      Up      ToC       Page 26 
   Batching is the ability to send multiple commands (such as Config) in
   one Protocol Data Unit (PDU).  The size of the batch will be affected
   by, among other things, the path MTU.  The commands may be part of
   the same transaction or may be part of unrelated transactions that
   are independent of each other.

   Command pipelining allows for pipelining of independent transactions
   that do not affect each other.  Each independent transaction could
   consist of one or more batches.

4.3.2.1.  Batching

   There are several batching levels at different protocol hierarchies.

   o  Multiple PL PDUs can be aggregated under one TML message.

   o  Multiple LFB classes and instances (as indicated in the LFB
      selector) can be addressed within one PL PDU.

   o  Multiple operations can be addressed to a single LFB class and
      instance.

4.3.2.2.  Command Pipelining

   The protocol allows any number of messages to be issued by the CE
   before the corresponding acknowledgments (if requested) have been
   returned by the FE.  Hence, pipelining is inherently supported by the
   protocol.  Matching responses with requests messages can be done
   using the correlator field in the message header.

4.3.3.  Heartbeat Mechanism

   Heartbeats (HBs) between FEs and CEs are traffic sensitive.  An HB is
   sent only if no PL traffic is sent between the CE and FE within a
   configured interval.  This has the effect of reducing the amount of
   HB traffic in the case of busy PL periods.

   An HB can be sourced by either the CE or FE.  When sourced by the CE,
   a response can be requested (similar to the ICMP ping protocol).  The
   FE can only generate HBs in the case of being configured to do so by
   the CE.  Refer to Section 7.3.1 and Section 7.10 for details.

Top      Up      ToC       Page 27 
4.3.4.  FE Object and FE Protocol LFBs

   All PL messages operate on LFB constructs, as this provides more
   flexibility for future enhancements.  This means that maintenance and
   configurability of FEs, NE, and the ForCES protocol itself MUST be
   expressed in terms of this LFB architecture.  For this reason,
   special LFBs are created to accommodate this need.

   In addition, this shows how the ForCES protocol itself can be
   controlled by the very same type of structures (LFBs) it uses to
   control functions such as IP forwarding, filtering, etc.

   To achieve this, the following specialized LFBs are introduced:

   o  FE Protocol LFB, which is used to control the ForCES protocol.

   o  FE Object LFB, which is used to control components relative to the
      FE itself.  Such components include FEState [RFC5812], vendor,
      etc.

   These LFBs are detailed in Section 7.3.

4.4.  Protocol Scenarios

   This section provides a very high level description of sample message
   sequences between a CE and an FE.  For protocol message encoding
   refer to Section 6.1, and for the semantics of the protocol refer to
   Section 4.3.

4.4.1.  Association Setup State

   The associations among CEs and FEs are initiated via the Association
   Setup message from the FE.  If a Setup Request is granted by the CE,
   a successful Setup Response message is sent to the FE.  If CEs and
   FEs are operating in an insecure environment, then the security
   associations have to be established between them before any
   association messages can be exchanged.  The TML MUST take care of
   establishing any security associations.

   This is typically followed by capability query, topology query, etc.
   When the FE is ready to start processing the data path, it sets the
   FEO FEState component to OperEnable (refer to [RFC5812] for details)
   and reports it to the CE as such when it is first queried.
   Typically, the FE is expected to be ready to process the data path
   before it associates, but there may be rare cases where it needs time
   do some pre-processing -- in such a case, the FE will start in the
   OperDisable state and when it is ready will transition to the
   OperEnable state.  An example of an FE starting in OperDisable then

Top      Up      ToC       Page 28 
   transitioning to OperEnable is illustrated in Figure 8.  The CE could
   at any time also disable the FE's data path operations by setting the
   FEState to AdminDisable.  The FE MUST NOT process packets during this
   state unless the CE sets the state back to OperEnable.  These
   sequences of messages are illustrated in Figure 8 below.

           FE PL                  CE PL

             |                       |
             |   Asso Setup Req      |
             |---------------------->|
             |                       |
             |   Asso Setup Resp     |
             |<----------------------|
             |                       |
             | LFBx Query capability |
             |<----------------------|
             |                       |
             | LFBx Query Resp       |
             |---------------------->|
             |                       |
             | FEO Query (Topology)  |
             |<----------------------|
             |                       |
             | FEO Query Resp        |
             |---------------------->|
             |                       |
             | FEO OperEnable Event  |
             |---------------------->|
             |                       |
             |  Config FEO Adminup   |
             |<----------------------|
             |                       |
             | FEO Config-Resp       |
             |---------------------->|
             |                       |

   Figure 8: Message Exchange between CE and FE to Establish
   an NE Association

   On successful completion of this state, the FE joins the NE.

Top      Up      ToC       Page 29 
4.4.2.  Association Established State or Steady State

   In this state, the FE is continuously updated or queried.  The FE may
   also send asynchronous event notifications to the CE, synchronous
   Heartbeat messages, or Packet Redirect message to the CE.  This
   continues until a termination (or deactivation) is initiated by
   either the CE or FE.  Figure 9 below, helps illustrate this state.

Top      Up      ToC       Page 30 
           FE PL                          CE PL

             |                              |
             |    Heartbeat                 |
             |<---------------------------->|
             |                              |
             |   Heartbeat                  |
             |----------------------------->|
             |                              |
             | Config-set LFBy (Event sub.) |
             |<-----------------------------|
             |                              |
             |     Config Resp LFBy         |
             |----------------------------->|
             |                              |
             |  Config-set LFBx Component   |
             |<-----------------------------|
             |                              |
             |     Config Resp  LFBx        |
             |----------------------------->|
             |                              |
             |Config-Query LFBz (Stats)     |
             |<--------------------------- -|
             |                              |
             |    Query Resp LFBz           |
             |----------------------------->|
             |                              |
             |    FE Event Report           |
             |----------------------------->|
             |                              |
             |  Config-Del LFBx Component   |
             |<-----------------------------|
             |                              |
             |     Config Resp LFBx         |
             |----------------------------->|
             |                              |
             |    Packet Redirect LFBx      |
             |----------------------------->|
             |                              |
             |    Heartbeat                 |
             |<-----------------------------|
             .                              .
             .                              .
             |                              |

   Figure 9: Message Exchange between CE and FE during
   Steady-State Communication

Top      Up      ToC       Page 31 
   Note that the sequence of messages shown in the figure serve only as
   examples and the message exchange sequences could be different from
   what is shown in the figure.  Also, note that the protocol scenarios
   described in this section do not include all the different message
   exchanges that would take place during failover.  That is described
   in the HA section (Section 8).



(page 31 continued on part 3)

Next RFC Part