Internet Engineering Task Force (IETF) B. Decraene Request for Comments: 6198 France Telecom Category: Informational P. Francois ISSN: 2070-1721 UCL C. Pelsser IIJ Z. Ahmad Orange Business Services A.J. Elizondo Armengol Telefonica I+D T. Takeda NTT April 2011 Requirements for the Graceful Shutdown of BGP Sessions
AbstractThe Border Gateway Protocol (BGP) is heavily used in Service Provider networks for both Internet and BGP/MPLS VPN services. For resiliency purposes, redundant routers and BGP sessions can be deployed to reduce the consequences of an Autonomous System Border Router (ASBR) or BGP session breakdown on customers' or peers' traffic. However, simply taking down or even bringing up a BGP session for maintenance purposes may still induce connectivity losses during the BGP convergence. This is no longer satisfactory for new applications (e.g., voice over IP, online gaming, VPN). Therefore, a solution is required for the graceful shutdown of a (set of) BGP session(s) in order to limit the amount of traffic loss during a planned shutdown. This document expresses requirements for such a solution. Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 5741. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc6198.
Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 1. Introduction ....................................................2 2. Conventions Used in This Document ...............................3 3. Problem Statement ...............................................4 3.1. Example of Undesirable BGP Routing Behavior ................4 3.2. Causes of Packet Loss ......................................5 4. Terminology .....................................................6 5. Goals and Requirements ..........................................7 6. Security Considerations ........................................10 7. References .....................................................10 7.1. Normative References ......................................10 7.2. Informative References ....................................10 Acknowledgments ...................................................11 Appendix A. Reference BGP Topologies ..............................12 A.1. EBGP Topologies ...........................................12 A.2. IBGP Topologies ...........................................15 A.3. Routing Decisions .........................................19 RFC4271] is heavily used in Service Provider networks for both Internet and BGP/MPLS VPN services [RFC4364]. For resiliency purposes, redundant routers and BGP sessions can be deployed to reduce the consequences of an Autonomous System Border Router (ASBR) or BGP session breakdown on customers' or peers' traffic. We place ourselves in the context where a Service Provider performs a maintenance operation and needs to shut down one or multiple BGP peering link(s) or a whole ASBR. If an alternate path is available within the Autonomous System (AS), the requirement is to avoid or reduce customer or peer traffic loss during the BGP convergence.
Indeed, as an alternate path is available in the AS, it should be made possible to reroute the customer or peer traffic on this backup path before the BGP session(s) is/are torn down, the nominal path withdrawn, and the forwarding stopped. The requirements also cover the subsequent re-establishment of the BGP session as even this "UP" case can currently trigger route loss, and thus traffic loss, at some routers. BGP [RFC4271] and MP-BGP [RFC4760] do not currently have a mechanism to gracefully migrate traffic from one BGP next-hop to another without interrupting the flow of traffic. When a BGP session is taken down, BGP behaves as if there were a sudden link or router failure and withdraws the prefixes learned over that session, which may trigger traffic loss. While still being advertised as reachable, there is no mechanism to advertise to its BGP peers that the prefix will soon be unreachable. When applicable, such mechanism would reduce or prevent traffic loss. It would typically be applicable in case of a maintenance operation requiring the shutdown of a forwarding resource. Typical examples would be a link or line card maintenance, replacement, or upgrade. It may also be applicable for a software upgrade, as it may involve a firmware reset on the line cards and hence forwarding interruption. The introduction of route reflectors (RRs) as per [RFC4456] to solve scalability issues bound to Internal BGP (IBGP) full-meshes has worsened the duration of routing convergence as some route reflectors may hide the backup path. Thus, depending on RR topology, more IBGP hops may be involved in the IBGP convergence. Note that these planned maintenance operations cannot be addressed by Graceful Restart (GR) extensions [RFC4724] as GR only applies when the forwarding is preserved during the control plane restart. On the contrary, graceful shutdown applies when the forwarding is interrupted. Also, note that some protocols are already considering such a graceful shutdown procedure (e.g., GMPLS in [RFC5817]). A metric of success is the degree to which such a mechanism eliminates traffic loss during maintenance operations. RFC2119].
RFC4271], when one (or many) BGP session(s) are shut down, a BGP NOTIFICATION message is sent to the peer and the session is then closed. A protocol convergence is then triggered both by the local router and by the peer. Alternate paths to the destination are selected, if known. If those alternate paths are not known prior to the BGP session shutdown, additional BGP convergence steps are required in each AS to search for an alternate path. This behavior is not satisfactory in a maintenance situation because the traffic that was directed towards the removed next-hops may be lost until the end of the BGP convergence. As it is a planned operation, a make-before-break solution should be made possible. As maintenance operations are frequent in large networks [Reliable], the global availability of the network is significantly impaired by this BGP maintenance issue. ' AS1 ' AS2 ' /-----------ASBR1--- / \ / \ CUST R1 \ / Z/z \ / \-----------ASBR2--- ' AS1 ' AS2 ' Figure 1. Dual-Attached Customer
Before the maintenance, packets for destination Z/z use the ASBR1- CUST link because R1 selects ASBR1's route based on the IGP cost. Let's assume the Service Provider wants to shut down the ASBR1-CUST link for maintenance purposes. Currently, when the shutdown is performed on ASBR1, the following steps are performed: 1. ASBR1 withdraws its prefix Z/z to its route reflector, R1. 2. R1 runs its decision process, selects the route from ASBR2, and advertises the new path to ASBR1. 3. ASBR1 runs its decision process and recovers the reachability of Z/z. Traffic is lost at step 1 when ASBR1 looses its route until step 3 when it discovers a new path. Note that this is a simplified description for illustrative purposes. In a bigger AS, multiple steps of BGP convergence may be required to find and select the best alternate path (e.g., ASBR1 may be chosen based on a higher LOCAL_PREF, hierarchical route reflectors may be used, etc.). When multiple BGP routers are involved and plenty of prefixes are affected, the recovery process can take longer than application requirements.
Forwarding Information Bases (FIBs) entries. This can lead to forwarding loops, which result in both link congestion and packet drops. The duration of these transient micro-loops is dependent on the IBGP topology (e.g., number of route reflectors between ingress and egress ASBR), implementation differences among router platforms (which result in differences in the time taken to update specific prefix in the FIB), and forwarding mode (hop-by-hop IP forwarding versus tunneling). Note that when an IP lookup is only performed on entry to the AS, for example, prior to entry into a tunnel across the AS, micro-loops will not occur. An example of this is when BGP is being used as the routing protocol for MPLS VPN as defined in [RFC4364]. Note that [RFC5715] defines a framework for loop-free convergence. It has been written in the context of IP fast reroute for link state IGP [RFC5714], but some concepts are also of interest for BGP convergence.
The solution mechanism MUST significantly reduce and, ideally, eliminate packet loss. A trade-off may be made between the degree of packet loss and the simplicity of the solution. b) An Internet-wide convergence is OPTIONAL. However, if the initiator AS and the neighbor AS(es) have a backup path, they SHOULD be able to gracefully converge before the nominal path is shut down. c) The proposed solution SHOULD be applicable to any kind of BGP sessions (External BGP (EBGP), IBGP, IBGP route reflector client, EBGP confederations, EBGP multi hop, MultiProtocol BGP extension, etc.) and any address family. If a BGP implementation allows the closing or enabling of a subset of Address Family Identifiers (AFIs) carried in an MP-BGP session, this mechanism MAY be applicable to this subset of AFIs. Depending on the kind of session, there may be some variations in the proposed solution in order to fulfill the requirements. The following cases should be handled in priority: - The shutdown of an inter-AS link and therefore the shutdown of an EBGP session; - The shutdown of an ASBR and therefore the shutdown of all its BGP sessions. Service Providers and platforms implementing a graceful shutdown solution should note that in BGP/MPLS VPN as per [RFC4364], the Provider Edge - Customer Edge (PE-CE) routing can be performed by protocols other than BGP (e.g., static routes, RIPv2, OSPF, IS-IS). This is out of scope of this document. d) The proposed solution SHOULD NOT change the BGP convergence behavior for the ASes exterior to the maintenance process, namely, ASes other than the initiator AS and its neighbor AS(es). e) An incremental deployment on a per-AS or per-BGP session basis MUST be made possible. In case of partial deployment, the proposed solution SHOULD incrementally improve the maintenance process. It should be noted that in an inter-domain relation, one AS may have more incentive to use graceful shutdown than the other. Similarly, in a BGP/MPLS VPN environment, it's much easier to upgrade the PE routers than the CE ones, mainly because there is at least an order of magnitude more CE and CE locations than PE and PE locations. As a consequence, when
splitting the cost of the solution between the g-shut initiator and the g-shut neighbor, the solution SHOULD favor a low-cost solution on the neighbor AS side in order to reduce the impact on the g-shut neighbor. Impact should be understood as a generic term that includes first hardware, then software, then configuration upgrade. f) Redistribution or advertisement of (static) IP routes into BGP SHOULD also be covered. g) The proposed solution MAY be designed in order to avoid transient forwarding loops. Indeed, forwarding loops increase packet transit-delay and may lead to link saturation. h) The specific procedure SHOULD end when the BGP session is closed following the g-shut and once the BGP session is gracefully opened following the g-noshut. In the end, once the planned maintenance is finished, the nominal BGP routing MUST be re- established. The duration of the g-shut procedure, and hence the time before the BGP session is safely closed, SHOULD be discussed by the solution document. Examples of possible solutions are the use of a pre-configured timer, the use of a message to signal the end of the BGP convergence, or the monitoring of the traffic on the g-shut interface. i) The solution SHOULD be simple and simple to operate. Hence, it MAY only cover a subset of the cases. As a consequence, most of the above requirements are expressed as "SHOULD" rather than "MUST". The metrics to evaluate and compare the proposed solutions are: - The duration of the remaining loss of connectivity when the BGP session is brought down or up; - The applicability to a wide range of BGP and network topologies; - The simplicity; - The duration of transient forwarding loops; - The additional load introduced in BGP (e.g., BGP messages sent to peer routers, peer ASes, the Internet).
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, January 2006. [RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter, "Multiprotocol Extensions for BGP-4", RFC 4760, January 2007. [RFC4456] Bates, T., Chen, E., and R. Chandra, "BGP Route Reflection: An Alternative to Full Mesh Internal BGP (IBGP)", RFC 4456, April 2006. [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, February 2006. [RFC5817] Ali, Z., Vasseur, JP., Zamfir, A., and J. Newton, "Graceful Shutdown in MPLS and Generalized MPLS Traffic Engineering Networks", RFC 5817, April 2010.
[RFC5715] Shand, M. and S. Bryant, "A Framework for Loop-Free Convergence", RFC 5715, January 2010. [RFC5714] Shand, M. and S. Bryant, "IP Fast Reroute Framework", RFC 5714, January 2010. [RFC4724] Sangli, S., Chen, E., Fernando, R., Scudder, J., and Y. Rekhter, "Graceful Restart Mechanism for BGP", RFC 4724, January 2007. [Reliable] Network Strategy Partners, LLC. "Reliable IP Nodes: A prerequisite to profitable IP services", November 2002. http://www.nspllc.com/NewPages/Reliable_IP_Nodes.pdf Acknowledgments The authors would like to thank Nicolas Dubois, Benoit Fondeviole, Christian Jacquenet, Olivier Bonaventure, Steve Uhlig, Xavier Vinet, Vincent Gillet, Jean-Louis le Roux, Pierre Alain Coste, and Ronald Bonica for their useful discussions on this subject, review, and comments. This document has been partly sponsored by the European project IST AGAVE.
' AS1 ' AS2 ' /----------- ASBR2.1 / ' / ' ASBR1.1 ' \ ' \ ' \----------- ASBR2.2 ' ' AS1 ' AS2 ' Figure 2. EBGP Topology with Redundant ASBR in One of the ASes BGP graceful shutdown is expected to be applicable for the maintenance of: - one of the routers of AS2; - one link between AS1 and AS2, performed either on an AS1 or AS2 router.
Note that in the case of maintenance of the whole router, all its BGP sessions need to be gracefully shutdown at the beginning of the maintenance and gracefully brought up at the end of the maintenance. ' AS1 ' AS2 ' ASBR1.1----------- ASBR2.1 ' ' ' ' ' ASBR1.2----------- ASBR2.2 ' AS1 ' AS2 ' Figure 3. EBGP Topology with Redundant ASBRs in Both ASes BGP graceful shutdown is expected to be applicable for the maintenance of: - any of the ASBR routers (in AS1 or AS2); - one link between AS1 and AS2, performed either on an AS1 or AS2 router.
' AS1 ' AS2 ' ASBR1.1----------- ASBR2.1 | ' | ' '''''|'''''''''' | ' | ' ASBR3.1----------- ASBR2.2 ' AS3 ' AS2 Figure 4. EBGP Topology of a Dual-Homed Customer As the requirement expressed in Section 5 is to advertise the maintenance only within the initiator and neighbor ASes, not Internet-wide, BGP graceful shutdown solutions may not be applicable to this topology. Depending on which routes are exchanged between these ASes, some protection for some of the traffic may be possible. For instance, if ASBR2.2 performs a maintenance affecting ASBR3.1, then ASBR3.1 will be notified. However, ASBR1.1 may not be notified of the maintenance of the EBGP session between ASBR3.1 and ASBR2.2.
P1 ----- P2 | \ / | | \ / | | \/ | AS1 | /\ | | / \ | | / \ | ASBR1.1--ASBR1.2 \ / \ / ''''''\'''/'''''''''''' \ / AS2 ASBR2.1 Figure 5. IBGP Full-Mesh When the session between ASBR1.1 and ASBR2.1 is gracefully shut down, it is required that all affected routers of AS1 reroute traffic to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 is shut down. Similarly, when the session between ASBR1.1 and ASBR2.1 is gracefully brought up, all affected routers of AS1 preferring ASBR1.1 over ASBR1.2 need to reroute traffic to ASBR1.1 before the less preferred path through ASBR1.2 is possibly withdrawn.
P1 (RR)-- P2 (RR) | \ / | | \ / | | \ / | AS1 | \/ | | /\ | | / \ | | / \ | | / \ | ASBR1.1 ASBR1.2 \ / \ / ''''''\''''''/'''''''''''' \ / \ / AS2 ASBR2.1 Figure 6. Route Reflector When the session between ASBR1.1 and ASBR2.1 is gracefully shut down, all BGP routers of AS1 need to reroute traffic to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 is shut down. Similarly, when the session between ASBR1.1 and ASBR2.1 is gracefully brought up, all affected routers of AS1 preferring ASBR1.1 over ASBR1.2 need to reroute traffic to ASBR1.1 before the less preferred path through ASBR1.2 is possibly withdrawn.
P1 (RR) -------- P2 (RR) | | | | | | AS1 | | | | P3 (RR) P4 (RR) | | | | | | AS1 | | | | ASBR1.1 ASBR1.2 \ / \ / ''''''\'''''''''/'''''''''''' \ / \ / AS2 ASBR2.1 Figure 7. Hierarchical Route Reflector When the session between ASBR1.1 and ASBR2.1 is gracefully shut down, all BGP routers of AS1 need to reroute traffic to ASBR1.2 before the session between ASBR1.1 and ASBR2.1 is shut down. Similarly, when the session between ASBR1.1 and ASBR2.1 is gracefully brought up, all affected routers of AS1 preferring ASBR1.1 over ASBR1.2 need to reroute traffic to ASBR1.1 before the less preferred path through ASBR1.2 is possibly withdrawn.
Confederations may be run with different sub-options. Regarding the IGP, each member AS can run its own IGP or they can all share the same IGP. Regarding BGP, LOCAL_PREF may or may not cross the member AS boundaries. A solution should support the graceful shutdown and graceful bringing up of EBGP sessions between member ASes in the confederation in addition to the graceful shutdown and graceful bringing up of EBGP sessions between a member-AS and an AS outside of the confederation. ASBR1C.1 ---------- ASBR1C.2 | | | | | AS1C | | | | | """|"""""""""""""""""""|""" | " | ASBR1A.2 " ASBR1B.2 | " | | " | | AS1A " AS1B | AS1 | " | | " | ASBR1A.1 " ASBR1B.1 \ " / \ " / ''''''\'''''''''''''/'''''''''''' \ / \ / AS2 ASBR2.1 Figure 8. Confederation In the above figure, member ASes AS1A, AS1B, and AS1C belong to a confederation of ASes in AS1. AS1A and AS1B are connected to AS2. In normal operation, for the traffic toward AS2: - AS1A sends the traffic directly to AS2 through ASBR1A.1. - AS1B sends the traffic directly to AS2 through ASBR1B.1. - AS1C load balances the traffic between AS1A and AS1B. When the session between ASBR1A.1 and ASBR2.1 is gracefully shut down, all BGP routers of AS1 need to reroute traffic to ASBR1B.1 before the session between ASBR1A.1 and ASBR2.1 is shut down.
Similarly, when the session between ASBR1A.1 and ASBR2.1 is gracefully brought up, all affected routers of AS1 preferring ASBR1A.1 over ASBR1B.1 need to reroute traffic to ASBR1A.1 before the less preferred path through ASBR1B.1 is possibly withdrawn. RFC4271], note that if tunnels are not used to forward packets between the ingress and egress ASBR; this can lead to persistent forwarding loops.