Tech-invite3GPPspecsGlossariesIETFRFCsGroupsSIPABNFs   Ti+SearchTech-invite World Map Symbol

RFC 7938


Use of BGP for Routing in Large-Scale Data Centers

Part 2 of 2, p. 20 to 35
Prev Section


prevText      Top      ToC       Page 20 
6.  ECMP Considerations

   This section covers the Equal Cost Multipath (ECMP) functionality for
   Clos topology and discusses a few special requirements.

6.1.  Basic ECMP

   ECMP is the fundamental load-sharing mechanism used by a Clos
   topology.  Effectively, every lower-tier device will use all of its
   directly attached upper-tier devices to load-share traffic destined
   to the same IP prefix.  The number of ECMP paths between any two Tier
   3 devices in Clos topology is equal to the number of the devices in
   the middle stage (Tier 1).  For example, Figure 5 illustrates a
   topology where Tier 3 device A has four paths to reach servers X and
   Y, via Tier 2 devices B and C and then Tier 1 devices 1, 2, 3, and 4,

                                Tier 1
                               | DEV |
                            +->|  1  |--+
                            |  +-----+  |
                    Tier 2  |           |   Tier 2
                   +-----+  |  +-----+  |  +-----+
     +------------>| DEV |--+->| DEV |--+--|     |-------------+
     |       +-----|  B  |--+  |  2  |  +--|     |-----+       |
     |       |     +-----+     +-----+     +-----+     |       |
     |       |                                         |       |
     |       |     +-----+     +-----+     +-----+     |       |
     | +-----+---->| DEV |--+  | DEV |  +--|     |-----+-----+ |
     | |     | +---|  C  |--+->|  3  |--+--|     |---+ |     | |
     | |     | |   +-----+  |  +-----+  |  +-----+   | |     | |
     | |     | |            |           |            | |     | |
   +-----+ +-----+          |  +-----+  |          +-----+ +-----+
   | DEV | |     | Tier 3   +->| DEV |--+   Tier 3 |     | |     |
   |  A  | |     |             |  4  |             |     | |     |
   +-----+ +-----+             +-----+             +-----+ +-----+
     | |     | |                                     | |     | |
     O O     O O            <- Servers ->            X Y     O O

               Figure 5: ECMP Fan-Out Tree from A to X and Y

   The ECMP requirement implies that the BGP implementation must support
   multipath fan-out for up to the maximum number of devices directly
   attached at any point in the topology in the upstream or downstream
   direction.  Normally, this number does not exceed half of the ports
   found on a device in the topology.  For example, an ECMP fan-out of
   32 would be required when building a Clos network using 64-port

Top      Up      ToC       Page 21 
   devices.  The Border Routers may need to have wider fan-out to be
   able to connect to a multitude of Tier 1 devices if route
   summarization at Border Router level is implemented as described in
   Section 5.2.5.  If a device's hardware does not support wider ECMP,
   logical link-grouping (link-aggregation at Layer 2) could be used to
   provide "hierarchical" ECMP (Layer 3 ECMP coupled with Layer 2 ECMP)
   to compensate for fan-out limitations.  However, this approach
   increases the risk of flow polarization, as less entropy will be
   available at the second stage of ECMP.

   Most BGP implementations declare paths to be equal from an ECMP
   perspective if they match up to and including step (e) in
   Section of [RFC4271].  In the proposed network design there
   is no underlying IGP, so all IGP costs are assumed to be zero or
   otherwise the same value across all paths and policies may be applied
   as necessary to equalize BGP attributes that vary in vendor defaults,
   such as the MULTI_EXIT_DISC (MED) attribute and origin code.  For
   historical reasons, it is also useful to not use 0 as the equalized
   MED value; this and some other useful BGP information is available in
   [RFC4277].  Routing loops are unlikely due to the BGP best-path
   selection process (which prefers shorter AS_PATH length), and longer
   paths through the Tier 1 devices (which don't allow their own ASN in
   the path) are not possible.

6.2.  BGP ECMP over Multiple ASNs

   For application load-balancing purposes, it is desirable to have the
   same prefix advertised from multiple Tier 3 devices.  From the
   perspective of other devices, such a prefix would have BGP paths with
   different AS_PATH attribute values, while having the same AS_PATH
   attribute lengths.  Therefore, BGP implementations must support load-
   sharing over the above-mentioned paths.  This feature is sometimes
   known as "multipath relax" or "multipath multiple-AS" and effectively
   allows for ECMP to be done across different neighboring ASNs if all
   other attributes are equal as already described in the previous

6.3.  Weighted ECMP

   It may be desirable for the network devices to implement "weighted"
   ECMP, to be able to send more traffic over some paths in ECMP fan-
   out.  This could be helpful to compensate for failures in the network
   and send more traffic over paths that have more capacity.  The
   prefixes that require weighted ECMP would have to be injected using
   remote BGP speaker (central agent) over a multi-hop session as
   described further in Section 8.1.  If support in implementations is
   available, weight distribution for multiple BGP paths could be
   signaled using the technique described in [LINK].

Top      Up      ToC       Page 22 
6.4.  Consistent Hashing

   It is often desirable to have the hashing function used for ECMP to
   be consistent (see [CONS-HASH]), to minimize the impact on flow to
   next-hop affinity changes when a next hop is added or removed to an
   ECMP group.  This could be used if the network device is used as a
   load balancer, mapping flows toward multiple destinations -- in this
   case, losing or adding a destination will not have a detrimental
   effect on currently established flows.  One particular recommendation
   on implementing consistent hashing is provided in [RFC2992], though
   other implementations are possible.  This functionality could be
   naturally combined with weighted ECMP, with the impact of the next
   hop changes being proportional to the weight of the given next hop.
   The downside of consistent hashing is increased load on hardware
   resource utilization, as typically more resources (e.g., Ternary
   Content-Addressable Memory (TCAM) space) are required to implement a
   consistent-hashing function.

7.  Routing Convergence Properties

   This section reviews routing convergence properties in the proposed
   design.  A case is made that sub-second convergence is achievable if
   the implementation supports fast EBGP peering session deactivation
   and timely RIB and FIB updates upon failure of the associated link.

7.1.  Fault Detection Timing

   BGP typically relies on an IGP to route around link/node failures
   inside an AS, and implements either a polling-based or an event-
   driven mechanism to obtain updates on IGP state changes.  The
   proposed routing design does not use an IGP, so the remaining
   mechanisms that could be used for fault detection are BGP keep-alive
   time-out (or any other type of keep-alive mechanism) and link-failure

   Relying solely on BGP keep-alive packets may result in high
   convergence delays, on the order of multiple seconds (on many BGP
   implementations the minimum configurable BGP hold timer value is
   three seconds).  However, many BGP implementations can shut down
   local EBGP peering sessions in response to the "link down" event for
   the outgoing interface used for BGP peering.  This feature is
   sometimes called "fast fallover".  Since links in modern data centers
   are predominantly point-to-point fiber connections, a physical
   interface failure is often detected in milliseconds and subsequently
   triggers a BGP reconvergence.

Top      Up      ToC       Page 23 
   Ethernet links may support failure signaling or detection standards
   such as Connectivity Fault Management (CFM) as described in
   [IEEE8021Q]; this may make failure detection more robust.
   Alternatively, some platforms may support Bidirectional Forwarding
   Detection (BFD) [RFC5880] to allow for sub-second failure detection
   and fault signaling to the BGP process.  However, the use of either
   of these presents additional requirements to vendor software and
   possibly hardware, and may contradict REQ1.  Until recently with
   [RFC7130], BFD also did not allow detection of a single member link
   failure on a LAG, which would have limited its usefulness in some

7.2.  Event Propagation Timing

   In the proposed design, the impact of the BGP
   MinRouteAdvertisementIntervalTimer (MRAI timer), as specified in
   Section of [RFC4271], should be considered.  Per the
   standard, it is required for BGP implementations to space out
   consecutive BGP UPDATE messages by at least MRAI seconds, which is
   often a configurable value.  The initial BGP UPDATE messages after an
   event carrying withdrawn routes are commonly not affected by this
   timer.  The MRAI timer may present significant convergence delays
   when a BGP speaker "waits" for the new path to be learned from its
   peers and has no local backup path information.

   In a Clos topology, each EBGP speaker typically has either one path
   (Tier 2 devices don't accept paths from other Tier 2 in the same
   cluster due to same ASN) or N paths for the same prefix, where N is a
   significantly large number, e.g., N=32 (the ECMP fan-out to the next
   tier).  Therefore, if a link fails to another device from which a
   path is received there is either no backup path at all (e.g., from
   the perspective of a Tier 2 switch losing the link to a Tier 3
   device), or the backup is readily available in BGP Loc-RIB (e.g.,
   from the perspective of a Tier 2 device losing the link to a Tier 1
   switch).  In the former case, the BGP withdrawal announcement will
   propagate without delay and trigger reconvergence on affected
   devices.  In the latter case, the best path will be re-evaluated, and
   the local ECMP group corresponding to the new next-hop set will be
   changed.  If the BGP path was the best path selected previously, an
   "implicit withdraw" will be sent via a BGP UPDATE message as
   described as Option b in Section 3.1 of [RFC4271] due to the BGP
   AS_PATH attribute changing.

Top      Up      ToC       Page 24 
7.3.  Impact of Clos Topology Fan-Outs

   Clos topology has large fan-outs, which may impact the "Up->Down"
   convergence in some cases, as described in this section.  In a
   situation when a link between Tier 3 and Tier 2 device fails, the
   Tier 2 device will send BGP UPDATE messages to all upstream Tier 1
   devices, withdrawing the affected prefixes.  The Tier 1 devices, in
   turn, will relay these messages to all downstream Tier 2 devices
   (except for the originator).  Tier 2 devices other than the one
   originating the UPDATE should then wait for ALL upstream Tier 1
   devices to send an UPDATE message before removing the affected
   prefixes and sending corresponding UPDATE downstream to connected
   Tier 3 devices.  If the original Tier 2 device or the relaying Tier 1
   devices introduce some delay into their UPDATE message announcements,
   the result could be UPDATE message "dispersion", that could be as
   long as multiple seconds.  In order to avoid such a behavior, BGP
   implementations must support "update groups".  The "update group" is
   defined as a collection of neighbors sharing the same outbound policy
   -- the local speaker will send BGP updates to the members of the
   group synchronously.

   The impact of such "dispersion" grows with the size of topology fan-
   out and could also grow under network convergence churn.  Some
   operators may be tempted to introduce "route flap dampening" type
   features that vendors include to reduce the control-plane impact of
   rapidly flapping prefixes.  However, due to issues described with
   false positives in these implementations especially under such
   "dispersion" events, it is not recommended to enable this feature in
   this design.  More background and issues with "route flap dampening"
   and possible implementation changes that could affect this are well
   described in [RFC7196].

7.4.  Failure Impact Scope

   A network is declared to converge in response to a failure once all
   devices within the failure impact scope are notified of the event and
   have recalculated their RIBs and consequently updated their FIBs.
   Larger failure impact scope typically means slower convergence since
   more devices have to be notified, and results in a less stable
   network.  In this section, we describe BGP's advantages over link-
   state routing protocols in reducing failure impact scope for a Clos

   BGP behaves like a distance-vector protocol in the sense that only
   the best path from the point of view of the local router is sent to
   neighbors.  As such, some failures are masked if the local node can
   immediately find a backup path and does not have to send any updates
   further.  Notice that in the worst case, all devices in a data center

Top      Up      ToC       Page 25 
   topology have to either withdraw a prefix completely or update the
   ECMP groups in their FIBs.  However, many failures will not result in
   such a wide impact.  There are two main failure types where impact
   scope is reduced:

   o  Failure of a link between Tier 2 and Tier 1 devices: In this case,
      a Tier 2 device will update the affected ECMP groups, removing the
      failed link.  There is no need to send new information to
      downstream Tier 3 devices, unless the path was selected as best by
      the BGP process, in which case only an "implicit withdraw" needs
      to be sent and this should not affect forwarding.  The affected
      Tier 1 device will lose the only path available to reach a
      particular cluster and will have to withdraw the associated
      prefixes.  Such a prefix withdrawal process will only affect Tier
      2 devices directly connected to the affected Tier 1 device.  The
      Tier 2 devices receiving the BGP UPDATE messages withdrawing
      prefixes will simply have to update their ECMP groups.  The Tier 3
      devices are not involved in the reconvergence process.

   o  Failure of a Tier 1 device: In this case, all Tier 2 devices
      directly attached to the failed node will have to update their
      ECMP groups for all IP prefixes from a non-local cluster.  The
      Tier 3 devices are once again not involved in the reconvergence
      process, but may receive "implicit withdraws" as described above.

   Even in the case of such failures where multiple IP prefixes will
   have to be reprogrammed in the FIB, it is worth noting that all of
   these prefixes share a single ECMP group on a Tier 2 device.
   Therefore, in the case of implementations with a hierarchical FIB,
   only a single change has to be made to the FIB.  "Hierarchical FIB"
   here means FIB structure where the next-hop forwarding information is
   stored separately from the prefix lookup table, and the latter only
   stores pointers to the respective forwarding information.  See
   [BGP-PIC] for discussion of FIB hierarchies and fast convergence.

   Even though BGP offers reduced failure scope for some cases, further
   reduction of the fault domain using summarization is not always
   possible with the proposed design, since using this technique may
   create routing black-holes as mentioned previously.  Therefore, the
   worst failure impact scope on the control plane is the network as a
   whole -- for instance, in the case of a link failure between Tier 2
   and Tier 3 devices.  The amount of impacted prefixes in this case
   would be much less than in the case of a failure in the upper layers
   of a Clos network topology.  The property of having such large
   failure scope is not a result of choosing EBGP in the design but
   rather a result of using the Clos topology.

Top      Up      ToC       Page 26 
7.5.  Routing Micro-Loops

   When a downstream device, e.g., Tier 2 device, loses all paths for a
   prefix, it normally has the default route pointing toward the
   upstream device -- in this case, the Tier 1 device.  As a result, it
   is possible to get in the situation where a Tier 2 switch loses a
   prefix, but a Tier 1 switch still has the path pointing to the Tier 2
   device; this results in a transient micro-loop, since the Tier 1
   switch will keep passing packets to the affected prefix back to the
   Tier 2 device, and the Tier 2 will bounce them back again using the
   default route.  This micro-loop will last for the time it takes the
   upstream device to fully update its forwarding tables.

   To minimize impact of such micro-loops, Tier 2 and Tier 1 switches
   can be configured with static "discard" or "null" routes that will be
   more specific than the default route for prefixes missing during
   network convergence.  For Tier 2 switches, the discard route should
   be a summary route, covering all server subnets of the underlying
   Tier 3 devices.  For Tier 1 devices, the discard route should be a
   summary covering the server IP address subnets allocated for the
   whole data center.  Those discard routes will only take precedence
   for the duration of network convergence, until the device learns a
   more specific prefix via a new path.

8.  Additional Options for Design

8.1.  Third-Party Route Injection

   BGP allows for a "third-party", i.e., a directly attached BGP
   speaker, to inject routes anywhere in the network topology, meeting
   REQ5.  This can be achieved by peering via a multi-hop BGP session
   with some or even all devices in the topology.  Furthermore, BGP
   diverse path distribution [RFC6774] could be used to inject multiple
   BGP next hops for the same prefix to facilitate load balancing, or
   using the BGP ADD-PATH capability [RFC7911] if supported by the
   implementation.  Unfortunately, in many implementations, ADD-PATH has
   been found to only support IBGP properly in the use cases for which
   it was originally optimized; this limits the "third-party" peering to
   IBGP only.

   To implement route injection in the proposed design, a third-party
   BGP speaker may peer with Tier 3 and Tier 1 switches, injecting the
   same prefix, but using a special set of BGP next hops for Tier 1
   devices.  Those next hops are assumed to resolve recursively via BGP,
   and could be, for example, IP addresses on Tier 3 devices.  The
   resulting forwarding table programming could provide desired traffic
   proportion distribution among different clusters.

Top      Up      ToC       Page 27 
8.2.  Route Summarization within Clos Topology

   As mentioned previously, route summarization is not possible within
   the proposed Clos topology since it makes the network susceptible to
   route black-holing under single link failures.  The main problem is
   the limited number of redundant paths between network elements, e.g.,
   there is only a single path between any pair of Tier 1 and Tier 3
   devices.  However, some operators may find route aggregation
   desirable to improve control-plane stability.

   If any technique to summarize within the topology is planned,
   modeling of the routing behavior and potential for black-holing
   should be done not only for single or multiple link failures, but
   also for fiber pathway failures or optical domain failures when the
   topology extends beyond a physical location.  Simple modeling can be
   done by checking the reachability on devices doing summarization
   under the condition of a link or pathway failure between a set of
   devices in every tier as well as to the WAN routers when external
   connectivity is present.

   Route summarization would be possible with a small modification to
   the network topology, though the tradeoff would be reduction of the
   total size of the network as well as network congestion under
   specific failures.  This approach is very similar to the technique
   described above, which allows Border Routers to summarize the entire
   data center address space.

8.2.1.  Collapsing Tier 1 Devices Layer

   In order to add more paths between Tier 1 and Tier 3 devices, group
   Tier 2 devices into pairs, and then connect the pairs to the same
   group of Tier 1 devices.  This is logically equivalent to
   "collapsing" Tier 1 devices into a group of half the size, merging
   the links on the "collapsed" devices.  The result is illustrated in
   Figure 6.  For example, in this topology DEV C and DEV D connect to
   the same set of Tier 1 devices (DEV 1 and DEV 2), whereas before they
   were connecting to different groups of Tier 1 devices.

Top      Up      ToC       Page 28 
                    Tier 2       Tier 1       Tier 2
                   +-----+      +-----+      +-----+
     +-------------| DEV |------| DEV |------|     |-------------+
     |       +-----|  C  |--++--|  1  |--++--|     |-----+       |
     |       |     +-----+  ||  +-----+  ||  +-----+     |       |
     |       |              ||           ||              |       |
     |       |     +-----+  ||  +-----+  ||  +-----+     |       |
     | +-----+-----| DEV |--++--| DEV |--++--|     |-----+-----+ |
     | |     | +---|  D  |------|  2  |------|     |---+ |     | |
     | |     | |   +-----+      +-----+      +-----+   | |     | |
     | |     | |                                       | |     | |
   +-----+ +-----+                                   +-----+ +-----+
   | DEV | | DEV |                                   |     | |     |
   |  A  | |  B  | Tier 3                     Tier 3 |     | |     |
   +-----+ +-----+                                   +-----+ +-----+
     | |     | |                                       | |     | |
     O O     O O             <- Servers ->             O O     O O

                      Figure 6: 5-Stage Clos Topology

   Having this design in place, Tier 2 devices may be configured to
   advertise only a default route down to Tier 3 devices.  If a link
   between Tier 2 and Tier 3 fails, the traffic will be re-routed via
   the second available path known to a Tier 2 switch.  It is still not
   possible to advertise a summary route covering prefixes for a single
   cluster from Tier 2 devices since each of them has only a single path
   down to this prefix.  It would require dual-homed servers to
   accomplish that.  Also note that this design is only resilient to
   single link failures.  It is possible for a double link failure to
   isolate a Tier 2 device from all paths toward a specific Tier 3
   device, thus causing a routing black-hole.

   A result of the proposed topology modification would be a reduction
   of the port capacity of Tier 1 devices.  This limits the maximum
   number of attached Tier 2 devices, and therefore will limit the
   maximum DC network size.  A larger network would require different
   Tier 1 devices that have higher port density to implement this

   Another problem is traffic rebalancing under link failures.  Since
   there are two paths from Tier 1 to Tier 3, a failure of the link
   between Tier 1 and Tier 2 switch would result in all traffic that was
   taking the failed link to switch to the remaining path.  This will
   result in doubling the link utilization on the remaining link.

Top      Up      ToC       Page 29 
8.2.2.  Simple Virtual Aggregation

   A completely different approach to route summarization is possible,
   provided that the main goal is to reduce the FIB size, while allowing
   the control plane to disseminate full routing information.  Firstly,
   it could be easily noted that in many cases multiple prefixes, some
   of which are less specific, share the same set of the next hops (same
   ECMP group).  For example, from the perspective of Tier 3 devices,
   all routes learned from upstream Tier 2 devices, including the
   default route, will share the same set of BGP next hops, provided
   that there are no failures in the network.  This makes it possible to
   use the technique similar to that described in [RFC6769] and only
   install the least specific route in the FIB, ignoring more specific
   routes if they share the same next-hop set.  For example, under
   normal network conditions, only the default route needs to be
   programmed into the FIB.

   Furthermore, if the Tier 2 devices are configured with summary
   prefixes covering all of their attached Tier 3 device's prefixes, the
   same logic could be applied in Tier 1 devices as well and, by
   induction to Tier 2/Tier 3 switches in different clusters.  These
   summary routes should still allow for more specific prefixes to leak
   to Tier 1 devices, to enable detection of mismatches in the next-hop
   sets if a particular link fails, thus changing the next-hop set for a
   specific prefix.

   Restating once again, this technique does not reduce the amount of
   control-plane state (i.e., BGP UPDATEs, BGP Loc-RIB size), but only
   allows for more efficient FIB utilization, by detecting more specific
   prefixes that share their next-hop set with a subsuming less specific

8.3.  ICMP Unreachable Message Masquerading

   This section discusses some operational aspects of not advertising
   point-to-point link subnets into BGP, as previously identified as an
   option in Section 5.2.3.  The operational impact of this decision
   could be seen when using the well-known "traceroute" tool.
   Specifically, IP addresses displayed by the tool will be the link's
   point-to-point addresses, and hence will be unreachable for
   management connectivity.  This makes some troubleshooting more

   One way to overcome this limitation is by using the DNS subsystem to
   create the "reverse" entries for these point-to-point IP addresses
   pointing to the same name as the loopback address.  The connectivity
   then can be made by resolving this name to the "primary" IP address

Top      Up      ToC       Page 30 
   of the device, e.g., its Loopback interface, which is always
   advertised into BGP.  However, this creates a dependency on the DNS
   subsystem, which may be unavailable during an outage.

   Another option is to make the network device perform IP address
   masquerading, that is, rewriting the source IP addresses of the
   appropriate ICMP messages sent by the device with the "primary" IP
   address of the device.  Specifically, the ICMP Destination
   Unreachable Message (type 3) code 3 (port unreachable) and ICMP Time
   Exceeded (type 11) code 0 are required for correct operation of the
   "traceroute" tool.  With this modification, the "traceroute" probes
   sent to the devices will always be sent back with the "primary" IP
   address as the source, allowing the operator to discover the
   "reachable" IP address of the box.  This has the downside of hiding
   the address of the "entry point" into the device.  If the devices
   support [RFC5837], this may allow the best of both worlds by
   providing the information about the incoming interface even if the
   return address is the "primary" IP address.

9.  Security Considerations

   The design does not introduce any additional security concerns.
   General BGP security considerations are discussed in [RFC4271] and
   [RFC4272].  Since a DC is a single-operator domain, this document
   assumes that edge filtering is in place to prevent attacks against
   the BGP sessions themselves from outside the perimeter of the DC.
   This may be a more feasible option for most deployments than having
   to deal with key management for TCP MD5 as described in [RFC2385] or
   dealing with the lack of implementations of the TCP Authentication
   Option [RFC5925] available at the time of publication of this
   document.  The Generalized TTL Security Mechanism [RFC5082] could
   also be used to further reduce the risk of BGP session spoofing.

10.  References

10.1.  Normative References

   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
              DOI 10.17487/RFC4271, January 2006,

   [RFC6996]  Mitchell, J., "Autonomous System (AS) Reservation for
              Private Use", BCP 6, RFC 6996, DOI 10.17487/RFC6996, July
              2013, <>.

Top      Up      ToC       Page 31 
10.2.  Informative References

              Al-Fares, M., Loukissas, A., and A. Vahdat, "A Scalable,
              Commodity Data Center Network Architecture",
              DOI 10.1145/1402958.1402967, August 2008,

              Cisco Systems, "Allowas-in Feature in BGP Configuration
              Example", February 2015,

   [BGP-PIC]  Bashandy, A., Ed., Filsfils, C., and P. Mohapatra, "BGP
              Prefix Independent Convergence", Work in Progress,
              draft-ietf-rtgwg-bgp-pic-02, August 2016.

   [CLOS1953] Clos, C., "A Study of Non-Blocking Switching Networks",
              The Bell System Technical Journal, Vol. 32(2),
              DOI 10.1002/j.1538-7305.1953.tb01433.x, March 1953.

              Cisco Systems, "Configuring and Verifying the BGP
              Conditional Advertisement Feature", August 2005,

              Wikipedia, "Consistent Hashing", July 2016,

   [FB4POST]  Farrington, N. and A. Andreyev, "Facebook's Data Center
              Network Architecture", May 2013,

              Greenberg, A., Hamilton, J., and D. Maltz, "The Cost of a
              Cloud: Research Problems in Data Center Networks",
              DOI 10.1145/1496091.1496103, January 2009,

   [HADOOP]   Apache, "Apache Hadoop", April 2016,

Top      Up      ToC       Page 32 
   [IANA.AS]  IANA, "Autonomous System (AS) Numbers",

              IEEE, "IEEE Standard for Local and Metropolitan Area
              Networks: Media Access Control (MAC) Bridges", IEEE
              Std 802.1D, DOI 10.1109/IEEESTD.1991.101050, 1991,

              IEEE, "IEEE Standard for Local and Metropolitan Area
              Networks: Media Access Control (MAC) Bridges", IEEE
              Std 802.1D, DOI 10.1109/IEEESTD.2004.94569, June 2004,

              IEEE, "IEEE Standard for Local and Metropolitan Area
              Networks: Bridges and Bridged Networks", IEEE Std 802.1Q,
              DOI 10.1109/IEEESTD.2014.6991462,

              IEEE, "Amendment to Carrier Sense Multiple Access With
              Collision Detection (CSMA/CD) Access Method and Physical
              Layer Specifications - Aggregation of Multiple Link
              Segments", IEEE Std 802.3ad,
              DOI 10.1109/IEEESTD.2000.91610, October 2000,

   [INTERCON] Dally, W. and B. Towles, "Principles and Practices of
              Interconnection Networks", ISBN 978-0122007514, January
              2004, <>.

              Jakma, P., "BGP Path Hunting", 2008,

   [L3DSR]    Schaumann, J., "L3DSR - Overcoming Layer 2 Limitations of
              Direct Server Return Load Balancing", 2011,

   [LINK]     Mohapatra, P. and R. Fernando, "BGP Link Bandwidth
              Extended Community", Work in Progress, draft-ietf-idr-
              link-bandwidth-06, January 2013.

Top      Up      ToC       Page 33 
   [REMOVAL]  Mitchell, J., Rao, D., and R. Raszuk, "Private Autonomous
              System (AS) Removal Requirements", Work in Progress,
              draft-mitchell-grow-remove-private-as-04, April 2015.

   [RFC2328]  Moy, J., "OSPF Version 2", STD 54, RFC 2328,
              DOI 10.17487/RFC2328, April 1998,

   [RFC2385]  Heffernan, A., "Protection of BGP Sessions via the TCP MD5
              Signature Option", RFC 2385, DOI 10.17487/RFC2385, August
              1998, <>.

   [RFC2992]  Hopps, C., "Analysis of an Equal-Cost Multi-Path
              Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000,

   [RFC4272]  Murphy, S., "BGP Security Vulnerabilities Analysis",
              RFC 4272, DOI 10.17487/RFC4272, January 2006,

   [RFC4277]  McPherson, D. and K. Patel, "Experience with the BGP-4
              Protocol", RFC 4277, DOI 10.17487/RFC4277, January 2006,

   [RFC4786]  Abley, J. and K. Lindqvist, "Operation of Anycast
              Services", BCP 126, RFC 4786, DOI 10.17487/RFC4786,
              December 2006, <>.

   [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
              Pignataro, "The Generalized TTL Security Mechanism
              (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,

   [RFC5837]  Atlas, A., Ed., Bonica, R., Ed., Pignataro, C., Ed., Shen,
              N., and JR. Rivers, "Extending ICMP for Interface and
              Next-Hop Identification", RFC 5837, DOI 10.17487/RFC5837,
              April 2010, <>.

   [RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
              (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,

   [RFC5925]  Touch, J., Mankin, A., and R. Bonica, "The TCP
              Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
              June 2010, <>.

Top      Up      ToC       Page 34 
   [RFC6325]  Perlman, R., Eastlake 3rd, D., Dutt, D., Gai, S., and A.
              Ghanwani, "Routing Bridges (RBridges): Base Protocol
              Specification", RFC 6325, DOI 10.17487/RFC6325, July 2011,

   [RFC6769]  Raszuk, R., Heitz, J., Lo, A., Zhang, L., and X. Xu,
              "Simple Virtual Aggregation (S-VA)", RFC 6769,
              DOI 10.17487/RFC6769, October 2012,

   [RFC6774]  Raszuk, R., Ed., Fernando, R., Patel, K., McPherson, D.,
              and K. Kumaki, "Distribution of Diverse BGP Paths",
              RFC 6774, DOI 10.17487/RFC6774, November 2012,

   [RFC6793]  Vohra, Q. and E. Chen, "BGP Support for Four-Octet
              Autonomous System (AS) Number Space", RFC 6793,
              DOI 10.17487/RFC6793, December 2012,

   [RFC7067]  Dunbar, L., Eastlake 3rd, D., Perlman, R., and I.
              Gashinsky, "Directory Assistance Problem and High-Level
              Design Proposal", RFC 7067, DOI 10.17487/RFC7067, November
              2013, <>.

   [RFC7130]  Bhatia, M., Ed., Chen, M., Ed., Boutros, S., Ed.,
              Binderberger, M., Ed., and J. Haas, Ed., "Bidirectional
              Forwarding Detection (BFD) on Link Aggregation Group (LAG)
              Interfaces", RFC 7130, DOI 10.17487/RFC7130, February
              2014, <>.

   [RFC7196]  Pelsser, C., Bush, R., Patel, K., Mohapatra, P., and O.
              Maennel, "Making Route Flap Damping Usable", RFC 7196,
              DOI 10.17487/RFC7196, May 2014,

   [RFC7911]  Walton, D., Retana, A., Chen, E., and J. Scudder,
              "Advertisement of Multiple Paths in BGP", RFC 7911,
              DOI 10.17487/RFC7911, July 2016,

              Cisco Systems, "Removing Private Autonomous System Numbers
              in BGP", August 2005,

Top      Up      ToC       Page 35 

   This publication summarizes the work of many people who participated
   in developing, testing, and deploying the proposed network design,
   some of whom were George Chen, Parantap Lahiri, Dave Maltz, Edet
   Nkposong, Robert Toomey, and Lihua Yuan.  The authors would also like
   to thank Linda Dunbar, Anoop Ghanwani, Susan Hares, Danny McPherson,
   Robert Raszuk, and Russ White for reviewing this document and
   providing valuable feedback, and Mary Mitchell for initial grammar
   and style suggestions.

Authors' Addresses

   Petr Lapukhov
   1 Hacker Way
   Menlo Park, CA  94025
   United States of America


   Ariff Premji
   Arista Networks
   5453 Great America Parkway
   Santa Clara, CA  95054
   United States of America


   Jon Mitchell (editor)