Network Working Group R. Bush Request for Comments: 3439 D. Meyer Updates: 1958 December 2002 Category: Informational Some Internet Architectural Guidelines and Philosophy Status of this Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (2002). All Rights Reserved.
AbstractThis document extends RFC 1958 by outlining some of the philosophical guidelines to which architects and designers of Internet backbone networks should adhere. We describe the Simplicity Principle, which states that complexity is the primary mechanism that impedes efficient scaling, and discuss its implications on the architecture, design and engineering issues found in large scale Internet backbones. 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Large Systems and The Simplicity Principle . . . . . . . . . 3 2.1. The End-to-End Argument and Simplicity . . . . . . . . . 3 2.2. Non-linearity and Network Complexity . . . . . . . . . . 3 2.2.1. The Amplification Principle. . . . . . . . . . . . . . . 4 2.2.2. The Coupling Principle . . . . . . . . . . . . . . . . . 5 2.3. Complexity lesson from voice. . . . . . . . . . . . . . . 6 2.4. Upgrade cost of complexity. . . . . . . . . . . . . . . . 7 3. Layering Considered Harmful. . . . . . . . . . . . . . . . . 7 3.1. Optimization Considered Harmful . . . . . . . . . . . . . 8 3.2. Feature Richness Considered Harmful . . . . . . . . . . . 9 3.3. Evolution of Transport Efficiency for IP. . . . . . . . . 9 3.4. Convergence Layering. . . . . . . . . . . . . . . . . . . 9 3.4.1. Note on Transport Protocol Layering. . . . . . . . . . . 11 3.5. Second Order Effects . . . . . . . . . . . . . . . . . . 11 3.6. Instantiating the EOSL Model with IP . . . . . . . . . . 12 4. Avoid the Universal Interworking Function. . . . . . . . . . 12 4.1. Avoid Control Plane Interworking . . . . . . . . . . . . . 13
5. Packet versus Circuit Switching: Fundamental Differences . . 13 5.1. Is PS is inherently more efficient than CS? . . . . . . . 13 5.2. Is PS simpler than CS? . . . . . . . . . . . . . . . . . . 14 5.2.1. Software/Firmware Complexity . . . . . . . . . . . . . . 15 5.2.2. Macro Operation Complexity . . . . . . . . . . . . . . . 15 5.2.3. Hardware Complexity. . . . . . . . . . . . . . . . . . . 15 5.2.4. Power. . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.2.5. Density. . . . . . . . . . . . . . . . . . . . . . . . . 16 5.2.6. Fixed versus variable costs. . . . . . . . . . . . . . . 16 5.2.7. QoS. . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.2.8. Flexibility. . . . . . . . . . . . . . . . . . . . . . . 17 5.3. Relative Complexity . . . . . . . . . . . . . . . . . . . 17 5.3.1. HBHI and the OPEX Challenge. . . . . . . . . . . . . . . 18 6. The Myth of Over-Provisioning. . . . . . . . . . . . . . . . 18 7. The Myth of Five Nines . . . . . . . . . . . . . . . . . . . 19 8. Architectural Component Proportionality Law. . . . . . . . . 20 8.1. Service Delivery Paths . . . . . . . . . . . . . . . . . . 21 9. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . 21 10. Security Considerations . . . . . . . . . . . . . . . . . . 22 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 23 12. References. . . . . . . . . . . . . . . . . . . . . . . . . 23 13. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . 27 14. Full Copyright Statement. . . . . . . . . . . . . . . . . . 28 RFC1958] describes the underlying principles of the Internet architecture. This note extends that work by outlining some of the philosophical guidelines to which architects and designers of Internet backbone networks should adhere. While many of the areas outlined in this document may be controversial, the unifying principle described here, controlling complexity as a mechanism to control costs and reliability, should not be. Complexity in carrier networks can derive from many sources. However, as stated in [DOYLE2002], "Complexity in most systems is driven by the need for robustness to uncertainty in their environments and component parts far more than by basic functionality". The major thrust of this document, then, is to raise awareness about the complexity of some of our current architectures, and to examine the effect such complexity will almost certainly have on the IP carrier industry's ability to succeed. The rest of this document is organized as follows: The first section describes the Simplicity Principle and its implications for the design of very large systems. The remainder of the document outlines the high-level consequences of the Simplicity Principle and how it should guide large scale network architecture and design approaches.
SALTZER] (as well as in RFC 1958 [RFC1958]), contends that "end-to-end protocol design should not rely on the maintenance of state (i.e., information about the state of the end-to-end communication) inside the network. Such state should be maintained only in the end points, in such a way that the state can only be destroyed when the end point itself breaks." This property has also been related to Clark's "fate-sharing" concept [CLARK]. We can see that the end-to-end principle leads directly to the Simplicity Principle by examining the so-called "hourglass" formulation of the Internet architecture [WILLINGER2002]. In this model, the thin waist of the hourglass is envisioned as the (minimalist) IP layer, and any additional complexity is added above the IP layer. In short, the complexity of the Internet belongs at the edges, and the IP layer of the Internet should remain as simple as possible. Finally, note that the End-to-End Argument does not imply that the core of the Internet will not contain and maintain state. In fact, a huge amount coarse grained state is maintained in the Internet's core (e.g., routing state). However, the important point here is that this (coarse grained) state is almost orthogonal to the state maintained by the end-points (e.g., hosts). It is this minimization of interaction that contributes to simplicity. As a result, consideration of "core vs. end-point" state interaction is crucial when analyzing protocols such as Network Address Translation (NAT), which reduce the transparency between network and hosts.
network, and as such doesn't have the same properties. In particular, the largest networks exhibit, both in theory and in practice, architecture, design, and engineering non-linearities which are not exhibited at smaller scale. We call this Architecture, Design, and Engineering (ADE) non-linearity. That is, systems such as the Internet could be described as highly self-dissimilar, with extremely different scales and levels of abstraction [CARLSON]. The ADE non-linearity property is based upon two well-known principles from non-linear systems theory [THOMPSON]: AHUJA]. A related result is that a small amount of inter-connectivity causes the output of a routing mesh to be significantly more complex than its input [GRIFFIN]. An important method for reducing amplification is ensure that local changes have only local effect (this is as opposed to systems in which local changes have global effect). Finally, ATM provides an excellent example of an amplification effect: if you lose one cell, you destroy
the entire packet (and it gets worse, as in the absence of mechanisms such as Early Packet Discard [ROMANOV], you will continue to carry the already damaged packet). Another interesting example of amplification comes from the engineering domain, and is described in [CARLSON]. They consider the Boeing 777, which is a "fly-by-wire" aircraft, containing as many as 150,000 subsystems and approximately 1000 CPUs. What they observe is that while the 777 is robust to large-scale atmospheric disturbances, turbulence boundaries, and variations in cargo loads (to name a few), it could be catastrophically disabled my microscopic alterations in a very few large CPUs (as the point out, fortunately this is a very rare occurrence). This example illustrates the issue "that complexity can amplify small perturbations, and the design engineer must ensure such perturbations are extremely rare." [CARLSON] WILLINGER2002]. Much of the non-linearity observed large systems is largely due to coupling. This coupling has both horizontal and vertical components. In the context of networking, horizontal coupling is exhibited between the same protocol layer, while vertical coupling occurs between layers. Coupling is exhibited by a wide variety of natural systems, including plasma macro-instabilities (hydro-magnetic, e.g., kink, fire-hose, mirror, ballooning, tearing, trapped-particle effects) [NAVE], as well as various kinds of electrochemical systems (consider the custom fluorescent nucleotide synthesis/nucleic acid labeling problem [WARD]). Coupling of clock physical periodicity has also been observed [JACOBSON], as well as coupling of various types of biological cycles. Several canonical examples also exist in well known network systems. Examples include the synchronization of various control loops, such as routing update synchronization and TCP Slow Start synchronization [FLOYD,JACOBSON]. An important result of these observations is that coupling is intimately related to synchronization. Injecting randomness into these systems is one way to reduce coupling.
Interestingly, in analyzing risk factors for the Public Switched Telephone Network (PSTN), Charles Perrow decomposes the complexity problem along two related axes, which he terms "interactions" and "coupling" [PERROW]. Perrow cites interactions and coupling as significant factors in determining the reliability of a complex system (and in particular, the PSTN). In this model, interactions refer to the dependencies between components (linear or non-linear), while coupling refers to the flexibility in a system. Systems with simple, linear interactions have components that affect only other components that are functionally downstream. Complex system components interact with many other components in different and possibly distant parts of the system. Loosely coupled systems are said to have more flexibility in time constraints, sequencing, and environmental assumptions than do tightly coupled systems. In addition, systems with complex interactions and tight coupling are likely to have unforeseen failure states (of course, complex interactions permit more complications to develop and make the system hard to understand and predict); this behavior is also described in [WILLINGER2002]. Tight coupling also means that the system has less flexibility in recovering from failure states. The PSTN's SS7 control network provides an interesting example of what can go wrong with a tightly coupled complex system. Outages such as the well publicized 1991 outage of AT&T's SS7 demonstrates the phenomenon: the outage was caused by software bugs in the switches' crash recovery code. In this case, one switch crashed due to a hardware glitch. When this switch came back up, it (plus a reasonably probable timing event) caused its neighbors to crash When the neighboring switches came back up, they caused their neighbors to crash, and so on [NEUMANN] (the root cause turned out to be a misplaced 'break' statement; this is an excellent example of cross- layer coupling). This phenomenon is similar to the phase-locking of weakly coupled oscillators, in which random variations in sequence times plays an important role in system stability [THOMPSON].
RFC1629]) Layering of this type does have various conceptual and structuring advantages. However, in the data networking context structured layering implies that the functions of each layer are carried out completely before the protocol data unit is passed to the next layer. This means that the optimization of each layer has to be done separately. Such ordering constraints are in conflict with efficient implementation of data manipulation functions. One could accuse the layered model (e.g., TCP/IP and ISO OSI) of causing this conflict. In fact, the operations of multiplexing and segmentation both hide vital information that lower layers may need to optimize their
performance. For example, layer N may duplicate lower level functionality, e.g., error recovery hop-hop versus end-to-end error recovery. In addition, different layers may need the same information (e.g., time stamp): layer N may need layer N-2 information (e.g., lower layer packet sizes), and the like [WAKEMAN]. A related and even more ironic statement comes from Tennenhouse's classic paper, "Layered Multiplexing Considered Harmful" [TENNENHOUSE]: "The ATM approach to broadband networking is presently being pursued within the CCITT (and elsewhere) as the unifying mechanism for the support of service integration, rate adaptation, and jitter control within the lower layers of the network architecture. This position paper is specifically concerned with the jitter arising from the design of the "middle" and "upper" layers that operate within the end systems and relays of multi-service networks (MSNs)." As a result of inter-layer dependencies, increased layering can quickly lead to violation of the Simplicity Principle. Industry experience has taught us that increased layering frequently increases complexity and hence leads to increases in OPEX, as is predicted by the Simplicity Principle. A corollary is stated in RFC 1925 [RFC1925], section 2(5): "It is always possible to agglutinate multiple separate problems into a single complex interdependent solution. In most cases this is a bad idea." The first order conclusion then, is that horizontal (as opposed to vertical) separation may be more cost-effective and reliable in the long term. SPILLMAN]. The implication here is that trying to squeeze out efficiency past that point only adds complexity, and hence leads to less reliable systems.
CAIDA] shows that about 95% of WAN bytes and 85% of packets are TCP. Much of this traffic is composed of 40/44 byte packets. Now, consider the case of a a DS3 backbone with PLCP turned on. Then the maximum cell rate is 96,000 cells/sec. If you multiply this value by the number of bits in the payload, you get: 96000 cells/sec * 48 bytes/cell * 8 = 36.864 Mbps. This, however, is unrealistic since it
assumes perfect payload packing. There are two other things that contribute to the ATM overhead (cell tax): The wasted padding and the 8 byte SNAP header. It is the SNAP header which causes most of the problems (and you can't do anything about this), forcing most small packets to consume two cells, with the second cell to be mostly empty padding (this interacts really poorly with the data quoted above, e.g., that most packets are 40-44 byte TCP Ack packets). This causes a loss of about another 16% from the 36.8 Mbps ideal throughput. So the total throughput ends up being (for a DS3): DS3 Line Rate: 44.736 PLCP Overhead - 4.032 Per Cell Header: - 3.840 SNAP Header & Padding: - 5.900 ========= 30.960 Mbps Result: With a DS3 line rate of 44.736 Mbps, the total overhead is about 31%. Another way to look at this is that since a large fraction of WAN traffic is comprised of TCP ACKs, one can make a different but related calculation. IP over ATM requires: IP data (40 bytes in this case) 8 bytes SNAP 8 bytes AAL5 stuff 5 bytes for each cell + as much more as it takes to fill out the last cell On ATM, this becomes two cells - 106 bytes to convey 40 bytes of information. The next most common size seems to be one of several sizes in the 504-556 byte range - 636 bytes to carry IP, TCP, and a 512 byte TCP payload - with messages larger than 1000 bytes running third. One would imagine that 87% payload (556 byte message size) is better than 37% payload (TCP Ack size), but it's not the 95-98% that customers are used to, and the predominance of TCP Acks skews the average.
TENNENHOUSE,WAKEMAN], there have been a few notable instances where efficiency can be and is gained. In particular, "X over Y efficiencies" can be realized when there is a kind of "isomorphism" between the X and Y (i.e., there is a small convergence layer). In these cases X's data, and possibly control traffic, are "encapsulated" and transported over Y. Examples include Frame Relay over ATM, and Frame Relay, AAL5 ATM and Ethernet over L2TPv3 [L2TPV3]; the simplifying factors here are that there is no requirement that a shared clock be recovered by the communicating end points, and that control-plane interworking is minimized. An alternative is to interwork the X and Y's control and data planes; control-plane interworking is discussed below. ROMANOV] on ATM showed that large UBR buffers (larger than one TCP window size) are required to achieve reasonable performance, that packet discard mechanisms (such as Early Packet Discard, or EPD) improve the effective usage of the bandwidth and that more elaborate service and drop strategies than FIFO+EPD, such as per VC queuing and accounting, might be required at the bottleneck to ensure both high efficiency and fairness. Though all studies clearly indicate that a buffer size not less than one TCP window size is required, the amount of extra buffer required naturally depends on the packet discard mechanism used and is still an open issue. Examples of this kind of problem with layering abound in practical networking. Consider, for example, the effect of IP transport's implicit assumptions of lower layers. In particular: o Packet loss: TCP assumes that packet losses are indications of congestion, but sometimes losses are from corruption on a wireless link [RFC3115]. o Reordered packets: TCP assumes that significantly reordered packets are indications of congestion. This is not always the case [FLOYD2001].
o Round-trip times: TCP measures round-trip times, and assumes that the lack of an acknowledgment within a period of time based on the measured round-trip time is a packet loss, and therefore an indication of congestion [KARN]. o Congestion control: TCP congestion control implicitly assumes that all the packets in a flow are treated the same by the network, but this is not always the case [HANDLEY].
RFC2283] is a subtly different but notable exception. In this case, the control plane is independent of the format of the control data. That is, no control plane data conversion is required, in contrast with control plane interworking models such as the ATM/IP interworking envisioned by some soft-switch manufacturers, and the so-called "PNNI-MPLS SIN" interworking [ATMMPLS]. MOLINERO2002]. Further, it is widely assumed that IP is simpler that circuit switching, and hence should be more economical to deploy and manage [MCK2002]. However, if one examines these and related assumptions, a different picture emerges (see for example [ODLYZKO98]). The following sections discuss these assumptions. BARAN]. This efficiency is based on the statistical multiplexing inherent in packet switching. However, we continue to be puzzled by what is generally believed to be the low utilization of
Internet backbones. The first question we might ask is what is the current average utilization of Internet backbones, and how does that relate to the utilization of long distance voice networks? Odlyzko and Coffman [ODLYZKO,COFFMAN] report that the average utilization of links in the IP networks was in the range between 3% and 20% (corporate intranets run in the 3% range, while commercial Internet backbones run in the 15-20% range). On the other hand, the average utilization of long haul voice lines is about 33%. In addition, for 2002, the average utilization of optical networks (all services) appears to be hovering at about 11%, while the historical average is approximately 15% [ML2002]. The question then becomes why we see such utilization levels, especially in light of the assumption that PS is inherently more efficient than CS. The reasons cited by Odlyzko and Coffman include: (i). Internet traffic is extremely asymmetric and bursty, but links are symmetric and of fixed capacity (i.e., don't know the traffic matrix, or required link capacities); (ii). It is difficult to predict traffic growth on a link, so operators tend to add bandwidth aggressively; (iii). Falling prices for coarser bandwidth granularity make it appear more economical to add capacity in large increments. Other static factors include protocol overhead, other kinds of equipment granularity, restoration capacity, and provisioning lag time all contribute to the need to "over-provision" [MC2001].
MCK2002]. This difference in software complexity has tended to make Internet routers unreliable, and has notable other second order effects (e.g., it may take a long time to reboot such a router). As another point of comparison, consider that the AT&T (Lucent) 5ESS class 5 switch, which has a huge number of calling features, requires only about twice the number of lines of code as an Internet core router [EICK]. Finally, since routers are as much or more software than hardware devices, another result of the code complexity is that the cost of routers benefits less from Moore's Law than less software-intensive devices. This causes a bandwidth/device trade-off that favors bandwidth more than less software-intensive devices. MOLINERO2002]. Consider the case of a high-speed Internet router line card: An OC192 POS router line card contains at least 30 million gates in ASICs, at least one CPU, 300 Mbytes of packet buffers, 2 Mbytes of forwarding table, and 10 Mbytes of other
state memory. On the other hand, a comparable transport switch line card has 7.5 million logic gates, no CPU, no packet buffer, no forwarding table, and an on-chip state memory. Rather, the line-card of an electronic transport switch typically contains a SONET framer, a chip to map ingress time-slots to egress time-slots, and an interface to the switch fabric. PMC]. CISCO,CIENA], and sell for about one-third as much per Gigabit/sec. Optical (OOO) technology pushes this complexity difference further (e.g., tunable lasers, MEMs switches. e.g., [CALIENT]), and DWDM multiplexers provide technology to build extremely high capacity, low power transport switches. A related metric is physical footprint. In general, by virtue of their higher density, transport switches have a smaller "per-gigabit" physical footprint.
ODLYZKO98A]. This phenomenon is also described in [PERROW]. PARK]. As such, the sections above seek to describe the complexity in terms of observable artifacts. With this in mind, it is clear that the fundamental driver producing the increased complexities outlined above is the hop-by-hop independence (HBHI) inherent in the IP architecture. This is in contrast to the end to end architectures such as ATM or Frame Relay. [WILLINGER2002] describes this phenomenon in terms of the robustness requirement of the original Internet design, and how this requirement has the driven complexity of the network. In particular, they describe a "complexity/robustness" spiral, in which increases in complexity create further and more serious sensitivities, which then requires additional robustness (hence the spiral).
The important lesson of this section is that the Simplicity Principle, while applicable to circuit switching as well as packet switching, is crucial in controlling the complexity (and hence OPEX and CAPEX properties) of packet networks. This idea is reinforced by the observation that while packet switching is a younger, less mature discipline than circuit switching, the trend in packet switches is toward more complex line cards, while the complexity of circuit switches appears to be scaling linearly with line rates and aggregate capacity. MC2001] and elsewhere, much of the complexity we observe in today's Internet is directed at increasing bandwidth utilization. As a result, the desire of network engineers to keep network utilization below 50% has been termed "over-provisioning". However, this use of the term over-provisioning is a misnomer. Rather, in modern Internet backbones the unused capacity is actually protection capacity. In particular, one might view this as "1:1 protection at the IP layer". Viewed in this way, we see that an IP network provisioned to run at 50% utilization is no more over-provisioned than the typical SONET network. However, the important advantages
that accrue to an IP network provisioned in this way include close to speed of light delay and close to zero packet loss [FRALEIGH]. These benefits can been seen as a "side-effect" of 1:1 protection provisioning. There are also other, system-theoretic reasons for providing 1:1-like protection provisioning. Most notable among these reasons is that packet-switched networks with in-band control loops can become unstable and can experience oscillations and synchronization when congested. Complex and non-linear dynamic interaction of traffic means that congestion in one part of the network will spread to other parts of the network. When routing protocol packets are lost due to congestion or route-processor overload, it causes inconsistent routing state, and this may result in traffic loops, black holes, and lost connectivity. Thus, while statistical multiplexing can in theory yield higher network utilization, in practice, to maintain consistent performance and a reasonably stable network, the dynamics of the Internet backbones favor 1:1 provisioning and its side effects to keep the network stable and delay low. BARAN77]. Today we refer to this phenomenon as "the myth of five nines". Specifically, so-called five nines reliability in packet network elements is consider a myth for the following reasons: First, since 80% of unscheduled outages are caused by people or process errors [SCOTT], there is only a 20% window in which to optimize. Thus, in order to increase component reliability, we add complexity (optimization frequently leads to complexity), which is the root cause of 80% of the unplanned outages. This effectively narrows the 20% window (i.e., you increase the likelihood of people and process failure). This phenomenon is also characterized as a "complexity/robustness" spiral [WILLINGER2002], in which increases in complexity create further and more serious sensitivities, which then requires additional robustness, and so on (hence the spiral). The conclusion, then is that while a system like the Internet can reach five-nines-like reliability, it is undesirable (and likely impossible) to try to make any individual component, especially the most complex ones, reach that reliability standard.
HOT] attempts to characterize complexity in general terms (HOT is one recent attempt to develop a general framework for the study of complexity, and is a member of a family of abstractions generally termed "the new science of complexity" or "complex adaptive
systems"). Tolerance, in HOT semantics, means that "robustness in complex systems is a constrained and limited quantity that must be carefully managed and protected." One focus of the HOT model is to characterize heavy-tailed distributions such as Complexity(A) in the above example (other examples include forest fires, power outages, and Internet traffic distributions). In particular, Complexity(A) attempts to map the extreme heterogeneity of the parts of the system (Internet), and the effect of their organization into highly structured networks, with hierarchies and multiple scales. ZHANG] is missing as you move closer to the edge, as well as having complex interactions with backoffice and CRM systems. Examples of architectures that haven't found a market due to this effect include TINA-based CRM systems, CORBA/TINA based service architectures. The basic lesson here was that the only possibilities for deploying these systems were "Limited scale deployments (such) as in Starvision can avoid coping with major unproven scalability issues", or "Otherwise need massive investments (like the carrier- grade ORB built almost from scratch)" [TINA]. In other words, these systems had complex service delivery paths, and were too complex to be feasibly deployed.
ponenda sine neccesitate" or "plurality should not be posited without necessity." (hence Occam's Razor is sometimes called "the principle of unnecessary plurality" and " the principle of simplicity"). A perhaps more contemporary formulation of Occam's Razor states that the simplest explanation for a phenomenon is the one preferred by nature. Other formulations of the same idea can be found in the KISS (Keep It Simple Stupid) principle and the Principle of Least Astonishment (the assertion that the most usable system is the one that least often leaves users astonished). [WILLINGER2002] provides a more theoretical discussion of "robustness through simplicity", and in discussing the PSTN, [KUHN87] states that in most systems, "a trade-off can be made between simplicity of interactions and looseness of coupling". When applied to packet switched network architectures, the Simplicity Principle has implications that some may consider heresy, e.g., that highly converged approaches are likely to be less efficient than "less converged" solutions. Otherwise stated, the "optimal" convergence layer may be much lower in the protocol stack that is conventionally believed. In addition, the analysis above leads to several conclusions that are contrary to the conventional wisdom surrounding packet networking. Perhaps most significant is the belief that packet switching is simpler than circuit switching. This belief has lead to conclusions such as "since packet is simpler than circuit, it must cost less to operate". This study finds to the contrary. In particular, by examining the metrics described above, we find that packet switching is more complex than circuit switching. Interestingly, this conclusion is borne out by the fact that normalized OPEX for data networks is typically significantly greater than for voice networks [ML2002]. Finally, the important conclusion of this work is that for packet networks that are of the scale of today's Internet or larger, we must strive for the simplest possible solutions if we hope to build cost effective infrastructures. This idea is eloquently stated in [DOYLE2002]: "The evolution of protocols can lead to a robustness/complexity/fragility spiral where complexity added for robustness also adds new fragilities, which in turn leads to new and thus spiraling complexities". This is exactly the phenomenon that the Simplicity Principle is designed to avoid.
to find and understand the security implications inherent in its architecture, design, and implementation. [AHUJA] "The Impact of Internet Policy and Topology on Delayed Routing Convergence", Labovitz, et. al. Infocom, 2001. [ATMMPLS] "ATM-MPLS Interworking Migration Complexities Issues and Preliminary Assessment", School of Interdisciplinary Computing and Engineering, University of Missouri-Kansas City, April 2002 [BARAN] "On Distributed Communications", Paul Baran, Rand Corporation Memorandum RM-3420-PR, http://www.rand.org/publications/RM/RM3420", August, 1964. [BARAN77] "SOME PERSPECTIVES ON NETWORKS--PAST, PRESENT AND FUTURE", Paul Baran, Information Processing 77, North-Holland Publishing Company, 1977, [BRYANT] "Protocol Layering in PWE3", Bryant et al, Work in Progress. [CAIDA] http://www.caida.org [CALLIENT] http://www.calient.net/home.html [CARLSON] "Complexity and Robustness", J.M. Carlson and John Doyle, Proc. Natl. Acad. Sci. USA, Vol. 99, Suppl. 1, 2538-2545, February 19, 2002. http://www.pnas.org/cgi/doi/10.1073/pnas.012582499 [CIENA] "CIENA Multiwave CoreDiretor", http://www.ciena.com/downloads/products/ coredirector.pdf
[CISCO] http://www.cisco.com [CLARK] "The Design Philosophy of the DARPA Internet Protocols", D. Clark, Proc. of the ACM SIGCOMM, 1988. [COFFMAN] "Internet Growth: Is there a 'Moores Law' for Data Traffic", K.G. Coffman and A.M. Odlyzko, pp. 47-93, Handbook of Massive Data Stes, J. Elli, P. M. Pardalos, and M. G. C. Resende, Editors. Kluwer, 2002. [DOYLE2002] "Robustness and the Internet: Theoretical Foundations", John C. Doyle, et. al. Work in Progress. [EICK] "Visualizing Software Changes", S.G. Eick, et al, National Institute of Statistical Sciences, Technical Report 113, December 2000. [MOLINERO2002] "TCP Switching: Exposing Circuits to IP", Pablo Molinero-Fernandez and Nick McKeown, IEEE January, 2002. [FLOYD] "The Synchronization of Periodic Routing Messages", Sally Floyd and Van Jacobson, IEEE ACM Transactions on Networking, 1994. [FLOYD2001] "A Report on Some Recent Developments in TCP Congestion Control, IEEE Communications Magazine, S. Floyd, April 2001. [FRALEIGH] "Provisioning IP Backbone Networks to Support Delay- Based Service Level Agreements", Chuck Fraleigh, Fouad Tobagi, and Christophe Diot, 2002. [GRIFFIN] "What is the Sound of One Route Flapping", Timothy G. Griffin, IPAM Workshop on Large-Scale Communication Networks: Topology, Routing, Traffic, and Control, March, 2002. [HANDLEY] "On Inter-layer Assumptions (A view from the Transport Area), slides from a presentation at the IAB workshop on Wireless Internetworking", M. Handley, March 2000. [HOT] J.M. Carlson and John Doyle, Phys. Rev. E 60, 1412- 1427, 1999.
[ISO10589] "Intermediate System to Intermediate System Intradomain Routing Exchange Protocol (IS-IS)". [JACOBSON] "Congestion Avoidance and Control", Van Jacobson, Proceedings of ACM Sigcomm 1988, pp. 273-288. [KARN] "TCP vs Link Layer Retransmission" in P. Karn et al., Advice for Internet Subnetwork Designers, Work in Progress. [KUHN87] "Sources of Failure in the Public Switched Telephone Network", D. Richard Kuhn, EEE Computer, Vol. 30, No. 4, April, 1997. [L2TPV3] Lan, J., et. al., "Layer Two Tunneling Protocol (Version 3) -- L2TPv3", Work in Progress. [MC2001] "U.S Communications Infrastructure at A Crossroads: Opportunities Amid the Gloom", McKinsey&Company for Goldman-Sachs, August 2001. [MCK2002] Nick McKeown, personal communication, April, 2002. [ML2002] "Optical Systems", Merril Lynch Technical Report, April, 2002. [NAVE] "The influence of mode coupling on the non-linear evolution of tearing modes", M.F.F. Nave, et al, Eur. Phys. J. D 8, 287-297. [NEUMANN] "Cause of AT&T network failure", Peter G. Neumann, http://catless.ncl.ac.uk/Risks/9.62.html#subj2 [ODLYZKO] "Data networks are mostly empty for good reason", A.M. Odlyzko, IT Professional 1 (no. 2), pp. 67-69, Mar/Apr 1999. [ODLYZKO98A] "Smart and stupid networks: Why the Internet is like Microsoft". A. M. Odlyzko, ACM Networker, 2(5), December, 1998. [ODLYZKO98] "The economics of the Internet: Utility, utilization, pricing, and Quality of Service", A.M. Odlyzko, July, 1998. http://www.dtc.umn.edu/~odlyzko/doc/networks.html
[PARK] "The Internet as a Complex System: Scaling, Complexity and Control", Kihong Park and Walter Willinger, AT&T Research, 2002. [PERROW] "Normal Accidents: Living with High Risk Technologies", Basic Books, C. Perrow, New York, 1984. [PMC] "The Design of a 10 Gigabit Core Router Architecture", PMC-Sierra, http://www.pmc- sierra.com/products/diagrams/CoreRouter_lg.html [RFC1629] Colella, R., Callon, R., Gardner, E. and Y. Rekhter, "Guidelines for OSI NSAP Allocation in the Internet", RFC 1629, May 1994. [RFC1925] Callon, R., "The Twelve Networking Truths", RFC 1925, 1 April 1996. [RFC1958] Carpenter, B., Ed., "Architectural principles of the Internet", RFC 1958, June 1996. [RFC2283] Bates, T., Chandra, R., Katz, D. and Y. Rekhter, "Multiprotocol Extensions for BGP4", RFC 2283, February 1998. [RFC3155] Dawkins, S., Montenegro, G., Kojo, M. and N. Vaidya, "End-to-end Performance Implications of Links with Errors", BCP 50, RFC 3155, May 2001. [ROMANOV] "Dynamics of TCP over ATM Networks", A. Romanov, S. Floyd, IEEE JSAC, vol. 13, No 4, pp.633-641, May 1995. [SALTZER] "End-To-End Arguments in System Design", J.H. Saltzer, D.P. Reed, and D.D. Clark, ACM TOCS, Vol 2, Number 4, November 1984, pp 277-288. [SCOTT] "Making Smart Investments to Reduce Unplanned Downtime", D. Scott, Tactical Guidelines, TG-07-4033, Gartner Group Research Note, March 1999. [SPILLMAN] "The Law of Diminishing Returns:, W. J. Spillman and E. Lang, 1924. [STALLINGS] "Data and Computer Communications (2nd Ed)", William Stallings, Maxwell Macmillan, 1989.
[TENNENHOUSE] "Layered multiplexing considered harmful", D. Tennenhouse, Proceedings of the IFIP Workshop on Protocols for High-Speed Networks, Rudin ed., North Holland Publishers, May 1989. [THOMPSON] "Nonlinear Dynamics and Chaos". J.M.T. Thompson and H.B. Stewart, John Wiley and Sons, 1994, ISBN 0471909602. [TINA] "What is TINA and is it useful for the TelCos?", Paolo Coppo, Carlo A. Licciardi, CSELT, EURESCOM Participants in P847 (FT, IT, NT, TI) [WAKEMAN] "Layering considered harmful", Ian Wakeman, Jon Crowcroft, Zheng Wang, and Dejan Sirovica, IEEE Network, January 1992, p. 7-16. [WARD] "Custom fluorescent-nucleotide synthesis as an alternative method for nucleic acid labeling", Octavian Henegariu*, Patricia Bray-Ward and David C. Ward, Nature Biotech 18:345-348 (2000). [WILLINGER2002] "Robustness and the Internet: Design and evolution", Walter Willinger and John Doyle, 2002. [ZHANG] "Impact of Aggregation on Scaling Behavior of Internet Backbone Traffic", Sprint ATL Technical Report TR02-ATL-020157 Zhi-Li Zhang, Vinay Ribeiroj, Sue Moon, Christophe Diot, February, 2002.
Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society.