Network Working Group J. Peterson Request for Comments: 5594 NeuStar Category: Informational A. Cooper Center for Democracy & Technology July 2009 Report from the IETF Workshop on Peer-to-Peer (P2P) Infrastructure, May 28, 2008
AbstractThis document reports the outcome of a workshop organized by the Real-time Applications and Infrastructure Area Directors of the IETF to discuss network delay and congestion issues resulting from increased Peer-to-Peer (P2P) traffic volumes. The workshop was held on May 28, 2008 at MIT in Cambridge, MA, USA. The goals of the workshop were twofold: to understand the technical problems that ISPs and end users are experiencing as a result of high volumes of P2P traffic, and to begin to understand how the IETF may be helpful in addressing these problems. Gaining an understanding of where in the IETF this work might be pursued and how to extract feasible work items were highlighted as important tasks in pursuit of the latter goal. The workshop was very well attended and produced several work items that have since been taken up by members of the IETF community. Status of This Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document.
1. Introduction ....................................................3 2. Scoping of the Problem and Solution Spaces ......................4 3. Service Provider Perspective ....................................4 3.1. DOCSIS Architecture and Upstream Contention ................4 3.2. TCP Flow Fairness and Service Flows ........................5 3.3. Service Provider Responses .................................6 4. Application Provider Perspective ................................7 5. Potential Solution Areas ........................................7 5.1. Improving Peer Selection: Information Sharing, Localization, and Caches ...................................8 5.1.1. Leveraging AS Numbers ...............................9 5.1.2. P4P: Provider Portal for P2P Applications ...........9 5.1.3. Multi-Layer, Tracker-Based Architecture ............10 5.1.4. ISP-Aided Neighbor Selection .......................11 5.1.5. Caches .............................................12 5.1.6. Potential IETF Work ................................12 5.2. New Approaches to Congestion Control ......................14 5.2.1. End-to-End Congestion Control ......................15 5.2.2. Weighted Congestion Control ........................15 5.3. Quality of Service ........................................16 6. Applications Opening Multiple TCP Connections ..................17 7. Costs and Congestion ...........................................18 8. Next Steps .....................................................18 8.1. Transport Issues ..........................................19 8.2. Improved Peer Selection ...................................19 9. Security Considerations ........................................19 10. Acknowledgements ..............................................19 11. Informative References ........................................20 Appendix A. Program Committee ....................................21 Appendix B. Workshop Participants ................................21 Appendix C. Workshop Agenda ......................................24 Appendix D. Slides and Position Papers ..........................25
Appendices A and B, respectively. The agenda of the workshop can be found in Appendix C. A link to the presentations given at the workshop and the position papers submitted prior to the workshop is in Appendix D. The workshop showcased the IETF community's recognition of the impact of P2P and other high-volume applications on the Internet as a whole. Participants welcomed the opportunity to discuss potential standardization work that network operators, applications providers, and end users would all find mutually beneficial. Two transport- related work items gained significant traction: designing a protocol for very deferential end-to-end congestion control for delay-tolerant applications, and producing an informational document about the reasoning behind and effects of applications opening multiple transport connections at once. A separate area of interest that emerged at the workshop focused on improving peer selection by having
networks make more information available to applications. Finally, presenters also covered traditional approaches to multiple service- tier queuing such as Diffserv. DOCSIS] that is used for many cable systems, there may be a single Cable Modem Termination System (CMTS) serving hundreds or thousands of residential cable customers. Each CMTS has multiple
DOCSIS domains, each of which typically has a single downstream link and a number of upstream links. Each CMTS is connected through a hybrid fiber-coaxial (HFC) network to subscribers' cable modems. The limiting resource in this architecture is usually bandwidth, so bandwidth is typically the measure used for capacity planning. As with all networks, congestion manifests itself when instantaneous load exceeds available capacity. In the upstream direction, any cable modem connected to a CMTS can make a request to the CMTS to transmit, and requests are randomized to minimize collisions. With many cable modems issuing requests at once, the requests may collide, resulting in delays. DOCSIS does not specify a size for cable modem buffers, but buffer delays of one to four seconds have been observed with various cable modems from different vendors. Once the CMTS has granted a cable modem the ability to transmit its data PDU, the modem can piggyback its next request on top of that data PDU. In situations with a lot of upstream traffic, piggybacking happens more often, which sends heavy upstream users to the front of the CMTS queue, ahead of interactive but less-upstream-intensive applications. For example, if the CMTS is granting requests approximately every one to three milliseconds, then a cable modem transmitting data for a service like VoIP with a packetization delay of 20-30 milliseconds may get into contention with another modem on the same CMTS that is constantly transmitting upstream and piggybacking each new request. This may explain how heavy upstream users ultimately dominate the pipe over more interactive applications. Consequentially, it is imperative that assessments of the problem space and potential solutions are mindful of the influence that specific layer-2 networks may exert on the behavior of Internet traffic, especially when considering the alleviation of congestion in an access network.
with other modems). The service flow would have upstream capacity provisioned for it. The modem would have a separate service flow for best efforts traffic. Some ISPs provision such a flow for their own VoIP offerings; others allow subscribers to pay extra to have particular traffic assigned to a provisioned service flow. It may also be possible for an ISP to provision such a flow on the fly when it recognizes the need for it. Diffserv [RFC2475] bits set by the customer premises equipment could be used to classify flows, for example.
subscribers who pay for more bandwidth will have higher thresholds. The re-marking will not distinguish between multiple users of the same subscriber connection, so one family member's P2P usage could cause another family member's Web browsing traffic to be lowered in priority. There is no current mechanism for users to determine that their traffic has been re-marked. By temporarily reducing the traffic priority of subscribers who have been consuming bandwidth in bulk for lengthy periods, this congestion management technique aims to preserve a good user experience for subscribers with burstier traffic patterns, including those using real-time applications. As compared to an approach that reduces particular subscribers' bandwidth during periods of congestion, this technique eliminates the ability for applications to set their own priority levels, but it also avoids the negative connotations that some users may associate with bandwidth reductions. This approach involves many tweakable parameters. A large part of the trial process is aimed at determining the best settings for these parameters, but there may also be opportunities to work with the research community to identify the best way to adjust the thresholds necessary to optimize the performance of the management technique. Section 5. As uptake in P2P usage has grown, so has end-user latency. For example, a user whose uplink capacity is 250-500 Kbps and whose modem buffer has a capacity of 32-64 Kbps may easily fill the buffer (unless the modem uses Adaptive Queue Management (AQM), which is uncommon). This can result in delay on the order of seconds, with disastrous effects on application performance. On a cable system with shared capacity between neighbors, one neighbor could saturate the buffer and affect the latency of another neighbor's traffic. Even users with dedicated bandwidth can experience delays as their own P2P traffic saturates the link and dominates their own more latency-sensitive traffic.
or more of three topic areas: improving peer selection, new approaches to congestion control, and quality-of-service mechanisms. The workshop discussions and outcomes in each area are described below.
experiences will not necessarily be uniform depending on the scope of the information provided and the peer's location. Localization information could form one component of a peer-selection decision, but it will likely need to be balanced against other factors. Workshop participants discussed both current research efforts in this area and how IETF standards work may be useful in furthering the general concept of improved peer selection. Those discussions are summarized below. P4P] aims to design a framework to enable cooperation between providers and applications (including P2P), where "providers" may be ISPs, content
distribution networks, or caching services. In this architecture, each provider can communicate information to P2P clients through a portal known as an iTracker. An iTracker could be identified through a DNS SRV record (perhaps with its own new record type), a whois look-up, or a trusted third party. An iTracker has different interfaces for different types of information that the provider may want to share. The core interface allows the provider to express the "virtual cost" of its intradomain or interdomain links. Virtual cost may reflect any kind of provider preferences and may be based on the provider's choice of metrics, including utilization, transit costs, or geography. It is up to the provider to decide how dynamic it wants to be in updating its virtual cost determinations. In tests of this framework, two parallel swarms were created with approximately the same number of clients and similar geographical and network distributions, both sharing the same file. One of the swarms used the P4P framework, with the ISP's network topology map as input to its iTracker, and the other swarm used traditional peer selection. The swarm without P4P saw 98% of traffic to and from peers external to the ISP, whereas with P4P that number was 50%. Download completion times for the P4P-enabled swarm improved approximately 20% on average.
One challenge in this architecture is determining what "local" means for trackers, seeds, and peers. Locality could be dependent on traffic conditions, load balancing, static topology, policy, or some other metric. These same considerations would also be crucial for determining appropriate cache placement in a hybrid network. This architecture presents in the abstract the problem of re- directing from a global entity to a local entity. Client queries need to find their way to the appropriate local tracker. This can be accomplished through an off-path, explicit mechanism where local trackers register with the global tracker in advance, or through an on-path approach where the network proxies P2P requests. The off- path tracker format approach is preferable for performance and reliability reasons. Inasmuch as the multi-layer scheme might require ISPs to aid peers in finding optimal paths to unauthorized copies of copyrighted content, ISPs may be concerned about the legal liability of participating.
The main challenge in implementing the oracle service is scalability. If peers need to communicate to the oracle the IP address of every peer they know, the size of oracle requests may be inordinately large. Additionally, today's largest swarms approach 10000 peers, and with every peer requesting a sorted list, the oracle request volume will swell. With the growth of business models dependent upon P2P for distribution of content, swarms in the future may be far larger, further exacerbating the problem. Potential mitigations include having trackers, instead of peers, issue oracle requests, and using other peers' sorted lists as input, rather than always using an unsorted list. On the other hand, this approach is advantageous from a legal liability perspective, because it does not require ISPs to have any knowledge of where particular content might be located or to have any role in directing peers to particular content.
Against this backdrop, the key task for the IETF (as identified at the workshop) is to pinpoint incrementally beneficial work items in the spaces discussed above. In the future, it may be possible to standardize entire P2P mechanisms but, as a starting point, it makes more sense to single out core manageable components for standardization. The focus should be on items that are not so specific to one ISP or P2P network that standardization is rendered useless. Ideally, any mechanisms resulting from this work might apply to future applications that exhibit the same bandwidth- intensive properties as today's P2P file-sharing. In considering any of these items, it will be necessary to ensure that the information exchanged by networks and applications does not harm any of the parties involved. Not every piece of information exchanged will be beneficial or verifiable, and this fact must be recognized and accounted for. Solutions that leave applications or networks worse off than they already are today will not gain any traction. It should also not be assumed that a particular party will be best suited to provide a particular kind of information. For example, an ISP may not know what the connection costs are in other ISPs' networks, whereas an overlay network that receives cost information from several ISPs may have a better handle on this kind of data. Standardization of information sharing should not assume the identity of particular parties doing the sharing. The list of potential work items discussed at the workshop is provided below. Workshop participants showed particular interest in pursuing the first three items further.
With Diffserv, a light user may waste his or her priority connecting to a heavy user on another network, which is not a problem with host- controlled weighting. Weighted congestion control is just one example of a generalized set of features that characterize useful approaches to congestion control. These characteristics include full user control of priorities within a user's own scope and no possibility of interpreting ISP behavior as discriminatory. The former means that ISPs should not override user decisions arbitrarily (though this does not preclude an ISP from offering prioritization as an option). The latter means that the metric for decision-making needs to obviate suspicion of ISP motivations. One metric that meets these criteria is a harm (cost) metric, where cost is equal to the amount of data that was not served to its destination. Using this metric, cost is greatest when traffic peaks are greatest. It allows for a policy of not sending too much data during times of congestion, without specifying exactly how much is too much. The cost metric could be used either for a Diffserv approach or for weighted congestion control. One important limitation on ISPs from a congestion-control perspective is that they do not have a window into congestion on other ISPs' networks. Solving this problem requires a separate mechanism to express congestion across domains. One potential avenue for the IETF or IRTF to pursue would be to establish a long-term design team to assess congestion problems in general and the long-term effects of any proposed quick fixes. These issues are not necessarily confined to P2P and should be viewed in the broader context of massive increases in bandwidth use.
Quality of service (QoS) arrangements may be more suitable in the long term. One approach that distinguishes certain classes of traffic during times of congestion was described in Section 3.3. A standardized mechanism that may be useful for implementing QoS is Differentiated Services Code Points (DSCP) [RFC2474]. With DSCP, devices at the edge of the network mark packets with the service level they should receive. Nodes within the network do not need to remember anything about service flows, and applications do not need to request a particular service level. Users effectively avoid self-interference through service classification. Although DSCP may have many uses, perhaps the most relevant to the P2P congestion issue is its ability to facilitate usage-based charging. User pricing agreements that charge a premium for real- time traffic and best-effort traffic could potentially shape user behavior, resulting in reduced congestion (although ISPs would need a mechanism to mitigate the risk of charging subscribers for things like unintentional malware downloads or DoS attacks). DSCP could also be used to limit a user's supply of high-priority bandwidth, resulting in a similar effect. Equipment to support DSCP is already available. Although there has been some concern about a perceived lack of DSCP deployment, it is widely used by enterprise customers, and growth has been strong due to uptake in VoIP at the enterprise level. However, DSCP still faces deployment hurdles on many networks. Perhaps the largest barrier of all to wide deployment is the lack of uniform code points to be used across networks. For example, the latest Windows Vista API marks voice traffic as CS7, above the priority reserve for router traffic. To properly take advantage of this change, every switch will need to re-mark all traffic. In addition, disparate ISPs are currently without a means of verifying each others' markings and thus may be unwilling to trust the markings they receive.
connections. A single user with multiple open connections is not necessarily a problem on its face, but the fact that fairness is determined per flow rather than per user leaves that impression. Workshop participants thought it may be useful for the IETF to provide some information about such situations.
Section 5.2.1). The second was an informational document that describes the current practice of P2P applications opening multiple transport connections and that makes recommendations about the best practices for doing so (as discussed in Section 6). Section 5.1 that support mechanisms for information sharing between networks and applications to help improve peer selection. Adding to the appeal of this topic is its potential utility for applications other than P2P that may also benefit from information about the network. Because the scope of potential solutions discussed at the workshop was broad, extracting out the most feasible pieces to pursue is the necessary first step. Appendix A) for their time and energy in organizing the workshop, reviewing the position papers, and crafting an event of value for all participants. The IETF would also like to thank the scribes, Spencer Dawkins and Alissa Cooper, who diligently recorded the proceedings during the workshop.
A special thanks to all the participants in the workshop (listed in Appendix B) who took the time, came to the workshop to participate in the discussions, and put in the effort to make this workshop a success. The IETF especially appreciates the effort of those that prepared and made presentations at the workshop. [DOCSIS] CableLabs, "DOCSIS Specifications - DOCSIS 2.0 Interface", 2008, <http://www.cablemodem.com/specifications/ specifications20.html>. [P4P] Xie, H., Yang, Y., Krishnamurthy, A., and A. Silberschatz, "P4P: Provider Portal for Applications", August 2008, <http://uwnews.org/relatedcontent/2008/August/ rc_parentID43281_thisID43282.pdf>. [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998. [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998.
Rex Bullinger, National Cable & Telecommunications Association Gonzalo Camarillo, Ericsson Mary-Luc Champel, Thomson William Check, NCTA Alissa Cooper, Center for Democracy and Technology Patrick Crowley, Washington University Leslie Daigle, Internet Society Spencer Dawkins John Dickinson, Bright House Networks Lisa Dusseault, CommerceNet Lars Eggert, Nokia Research Center Joe Godas, Cablevision Vernon Groves, Microsoft Daniel Grunberg, Immedia Semiconductor Carmen Guerrero, University Carlos III Madrid Vijay Gurbani, Bell Laboratories/Alcatel-Lucent William Hawkins III, ITT Volker Hilt, Bell Labs, Alcatel-Lucent Russell Housley, Vigil Security, LLC Robert Jackson, Camiant Cullen Jennings, Cisco Systems Paul Jessop, RIAA XingFeng Jiang, Huawei Michael Kelsen, Time Warner Cable
Tom Klieber, Comcast Eric Klinker, BitTorrent Inc. Umesh Krishnaswamy Gregory Lebovitz, Juniper Erran Li, Bell-Labs Jason Livingood, Comcast Andrew Malis, Verizon Enrico Marocco, Telecom Italia Lab Marcin Matuszewski, Nokia Danny McPherson, Arbor Networks, Inc. Michael Merritt, AT&T Lyle Moore, Bell Canada John Morris, Center for Democracy and Technology Jean-Francois Mule, Cablelabs David Oran, Cisco Systems Reinaldo Penno, Juniper Networks Jon Peterson, NeuStar Howard Pfeffer, Time Warner Cable Laird Popkin, Pando Networks Stefano Previdi, Cisco systems Satish Putta Eric Pescorla Benny Rodrig, Avaya Damien Saucez, UCLouvain (UCL)
Henning Schulzrinne, Columbia University Michael Sheehan, Juniper Networks Don Shulzrinne, Immedia Semiconductor David Sohn, Center for Democracy and Technology Martin Stiemerling, NEC Clint Summers, Cox Communications Robert Topolski Mark Townsley, Cisco Systems Yushun Wang, Microsoft Hao Wang, Yale University Ye Wang, Yale University David Ward, Cisco Nicholas Weaver, ICSI Daniel Weitzner, MIT Magnus Westerlund, Ericsson Thomas Woo, Bell Labs Steve Worona, EDUCAUSE Richard Woundy, Comcast Haiyong Xie Richard Yang, Yale University
3. Application Designer Perspective (BitTorrent) Stanislav Shalunov 4. Lightning Talks & General Discussion Robb Topoloski Nick Weaver Leslie Daigle 5. Localization and Caches Laird Popkin and Haiyong Xie Yu-Shun Wang Vinay Aggrawal 6. New Approaches to Congestion Bob Briscoe Marcin Matuszewski 7. Quality of Service Mary Barnes Henning Schulzrinne 8. Conclusions & Wrap-Up http:// trac.tools.ietf.org/area/rai/trac/wiki/PeerToPeerInfrastructure. Position papers: Nick Weaver - The case for "Ugly Now" User Fairness Paul Jessop - Position paper of the RIAA Nikloaos Laotaris, Pablo Rodriguez, Laurent Massoulie - ECHOES: Edge Capacity Hosting Overlays of Nano Data Centers Bruce Davie, Stefano Previdi, Jan Medved, Albert Tian - Peer Selection Guidance Marie-Jose Montpetit - Community Networks: getting P2P out of prison - the next steps D. Bryan, S. Dawkins, B. Lowekamp, E. Shim - Infrastructure-related Attributes of App Scenarios for P2PSIP Jiang XingFeng - Analysis of the Service Discovery in DHT network
R. Penno - P2P Status and Requirements Patrick Crowley and Shakir James - Symbiotic P2P: Resolving the conflict between ISPs and BitTorrent through mutual cooperation Robb Topolski - Framing Peer to Peer File Sharing M. Stiemerling, S. Niccolini, S. Kiesel, J. Seedorf - A Network Cooperative Overlay System Y. Wang, S. Tan, R. Grove - Traffic Localization with Multi-Layer, Tracker-Based Peer-to-Peer Content Distribution Architecture Haiyong Xie, Y. Richard Yang, Avi Silberschatz, Arvind Krishnamurthy, Laird Popkin - P4P: Provider Portal for P2P Applications Michael Merritt, Doug Pasko, Laird Popkin - Network-Friendly Peer-to- Peer Services Camiant (Jackson) - Camiant Submission Jason Livingood, Rich Woundy - Comcast Submission Benny Rodrig - Enterprise IP Networks and the P2P Traffic Load Impact Ted Hardie - Peer-to-Peer traffic and "Unattended Consequences" Jiang XingFeng, Ning Zong - Content Replication for Internet P2P Applications Sandvine (Dundas) - Analysis of Traffic Demographics in Broadband networks Sandvine (Dundas) - Traffic Management in a World with Network Neutrality Stanislav Shalunov - Users want P2P, we make it work R. Cuevas, A. Cuevas, I. Martinez-Yelmo, C. Guerrero - Internet scale mobility service: a case study on building a DHT based service for ISPs M. Barnes, B. McCormick - Peer to Peer Infrastructure Considerations Henning Schulzrinne - Encouraging Bandwidth Efficiency for Peer-to- Peer Applications
Damien Saucez, Benoit Donnet, Olivier Bonaventure, Dimitri Papdimitriou - Towards an Open Path Selection Architecture Eric Rescorla - Notes on P2P Blocking and Evasion Vinay Aggrawal, Anja Feldmann - ISP-Aided Neighbor Selection in P2P Systems Enrico Marocco, Vijay K. Gurbani, Volker Hilt, Ivica Rimac, Marco Tomsu - Peer-to-Peer Infrastructure: A Survey of Research on the Application-Layer Traffic Optimization Problem and the Need for Layer Cooperation Tony Moncaster, Bob Briscoe, Louise Burness - Is There a Problem With Peer-to-peer Traffic? David Sohn, Alissa Cooper - Peer-to-Peer Infrastructure Considerations Bob Briscoe, Lou Burness, Tony Moncaster, Phil Eardley - Solving this traffic management problem... and the next, and the next Hannes Tschofenig, Marcin Matuszewski - Dealing with P2P Traffic in an Operator Network Jean-Francois Mule - CableLabs submission Alan Arolovitch - Peer-to-peer infrastructure: Case for cooperative P2P caching Leslie Daigle - Defining Success: Questions for the Future of the Internet and Bandwidth-Intensive Activities William Check, Rex Bullinger -- NCTA Position Paper Jari Arkko - Incentives and Deployment Considerations for P2PI Solutions