tech-invite   World Map     

3GPP     Specs     Glossaries     Architecture     IMS     UICC       IETF     RFCs     Groups     SIP     ABNFs       Search

RFC 8041

 
 
 

Use Cases and Operational Experience with Multipath TCP

Part 2 of 2, p. 16 to 30
Prev Section

 


prevText      Top      ToC       Page 16 
3.6.  Closing Subflows

                    client                       server
                       |                           |
   MPTCP: ESTABLISHED  |                           | MPTCP: ESTABLISHED
   Sub: ESTABLISHED    |                           | Sub: ESTABLISHED
                       |                           |
                       |         DATA_FIN          |
   MPTCP: CLOSE-WAIT   | <------------------------ | close()   (step 1)
   Sub: ESTABLISHED    |         DATA_ACK          |
                       | ------------------------> | MPTCP: FIN-WAIT-2
                       |                           | Sub: ESTABLISHED
                       |                           |
                       |  DATA_FIN + subflow-FIN   |
   close()/shutdown()  | ------------------------> | MPTCP: TIME-WAIT
   (step 2)            |        DATA_ACK           | Sub: CLOSE-WAIT
   MPTCP: CLOSED       | <------------------------ |
   Sub: FIN-WAIT-2     |                           |
                       |                           |
                       |        subflow-FIN        |
   MPTCP: CLOSED       | <------------------------ | subflow-close()
   Sub: TIME-WAIT      |        subflow-ACK        |
   (step 3)            | ------------------------> | MPTCP: TIME-WAIT
                       |                           | Sub: CLOSED
                       |                           |


    Figure 5: Multipath TCP may not be able to avoid time-wait state on
    the subflow (indicated as Sub in the drawing), even if enforced by
                    the application on the client-side.

   Figure 5 shows a very particular issue within Multipath TCP.  Many
   high-performance applications try to avoid TIME-WAIT state by
   deferring the closure of the connection until the peer has sent a
   FIN.  That way, the client on the left of Figure 5 does a passive
   closure of the connection, transitioning from CLOSE-WAIT to Last-ACK
   and finally freeing the resources after reception of the ACK of the
   FIN.  An application running on top of an MPTCP-enabled Linux kernel
   might also use this approach.  The difference here is that the
   close() of the connection (step 1 in Figure 5) only triggers the

Top      Up      ToC       Page 17 
   sending of a DATA_FIN.  Nothing guarantees that the kernel is ready
   to combine the DATA_FIN with a subflow-FIN.  The reception of the
   DATA_FIN will make the application trigger the closure of the
   connection (step 2), trying to avoid TIME-WAIT state with this late
   closure.  This time, the kernel might decide to combine the DATA_FIN
   with a subflow-FIN.  This decision will be fatal, as the subflow's
   state machine will not transition from CLOSE_WAIT to Last-ACK, but
   rather go through FIN_WAIT-2 into TIME-WAIT state.  The TIME-WAIT
   state will consume resources on the host for at least 2 MSL (Maximum
   Segment Lifetime).  Thus, a smart application that tries to avoid
   TIME-WAIT state by doing late closure of the connection actually ends
   up with one of its subflows in TIME-WAIT state.  A high-performance
   Multipath TCP kernel implementation should honor the desire of the
   application to do passive closure of the connection and successfully
   avoid TIME-WAIT state -- even on the subflows.

   The solution to this problem lies in an optimistic assumption that a
   host doing active-closure of a Multipath TCP connection by sending a
   DATA_FIN will soon also send a FIN on all its subflows.  Thus, the
   passive closer of the connection can simply wait for the peer to send
   exactly this FIN -- enforcing passive closure even on the subflows.
   Of course, to avoid consuming resources indefinitely, a timer must
   limit the time our implementation waits for the FIN.

3.7.  Packet Schedulers

   In a Multipath TCP implementation, the packet scheduler is the
   algorithm that is executed when transmitting each packet to decide on
   which subflow it needs to be transmitted.  The packet scheduler
   itself does not have any impact on the interoperability of Multipath
   TCP implementations.  However, it may clearly impact the performance
   of Multipath TCP sessions.  The Multipath TCP implementation in the
   Linux kernel supports a pluggable architecture for the packet
   scheduler [PaaschPhD].  As of this writing, two schedulers have been
   implemented: round-robin and lowest-rtt-first.  The second scheduler
   relies on the round-trip time (rtt) measured on each TCP subflow and
   sends first segments over the subflow having the lowest round-trip
   time.  They are compared in [CSWS14].  The experiments and
   measurements described in [CSWS14] show that the lowest-rtt-first
   scheduler appears to be the best compromise from a performance
   viewpoint.  Another study of the packet schedulers is presented in
   [PAMS2014].  This study relies on simulations with the Multipath TCP
   implementation in the Linux kernel.  They compare the lowest-rtt-
   first with the round-robin and a random scheduler.  They show some
   situations where the lowest-rtt-first scheduler does not perform as
   well as the other schedulers, but there are many scenarios where the

Top      Up      ToC       Page 18 
   opposite is true.  [PAMS2014] notes that "it is highly likely that
   the optimal scheduling strategy depends on the characteristics of the
   paths being used."

3.8.  Segment Size Selection

   When an application performs a write/send system call, the kernel
   allocates a packet buffer (sk_buff in Linux) to store the data the
   application wants to send.  The kernel will store at most one MSS
   (Maximum Segment Size) of data per buffer.  As the MSS can differ
   amongst subflows, an MPTCP implementation must select carefully the
   MSS used to generate application data.  The Linux kernel
   implementation had various ways of selecting the MSS: minimum or
   maximum amongst the different subflows.  However, these heuristics of
   MSS selection can cause significant performance issues in some
   environments.  Consider the following example.  An MPTCP connection
   has two established subflows that respectively use an MSS of 1420 and
   1428 bytes.  If MPTCP selects the maximum, then the application will
   generate segments of 1428 bytes of data.  An MPTCP implementation
   will have to split the segment in two (1420-byte and 8-byte) segments
   when pushing on the subflow with the smallest MSS.  The latter
   segment will introduce a large overhead as this single data segment
   will use 2 slots in the congestion window (in packets) therefore
   reducing by roughly twice the potential throughput (in bytes/s) of
   this subflow.  Taking the smallest MSS does not solve the issue as
   there might be a case where the subflow with the smallest MSS only
   sends a few packets, therefore reducing the potential throughput of
   the other subflows.

   The Linux implementation recently took another approach [DetalMSS].
   Instead of selecting the minimum and maximum values, it now
   dynamically adapts the MSS based on the contribution of all the
   subflows to the connection's throughput.  For each subflow, it
   computes the potential throughput achieved by selecting each MSS
   value and by taking into account the lost space in the congestion
   window.  It then selects the MSS that allows to achieve the highest
   potential throughput.

   Given the prevalence of middleboxes that clamp the MSS, Multipath TCP
   implementations must be able to efficiently support subflows with
   different MSS values.  The strategy described above is a possible
   solution to this problem.

Top      Up      ToC       Page 19 
3.9.  Interactions with the Domain Name System

   Multihomed clients such as smartphones can send DNS queries over any
   of their interfaces.  When a single-homed client performs a DNS
   query, it receives from its local resolver the best answer for its
   request.  If the client is multihomed, the answer in response to the
   DNS query may vary with the interface over which it has been sent.

                      cdn1
                       |
           client -- cellular -- internet -- cdn3
              |                   |
              +----- wifi --------+
                       |
                     cdn2


                     Figure 6: Simple Network Topology

   If the client sends a DNS query over the WiFi interface, the answer
   will point to the cdn2 server while the same request sent over the
   cellular interface will point to the cdn1 server.  This might cause
   problems for CDN providers that locate their servers inside ISP
   networks and have contracts that specify that the CDN server will
   only be accessed from within this particular ISP.  Assume now that
   both the client and the CDN servers support Multipath TCP.  In this
   case, a Multipath TCP session from cdn1 or cdn2 would potentially use
   both the cellular network and the WiFi network.  Serving the client
   from cdn2 over the cellular interface could violate the contract
   between the CDN provider and the network operators.  A similar
   problem occurs with regular TCP if the client caches DNS replies.
   For example, the client obtains a DNS answer over the cellular
   interface and then stops this interface and starts to use its WiFi
   interface.  If the client retrieves data from cdn1 over its WiFi
   interface, this may also violate the contract between the CDN and the
   network operators.

   A possible solution to prevent this problem would be to modify the
   DNS resolution on the client.  The client subnet Extension Mechanisms
   for DNS (EDNS) defined in [RFC7871] could be used for this purpose.
   When the client sends a DNS query from its WiFi interface, it should
   also send the client subnet corresponding to the cellular interface
   in this request.  This would indicate to the resolver that the answer
   should be valid for both the WiFi and the cellular interfaces (e.g.,
   the cdn3 server).

Top      Up      ToC       Page 20 
3.10.  Captive Portals

   Multipath TCP enables a host to use different interfaces to reach a
   server.  In theory, this should ensure connectivity when at least one
   of the interfaces is active.  However, in practice, there are some
   particular scenarios with captive portals that may cause operational
   problems.  The reference environment is shown in Figure 7.

           client -----  network1
                |
                +------- internet ------------- server

                    Figure 7: Issue with Captive Portal

   The client is attached to two networks: network1 that provides
   limited connectivity and the entire Internet through the second
   network interface.  In practice, this scenario corresponds to an open
   WiFi network with a captive portal for network1 and a cellular
   service for the second interface.  On many smartphones, the WiFi
   interface is preferred over the cellular interface.  If the
   smartphone learns a default route via both interfaces, it will
   typically prefer to use the WiFi interface to send its DNS request
   and create the first subflow.  This is not optimal with Multipath
   TCP.  A better approach would probably be to try a few attempts on
   the WiFi interface and then, upon failure of these attempts, try to
   use the second interface for the initial subflow as well.

3.11.  Stateless Webservers

   MPTCP has been designed to interoperate with webservers that benefit
   from SYN-cookies to protect against SYN-flooding attacks [RFC4987].
   MPTCP achieves this by echoing the keys negotiated during the
   MP_CAPABLE handshake in the third ACK of the three-way handshake.
   Reception of this third ACK then allows the server to reconstruct the
   state specific to MPTCP.

   However, one caveat to this mechanism is the unreliable nature of the
   third ACK.  Indeed, when the third ACK gets lost, the server will not
   be able to reconstruct the MPTCP state.  MPTCP will fall back to
   regular TCP in this case.  This is in contrast to regular TCP.  When
   the client starts sending data, the first data segment also includes
   the SYN-cookie, which allows the server to reconstruct the TCP-state.
   Further, this data segment will be retransmitted by the client in
   case it gets lost and thus is resilient against loss.  MPTCP does not
   include the keys in this data segment and thus the server cannot
   reconstruct the MPTCP state.

Top      Up      ToC       Page 21 
   This issue might be considered as a minor one for MPTCP.  Losing the
   third ACK should only happen when packet loss is high; in this case,
   MPTCP provides a lot of benefits as it can move traffic away from the
   lossy link.  It is undesirable that MPTCP has a higher chance to fall
   back to regular TCP in those lossy environments.

   [MPTCP-DEPLOY] discusses this issue and suggests a modified handshake
   mechanism that ensures reliable delivery of the MP_CAPABLE, following
   the three-way handshake.  This modification will make MPTCP reliable,
   even in lossy environments when servers need to use SYN-cookies to
   protect against SYN-flooding attacks.

3.12.  Load-Balanced Server Farms

   Large-scale server farms typically deploy thousands of servers behind
   a single virtual IP (VIP).  Steering traffic to these servers is done
   through Layer 4 load-balancers that ensure that a TCP-flow will
   always be routed to the same server [Presto08].

   As Multipath TCP uses multiple different TCP subflows to steer the
   traffic across the different paths, load-balancers need to ensure
   that all these subflows are routed to the same server.  This implies
   that the load-balancers need to track the MPTCP-related state,
   allowing them to parse the token in the MP_JOIN and assign those
   subflows to the appropriate server.  However, server farms typically
   deploy several load-balancers for reliability and capacity reasons.
   As a TCP subflow might get routed to any of these load-balancers,
   they would need to synchronize the MPTCP-related state -- a solution
   that is not feasible on a large scale.

   The token (carried in the MP_JOIN) contains the information
   indicating to which MPTCP-session the subflow belongs.  As the token
   is a hash of the key, servers are not able to generate the token in
   such a way that the token can provide the necessary information to
   the load-balancers, which would allow them to route TCP subflows to
   the appropriate server.  [MPTCP-LOAD] discusses this issue in detail
   and suggests two alternative MP_CAPABLE handshakes to overcome these.

4.  Security Considerations

   This informational document discusses use cases and operational
   experience with Multipath TCP.  An extensive analysis of the
   remaining security issues in the Multipath TCP specification has been
   published in [RFC7430], together with suggestions for possible
   solutions.

Top      Up      ToC       Page 22 
   From a security viewpoint, it is important to note that Multipath
   TCP, like other multipath solutions such as SCTP, has the ability to
   send packets belonging to a single connection over different paths.
   This design feature of Multipath TCP implies that middleboxes that
   have been deployed on-path assuming that they would observe all the
   packets exchanged for a given connection in both directions may not
   function correctly anymore.  A typical example are firewalls,
   Intrusion Detection System (IDS) or deep packet inspections (DPIs)
   deployed in enterprise networks.  Those devices expect to observe all
   the packets from all TCP connections.  With Multipath TCP, those
   middleboxes may not observe anymore all packets since some of them
   may follow a different path.  The two examples below illustrate
   typical deployments of such middleboxes.  The first example,
   Figure 8, shows an MPTCP-enabled smartphone attached to both an
   enterprise and a cellular network.  If a Multipath TCP connection is
   established by the smartphone towards a server, some of the packets
   sent by the smartphone or the server may be transmitted over the
   cellular network and thus be invisible for the enterprise middlebox.

     smartphone +----- enterprise net --- MBox----+------ server
                |                                 |
                +----- cellular net  -------------+

              Figure 8: Enterprise Middlebox May Not Observe
                     All Packets from Multihomed Host

   The second example, Figure 9, shows a possible issue when multiple
   middleboxes are deployed inside a network.  For simplicity, we assume
   that network1 is the default IPv4 path while network2 is the default
   IPv6 path.  A similar issue could occur with per-flow load-balancing
   such as ECMP [RFC2992].  With regular TCP, all packets from each
   connection would either pass through Mbox1 or Mbox2.  With Multipath
   TCP, the client can easily establish a subflow over network1 and
   another over network2 and each middlebox would only observe a part of
   the traffic of the end-to-end Multipath TCP connection.

     client ----R-- network1  --- MBox1 -----R------------- server
                |                            |
                +-- network2  --- MBox2 -----+

                      Figure 9: Interactions between
                  Load-Balancing and Security Middleboxes

   In these two cases, it is possible for an attacker to evade some
   security measures operating on the TCP byte stream and implemented on
   the middleboxes by controlling the bytes that are actually sent over
   each subflow and there are tools that ease those kinds of evasion
   [PZ15] [PT14].  This is not a security issue for Multipath TCP itself

Top      Up      ToC       Page 23 
   since Multipath TCP behaves correctly.  However, this demonstrates
   the difficulty of enforcing security policies by relying only on
   on-path middleboxes instead of enforcing them directly on the
   endpoints.

5.  References

5.1.  Normative References

   [RFC6182]  Ford, A., Raiciu, C., Handley, M., Barre, S., and J.
              Iyengar, "Architectural Guidelines for Multipath TCP
              Development", RFC 6182, DOI 10.17487/RFC6182, March 2011,
              <http://www.rfc-editor.org/info/rfc6182>.

   [RFC6824]  Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
              "TCP Extensions for Multipath Operation with Multiple
              Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013,
              <http://www.rfc-editor.org/info/rfc6824>.

5.2.  Informative References

   [BALIA]    Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath
              TCP: analysis, design, and implementation", IEEE/ACM
              Trans. on Networking (TON), Volume 24, Issue 1, February
              2016.

   [CACM14]   Paasch, C. and O. Bonaventure, "Multipath TCP",
              Communications of the ACM, 57(4):51-57, April 2014,
              <http://inl.info.ucl.ac.be/publications/multipath-tcp>.

   [Cellnet12]
              Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O.
              Bonaventure, "Exploring Mobile/WiFi Handover with
              Multipath TCP", ACM SIGCOMM workshop on Cellular
              Networks (Cellnet12), August 2012,
              <http://inl.info.ucl.ac.be/publications/
              exploring-mobilewifi-handover-multipath-tcp>.

   [COMCOM2016]
              Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O.
              Bonaventure, "Observing real Multipath TCP traffic",
              Computer Communications, DOI 10.1016/j.comcom.2016.01.014,
              April 2016, <http://inl.info.ucl.ac.be/publications/
              observing-real-multipath-tcp-traffic>.

Top      Up      ToC       Page 24 
   [COMMAG2016]
              De Coninck, Q., Baerts, M., Hesmans, B., and O.
              Bonaventure, "Observing Real Smartphone Applications over
              Multipath TCP", IEEE Communications Magazine Network
              Testing Series, 54(3), March 2016,
              <http://inl.info.ucl.ac.be/publications/observing-real-
              smartphone-applications-over-multipath-tcp>.

   [CONEXT12] Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J.
              Leboudec, "MPTCP is not Pareto-Optimal: Performance Issues
              and a Possible Solution", CoNEXT '12: Proceedings of the
              8th international conference on Emerging networking
              experiments and technologies, DOI 10.1145/2413176.2413178,
              December 2012.

   [CONEXT13] Paasch, C., Khalili, R., and O. Bonaventure, "On the
              Benefits of Applying Experimental Design to Improve
              Multipath TCP", Conference on emerging Networking
              EXperiments and Technologies (CoNEXT),
              DOI 10.1145/2535372.2535403, December 2013,
              <http://inl.info.ucl.ac.be/publications/benefits-applying-
              experimental-design-improve-multipath-tcp>.

   [CONEXT15] Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O.
              Bonaventure, "SMAPP: Towards Smart Multipath TCP-enabled
              APPlications", Proc. Conext 2015, Heidelberg, Germany,
              December 2015, <http://inl.info.ucl.ac.be/publications/
              smapp-towards-smart-multipath-tcp-enabled-applications>.

   [CSWS14]   Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure,
              "Experimental evaluation of multipath TCP schedulers",
              CSWS '14: Proceedings of the 2014 ACM SIGCOMM workshop on
              Capacity sharing workshop, DOI 10.1145/2630088.2631977,
              August 2014.

   [DetalMSS] Detal, G., "dynamically adapt mss value", Post on the
              mptcp-dev mailing list, September 2014,
              <https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/
              2014-09/msg00130.html>.

   [FreeBSD-MPTCP]
              Williams, N., "Multipath TCP For FreeBSD Kernel Patch
              v0.5", <http://caia.swin.edu.au/urp/newtcp/mptcp>.

Top      Up      ToC       Page 25 
   [GRE-NOTIFY]
              Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
              Zhang, "GRE Notifications for Hybrid Access", Work in
              Progress, draft-lhwxz-gre-notifications-hybrid-access-01,
              January 2015.

   [HAMPEL]   Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility
              using lightweight MPTCP proxy", MobiWac '13: Proceedings
              of the 11th ACM international symposium on Mobility
              management and wireless access,
              DOI 10.1145/2508222.2508226, November 2013.

   [HotMiddlebox13]
              Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O.
              Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT
              workshop Hot Middlebox, December 2013,
              <http://inl.info.ucl.ac.be/publications/
              are-tcp-extensions-middlebox-proof>.

   [HotMiddlebox13b]
              Detal, G., Paasch, C., and O. Bonaventure, "Multipath in
              the Middle(Box)", HotMiddlebox '13, December 2013,
              <http://inl.info.ucl.ac.be/publications/
              multipath-middlebox>.

   [HotNets]  Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A.,
              Wischik, D., and M. Handley, "Data center networking with
              multipath TCP", Hotnetx-IX: Proceedings of the 9th ACM
              SIGCOMM Workshop on Hot Topics in Networks Article No. 10,
              DOI 10.1145/1868447.1868457, October 2010,
              <http://doi.acm.org/10.1145/1868447.1868457>.

   [HYA-ARCH] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M.
              Zhang, "Hybrid Access Network Architecture", Work in
              Progress, draft-lhwxz-hybrid-access-network-
              architecture-02, January 2015.

   [ICNP12]   Cao, Y., Xu, M., and X. Fu, "Delay-based congestion
              control for multipath TCP", 20th IEEE International
              Conference on Network Protocols (INCP),
              DOI 10.1109/ICNP.2012.6459978, October 2012.

   [IETF88]   Stewart, L., "IETF 88 Meeting minutes of the MPTCP working
              group", November 2013, <https://www.ietf.org/proceedings/
              88/minutes/minutes-88-mptcp>.

Top      Up      ToC       Page 26 
   [IETFJ]    Bonaventure, O. and S. Seo, "Multipath TCP Deployments",
              IETF Journal, Vol. 12, Issue 2, November 2016.

   [IMC11]    Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A.,
              Handley, M., and H. Tokuda, "Is it still possible to
              extend TCP?", IMC '11: Proceedings of the 2011 ACM SIGCOMM
              conference on Internet measurement conference,
              DOI 10.1145/2068816.2068834, November 2011,
              <http://doi.acm.org/10.1145/2068816.2068834>.

   [IMC13a]   Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and
              B. Donnet, "Revealing Middlebox Interference with
              Tracebox", Proceedings of the 2013 ACM SIGCOMM conference
              on Internet measurement conference,
              DOI 10.1145/2504730.2504757, October 2013,
              <http://inl.info.ucl.ac.be/publications/
              revealing-middlebox-interference-tracebox>.

   [IMC13b]   Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R.,
              and D. Towsley, "A measurement-based study of MultiPath
              TCP performance over wireless network", ICM '13:
              Proceedings of the 2013 conference on Internet
              measurement conference, DOI 10.1145/2504730.2504751,
              October 2013,
              <http://doi.acm.org/10.1145/2504730.2504751>.

   [IMC13c]   Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush,
              "From Paris to Tokyo: on the suitability of ping to
              measure latency", IMC '13: Proceedings of the 2013
              conference on Internet measurement Conference,
              DOI 10.1145/2504730.2504765, October 2013,
              <http://doi.acm.org/10.1145/2504730.2504765>.

   [INFOCOM14]
              Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee,
              "Cross-layer path management in multi-path transport
              protocol for mobile devices", IEEE INFOCOM'14,
              DOI 10.1109/INFOCOM.2014.6848120, April 2014.

   [KT]       Seo, S., "KT's GiGA LTE", July 2015,
              <https://www.ietf.org/proceedings/93/slides/
              slides-93-mptcp-3.pdf>.

   [MBTest]   Hesmans, B., "MBTest", October 2013,
              <https://bitbucket.org/bhesmans/mbtest>.

Top      Up      ToC       Page 27 
   [Mobicom15]
              De Coninck, Q., Baerts, M., Hesmans, B., and O.
              Bonaventure, "Poster - Evaluating Android Applications
              with Multipath TCP", Mobicom 2015 (Poster),
              DOI 10.1145/2789168.2795165, September 2015.

   [MPTCP-DEPLOY]
              Paasch, C., Biswas, A., and D. Haas, "Making Multipath TCP
              robust for stateless webservers", Work in Progress,
              draft-paasch-mptcp-syncookies-02, October 2015.

   [MPTCP-LOAD]
              Paasch, C., Greenway, G., and A. Ford, "Multipath TCP
              behind Layer-4 loadbalancers", Work in Progress,
              draft-paasch-mptcp-loadbalancer-00, September 2015.

   [MPTCP-MAX-SUB]
              Boucadair, M. and C. Jacquenet, "Negotiating the Maximum
              Number of Multipath TCP (MPTCP) Subflows", Work in
              Progress draft-boucadair-mptcp-max-subflow-02, May 2016.

   [MPTCPBIB] Bonaventure, O., "Multipath TCP - Annotated bibliography",
              Technical report, April 2015,
              <https://github.com/obonaventure/mptcp-bib>.

   [MultipathTCP-Linux]
              Paasch, C., Barre, S., and . et al, "Multipath TCP - Linux
              Kernel implementation", <http://www.multipath-tcp.org>.

   [NSDI11]   Wischik, D., Raiciu, C., Greenhalgh, A., and M. Handley,
              "Design, implementation and evaluation of congestion
              control for multipath TCP", NSDI11: In Proceedings of the
              8th USENIX conference on Networked systems design
              and implementation, 2011.

   [NSDI12]   Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M.,
              Duchene, F., Bonaventure, O., and M. Handley, "How Hard
              Can It Be? Designing and Implementing a Deployable
              Multipath TCP", NSDI '12: USENIX Symposium of Networked
              Systems Design and implementation, April 2012,
              <http://inl.info.ucl.ac.be/publications/how-hard-can-it-
              be-designing-and-implementing-deployable-multipath-tcp>.

   [PaaschPhD]
              Paasch, C., "Improving Multipath TCP", Ph.D. Thesis ,
              November 2014, <http://inl.info.ucl.ac.be/publications/
              improving-multipath-tcp>.

Top      Up      ToC       Page 28 
   [PAM2016]  De Coninck, Q., Baerts, M., Hesmans, B., and O.
              Bonaventure, "A First Analysis of Multipath TCP on
              Smartphones", 17th International Passive and Active
              Measurements Conference (PAM2016) volume 17, March 2016,
              <http://inl.info.ucl.ac.be/publications/
              first-analysis-multipath-tcp-smartphones>.

   [PAMS2014] Arzani, B., Gurney, A., Cheng, S., Guerin, R., and B. Loo,
              "Impact of Path Selection and Scheduling Policies on MPTCP
              Performance", PAMS2014, DOI 10.1109/WAINA.2014.121, May
              2014.

   [Presto08] Greenberg, A., Lahiri, P., Maltz, D., Patel, P., and S.
              Sengupta, "Towards a next generation data center
              architecture: scalability and commoditization", ACM
              PRESTO 2008, DOI 10.1145/1397718.1397732, August 2008,
              <http://dl.acm.org/citation.cfm?id=1397732>.

   [PT14]     Pearce, C. and P. Thomas, "Multipath TCP Breaking Today's
              Networks with Tomorrow's Protocols", Proc.
              Blackhat Briefings, 2014, <http://www.blackhat.com/docs/
              us-14/materials/us-14-Pearce-Multipath-TCP-Breaking-
              Todays-Networks-With-Tomorrows-Protocols-WP.pdf>.

   [PZ15]     Pearce, C. and S. Zeadally, "Ancillary Impacts of
              Multipath TCP on Current and Future Network Security",
              IEEE Internet Computing, vol. 19, no. 5, pp. 58-65,
              DOI 10.1109/MIC.2015.70, September 2015.

   [RFC1812]  Baker, F., Ed., "Requirements for IP Version 4 Routers",
              RFC 1812, DOI 10.17487/RFC1812, June 1995,
              <http://www.rfc-editor.org/info/rfc1812>.

   [RFC1928]  Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and
              L. Jones, "SOCKS Protocol Version 5", RFC 1928,
              DOI 10.17487/RFC1928, March 1996,
              <http://www.rfc-editor.org/info/rfc1928>.

   [RFC2992]  Hopps, C., "Analysis of an Equal-Cost Multi-Path
              Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000,
              <http://www.rfc-editor.org/info/rfc2992>.

   [RFC4987]  Eddy, W., "TCP SYN Flooding Attacks and Common
              Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007,
              <http://www.rfc-editor.org/info/rfc4987>.

Top      Up      ToC       Page 29 
   [RFC6356]  Raiciu, C., Handley, M., and D. Wischik, "Coupled
              Congestion Control for Multipath Transport Protocols",
              RFC 6356, DOI 10.17487/RFC6356, October 2011,
              <http://www.rfc-editor.org/info/rfc6356>.

   [RFC7430]  Bagnulo, M., Paasch, C., Gont, F., Bonaventure, O., and C.
              Raiciu, "Analysis of Residual Threats and Possible Fixes
              for Multipath TCP (MPTCP)", RFC 7430,
              DOI 10.17487/RFC7430, July 2015,
              <http://www.rfc-editor.org/info/rfc7430>.

   [RFC7871]  Contavalli, C., van der Gaast, W., Lawrence, D., and W.
              Kumari, "Client Subnet in DNS Queries", RFC 7871,
              DOI 10.17487/RFC7871, May 2016,
              <http://www.rfc-editor.org/info/rfc7871>.

   [SIGCOMM11]
              Raiciu, C., Barre, S., Pluntke, C., Greenhalgh, A.,
              Wischik, D., and M. Handley, "Improving datacenter
              performance and robustness with multipath TCP", SIGCOMM
              '11: Proceedings of the ACM SIGCOMM 2011 conference,
              DOI 10.1145/2018436.2018467, August 2011,
              <http://doi.acm.org/10.1145/2018436.2018467>.

   [SOCKET]   Hesmans, B. and O. Bonaventure, "An enhanced socket API
              for Multipath TCP", Proceedings of the 2016 Applied
              Networking Research Workshop, DOI 10.1145/2959424.2959433,
              July 2016, <http://doi.acm.org/10.1145/2959424.2959433>.

   [StrangeMbox]
              Bonaventure, O., "Multipath TCP through a strange
              middlebox", Blog post, January 2015,
              <http://blog.multipath-tcp.org/blog/html/2015/01/30/
              multipath_tcp_through_a_strange_middlebox.html>.

   [TMA2015]  Hesmans, B., Tran Viet, H., Sadre, R., and O. Bonaventure,
              "A First Look at Real Multipath TCP Traffic", Traffic
              Monitoring and Analysis, 2015,
              <http://inl.info.ucl.ac.be/publications/
              first-look-real-multipath-tcp-traffic>.

   [TR-348]   Broadband Forum, ., "TR 348 - Hybrid Access Broadband
              Network Architecture", Issue: 1, July 2016,
              <https://www.broadband-forum.org/technical/download/
              TR-348.pdf>.

Top      Up      ToC       Page 30 
   [tracebox] Detal, G. and O. Tilmans, "Tracebox: A Middlebox Detection
              Tool", 2013, <http://www.tracebox.org>.

Acknowledgements

   This work was partially supported by the FP7-Trilogy2 project.  We
   would like to thank all the implementers and users of the Multipath
   TCP implementation in the Linux kernel.  This document has benefited
   from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley,
   Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu,
   Spencer Dawkins, and Ben Campbell.

Authors' Addresses

   Olivier Bonaventure
   UCLouvain

   Email: Olivier.Bonaventure@uclouvain.be


   Christoph Paasch
   Apple, Inc.

   Email: cpaasch@apple.com


   Gregory Detal
   Tessares

   Email: Gregory.Detal@tessares.net