Guidelines for QUIC Multipath over Path Aware Networks

1.1. SCION

One example of a PAN is SCION [SCION-CP], [SCION-DP]. SCION is an inter-domain routing protocol that provides path metadata on the level of links and router interfaces. This metadata provides many opportunities for improving congestion control and other algorithms. It also avoids some problems that gave rise to current algorithms.¶

The SCION protocol makes detailed path information available to endpoints. Besides the 4-tuple of address/IP at each endpoint, the information includes a list of all traversed ASes and respective links between ASes, as well as metadata about ASes and links, including MTU, bandwidth, latency, AS internal hopcount, or geolocation information.¶

2.1. Terminology

We assume that the reader is familiar with the terminology used in [QUIC-TRANSPORT] and [QUIC-MP]. We also draw on the terminology of SCION ([SCION-CP] and [SCION-DP]). For ease of reference, we have included some definitions here, but refer the reader to the references above for complete specifications of the relevant terminiology:¶

Autonomous System (AS): An autonomous system is a network under a common administrative control. For example, the network of an Internet service provider or organization can constitute an AS.¶

Endpoint: An endpoint is the start or the end of a path, as defined in [PATH-VOCABULARY].¶

Inter-AS Link: A direct link between two external interfaces of two ASes.¶

Intra-AS Link: A direct link between twi internal interfaces of a single AS. A direct link may contains several internal hops.¶

Link: General term that refers to "inter-AS links" and "intra-AS links".¶

Network Path: Consists of a 4-tuple of address/IP at each endpoint, a list of all traversed ASes, and links inside and between ASes, including interface IDs on border routers of each AS.¶

Path ID: The Path ID is defined in [QUIC-MP] . On the network layer, it is defined by a 4-tuple of IP/port of the local and remote endpoints.¶

Path Metadata: Path metadata is additional data that is available to clients when they request a selection of paths to a destination. Path metadata is authenticated wrt to owner of each link, but otherwise not verified. Path metadata includes data about ASes and links, such as MTU, bandwidth, latency, AS internal hopcount, or geolocation information. Path metadata is updated infrequently, probably at most once per hour. Properties such as bandwidth and latency represent hardware properties rather than live traffic information.¶

3.1. Path Metadata

We assume that path metadata is reflects hardware properties rather than live traffic information (especially for bandwidth and latency). One should not expect path metadata is updated to be more than once every hour. Path metadata is disseminated together with paths, so its freshness depends on the path livetime, wgich can be several hours.¶

3.2. Path Selection

We assume that a PAN protocol allows selecting paths explicitly, based, for example, on path metadata. Paths can be added and removed from a connection. Paths may have a best-before data and may expire, after which they are invalid.¶

5.1. Disjointness

For FT, paths are only interesting if they are disjoint. For BW, paths should mostly be disjunct, but overlap is acceptable if the overlapping links have high BW available (see Section 11.1.2).¶

For LAT and EVA, path disjointness is less important.¶

TODO Discuss link level, router level and AS level path disjunctness.¶

5.2. Path Metadata

SCION paths have associated metadata with latency and bandwidth information. The values represent best case values and tend to be updated rarely, for example every few hours (depending on the implementation).¶

Path metadata may also be incomplete, ASes are not required to provide or regularly update the data.¶

Users of path metadata must keep in mind that the data is mostly not verifiable but depends on the diligence and trustworthiness of the link owners. However, once disseminated by a link owner, the path metadata is authenticated and cannot be changed by other parties.¶

Due to the inherent unreliability, users should implement checks to verify that a link holds up to the promised capabilities.¶

6.1. 4-tuple changes

If the 4-tuple (IP/port of local/remote endpoint) changes, [QUIC-MP] and [QUIC-TRANSPORT] require several actions, including resetting congestion control and RTT estimation algorithms, and initiating path validation.¶

Using path aware networks affects this in two ways:¶

1. PANs can provide detailed path information that can be used to detected path changes with high granularity. This can affect algorithm design because they need to be less resilient against undetected path changes (diffserv, BGP route changes, ...) and can rely more on the path being consistent, at least to the granualrity offered by the PAN layer.¶

2. PANs, such as SCION, extend the network address with path, AS code and ISD code. ASes may use private IP ranges with IPs that are globally not unique; This would allow a man-in-the-middle attacker to construct a scenarion where the IP/port of an attacked machine can be duplicated in a different AS. The attacker can then change the destination (i.e. routing to a different AS) of a connection without changing the destination port/IP. This could be used to avoid path validation when coaxing a machine to send traffic to a new destination. This attack is still not easy to execute because it requires the attacker to have control over an AS that lies en-route between server and client.¶

     AS #100               AS #200                   AS #300
   +------------+        +----------------+        +----------------+
   | Server     |        | Attacker       |        | Victim         |
   | IP=1.2.3.4 | ------ | IP=192.168.0.1 | ------ | IP=192.168.0.1 |
   | port = 42  |        | port= 12345    |        | port= 12345    |
   +------------+        +----------------+        +----------------+

    **TODO** use IPs from documentation range

   Attacker                                                Server
   (Establish connection)
   Path=[#200, #100] -->
                                                <-- Path=[#100, #200]

   (Change Path)
   Path=[#300, #200, #100] -->
                                          <-- Path=[#100, #200, #300]

   Client: receives unwanted traffic

7.1. Initialization

When a QUIC(-MP) socket is created, it can be useful (depending on the programming language) to allow injection of a custom modules:¶

• Custom UDP underlay (for injecting a PAN, such as SCION).¶

• Custom DNS resolver (if the library resolves URLs)¶

• Custom congestion control and RTT estimation algorithms¶

• Custom Packet scheduling algorithm¶

• PAN exclusive: path selection algorithms¶

Many available implementations already allow injecting most of these modules.¶

7.2. Network Address and Network Path

Generally, it can be useful if any API of a QUIC-MP library uses not only IP/port for addressing local/remote peers, but uses a network address (IP + port + other identifiers, such as AS number) and a network path.¶

Note that a "network path" is different from [QUIC-MP] path ID. There is usually a 1:1 mapping from network path to path ID, but the network path may change while the path ID stays the same.¶

For example: - During path migration (TODO ref), a path ID may be (loosely) associated with multiple network paths. - A PAN library may perform probing (latency, etc.) and switch to a new path that is deemed better. - A PAN library may have paths that can expire and that can be renewed automatically. However, this is should usually result in an identical path (except for the expiration date).¶

7.3. General API...?

The API of a QUIC-MP implementation that works with PAN should probably provide interfaces for¶

The API of a QUIC-MP implementation that works with PAN should:¶

• Expose Path ID¶

• Expose API for custom congestion control algorithms¶

• Expose callback for QUIC-MP path abandon (REFERENCE!!!)¶

• Expose API for name resolution (only of library does that), not useful if API works with IP/port only.¶

• Expose API for usage profile. Applications will have very different requirements on a multipath API, see Section 4. A comprehensive API should therefore allow for mostly automatic selection of Path Selection and Congestion Control algorithms.¶

8.1. Path Change Detection - Path Injection

9.1. General

[QUIC-TRANSPORT] requires that there is no connection migration during the initial handshake, and that there are no other packets send (including probing packets) during the initial handshake, see Section 9 of [QUIC-TRANSPORT], paragraphs 2 and 3.¶

We need to ensure on some level that no path change or probing occurs.¶

TODO Read [QUIC-MP] on 4-tuple change / path validation. -> Section 9 in RFC 9000¶

9.2. Algorithms

• MPU detection algorithm can be removed/replaced with metadata query See Path MTU Discovery in Section 14.3 of [QUIC-TRANSPORT] and Section 5.8 of [QUIC-MP].¶

• Congestion control algorithms:¶

  • Must reset on path change (how?).¶

  • Should benefit from knowledge about path overlaps.¶

  • Follow BCP 133, see Section 7.10 of [CC-ALGORITHMS].¶

• Roundtrip time estimator.¶

  • Must reset on path change (how?). See also from Section 5.1 of [QUIC-MP]: "If path validation process succeeds, the endpoints set the path's congestion controller and round-trip time estimator according to Section 9.4 of [QUIC-TRANSPORT]."¶

  • Should benefit from knowledge about minimum latency expected on a path.¶

• Path selections algorithms¶

• Latency polling¶

  • How bad is latency polling?¶

  • Traceroute can help to reduce polling (ideally, every links is traversed only by one poll packet). Traceroute also allows to identify links with high variance or generally hogh latency.¶

11.1. Path Selection

11.2. MTU

SCION provides MTU information for every AS-level link on a path.¶

11.3. DMTP

One example of an application / algorithm is discussed in [DMTP].¶

11.4. Congestion Control

General recommendations for congestion control are defined in Congestion Control Principles [CC-PRINCIPLES]. Congestion control for QUIC is discussed in QUIC Loss Detection and Congestion Control [QUIC-RECOVERY]. More generally, congestion control for UDP is discussed in the UDP Usage Guidelines [UDP-GUIDELINES]. UDP MTU discovery is further developed in [MTU-DISCOVERY].¶

Multipath congestion control is also discussed for TCP in [CC-MULTIPATH-TCP]. They state that "One of the prominent problems is that running existing algorithms such as standard TCP independently on each path would give the multipath flow more than its fair share at a bottleneck link traversed by more than one of its subflows.". This can be avoided in SCION through link-level analysis of flows and selecting paths that do not create or share a bottleneck link. This avoids the need for Coupled Congestion Control.¶

Note that SCION uses different paths from IP. While some of the links may carry both IP and SCION traffic, the SCION traffic typically has its own bandwidth allocation.¶

The second point in [CC-MULTIPATH-TCP] is "Further, it is desirable that a source with multiple paths available will transfer more traffic using the least congested of the paths, achieving a property called "resource pooling" where a bundle of links effectively behaves like one shared link with bigger capacity.". This can be achieved with a simple load distribution algorithm.¶

TODO Differentiate "load distribution" from "path selection".¶

There are several congestion control algorithms proposed in literature, e.g. LIA, OLIA, BALIA and RSF. These combine congestion control with path selection algorithms. For simplicity, we suggest separating concerns in terms of congestion control and path selection. This allows us to better tailor the solutions to the different scenarios discussed in (SCENARIOS). The proposition is to use standard congestion control per path (LIST HERE), tailored for each use case (max bandwidth / low latency) and on top of that separately use path selection (load distribution) algorithms.¶

TODO Terminology: path selection vs load distribution?¶

11.5. Load Distribution (Scheduling)

Load distribution algorithms are mainly useful for high bandwidth (HBW) scenarios. Latency may still be relevant though, for example for high definition video streams. Scheduling halps distributing the transfer load efficiently over multiple path.¶

However, sending data stream over multiple paths in parallel will usually result in packets arriving out of order at the receiver. This should be avoided because:¶

• Packet reordering requires larger buffers on the receiver side which are used to put packets back in order. Latency, jitter and drop rate on different paths directly affect the required buffer size.¶

• Head of line blocking (HOLB): the latency of the slowest packet determines the effective latency of the stream. This may be ignored for data transfers where latency is irrelevant.¶

These problems can be mitigated, but it is difficult to do so for both problems at the same time. A simple way to mitigate these problem is to select multiple paths with similar latency.¶

Another way to mitigate these problems is to schedule packets on different paths such that they are likely to arrive in (more or less) the correct order. Care should be taken to avoid requiring an equally large buffer on the sender side.¶

TODO Check which existing algorithms do that.¶

TODO Can we facilitate QUIC streams for this?¶

13.1. Data Transfer

The aim of a data transfer application is to maximize throughput, regardless of latency, jitter or packet loss.¶

The solution here is to identify multiple paths that are either disjoint, or where the non-disjoint links allow higher throughput than other links on the same paths (i.e. high enough to prevent the link from being a bottleneck).¶

13.2. Low Latency

There are multiple approaches to transfer traffic with the lowest possible latency. For example:¶

1) With separate latency measurements. Latency measurements run in parallel to data traffic. This allows performing measurements at a different frequency and over many more paths than payload traffic. This can be useful if payload packets are large and sending them redundantly over multiple links is to costly or considered too invasive with respect to other network users. Low frequency measurements may be combined with traffic prediction algorithms in order to identify best paths between measurements. Due to low bandwidth overhead, this may be the most cost efficient approach. It may be possible to do this with SCION SCMP packets, either with ECHO packets or possibly with traceroute if the AS internal latency of the destination AS is negligible (because it may be small or it may be almost the same for all border routers that are on interesting paths).¶

2) With implicit measurements through QUIC ACK frames. Section 13.2 of [QUIC-TRANSPORT] recommends sending an ACK frame after receiving at least two ack-eliciting packets or after a delay. If the application sends data on multiple paths in parallel, this may be sufficient for some low latency applications. [QUIC-ACKFREQUENCY] proposes an extension that allows more control over how often and when ACK-FRAMES are sent. This approach can be useful if [QUIC-ACKFREQUENCY] becomes available or if the ACK behavior of the standard QUIC server is sufficient.¶

3) With implicit measurements through application specific ACKs. This is useful if the application can sensibly be adapted to have its own ACK protocol.¶

Network analysis can be used to identify, and subsequently avoid, links with high or unreliable latency.¶

13.3. High Availability / Redundancy

An approach to high availability is to send data on multiple paths in parallel. A tradeoff here is that sending on all available paths may be infeasible because of the number of available paths (with SCION we often see 100+ paths to a destination). Depending on cost factors, and to avoid overloading the network, any algorithms should keep redundant sending to a minimum.¶

Path analysis can be used to identify multiple paths that are mostly or completely (using multiple interfaces) disjoint, but that still satisfy latency, bandwidth, and other constraints.¶

Additional polling with SCMP or with an additional QUIC stream may be used to regularly measure packet drop rates or latency variations.¶

13.4. Multipathing for Anonymity

Multipathing could also be used for anonymity, e.g., by switching paths at random intervals. With a continuous data stream, care should be taken that new paths are not just switched over from one to the next, otherwise traffic characteristics may be used to identify paths. For example, paths could be identified by packet frequency, packet burst frequency or general bandwidth. For continuous stream, just moving one stream from one path to another may expose stream identity.¶

As mitigation, a sender could start sending on a new path for a while before stopping sending on the old path. The sender could send redundant data or random data at a suitable bandwidth.¶

TODO Should we really recommend bandwidth usage.¶

TODO Is this SCION specific? SCION allows for choosing paths based on trusted or untrusted ASes, but this is not specific to multipathing...¶

14.1. Multipathing for Applications

Applications will have very different requirements on a multipath API. A comprehensive API should therefore allow for mostly automatic selection of Section 11.1 Path Selection and Congestion Control algorithms Section 11.4.¶

At the same time it should give access to SCION paths and their metadata to allow implementation of custom algorithms.¶

14.2. Algorithm Integration

Several proposals in Section 13.2 and Section 13.3 suggest sending data redundantly in parallel on multiple paths. Similarly, some proposals suggest sending packets purely for latency measurements.¶

Congestion control algorithms and path selection algorithms should try to hide parallel transfer and measurement streams from the application. This may also depend on API designer to provide such transparent multipathing with additional code on the application level.¶

15.1. Latency Polling

If a user sends latency measurements on 10 paths in parallel every 5 seconds, then these 10 paths can (with some probability) be attributed to the same user. Solution: do not send all probes at the same time. To prevent patterns (1. after 0.1s, 2. after 0.3s, 3. after 0.35s, ...) the interval between packets on each path should also vary. Additionally, the number of polled paths should vary.¶

15.2. Path Validation

Following [QUIC-MP] (Section 5.1) and [QUIC-TRANSPORT] (Section 9), endpoints MUST drop a connection or perform path validation when the 4-tuple changes:¶

"Not all changes of peer address are intentional, or active, migrations. The peer could experience NAT rebinding: a change of address due to a middlebox, usually a NAT, allocating a new outgoing port or even a new outgoing IP address for a flow. An endpoint MUST perform path validation (Section 8.2) if it detects any change to a peer's address, unless it has previously validated that address." ¶

With SCION, endpoints may use private IPs, say 192.168.0.1. These IPs are obviously not unique. Two paths with an identical 4-tuple may therefor connect to two different machines if the machines are in different ASes but use the same IP/port.¶

In short, an attacker can impersonate a client by using an identical IP/port to connect a server. The server would probably just reverse the path and answer to the client without triggering a path validation.¶

TODO What is the implication of this?¶

• Can an attacker, without having being able to encrypt/decrypt data, cause any harm? Can they redurect traffic to themselves? To prevent this, the server SCION stack should accept a new path only if the QUIC stack also accepts it! This means we need to trigger a path validation and somehow learn whether it worked. Or we use our own validation process. Or we don't allow the path to change and require a new QUIC connection (with a new SCION connection)? SOLUTION: The SCION layer MUST NOT cache paths locally, instead paths must be accepted by the QUIC layer before being used.¶

Examples: * Java over DatagramChannel: * Comparing ResponsePath objects MUST return false if the paths differ. * The QUIC layer MUST use only those addresses for sending data that have previously been accepted. If an attacker sends a packet, it should be identified as malicious, be rejected, and the path should not be used. Why would an attacker packet be accepted? Replay should be impossible (there are apcket sequence IDs?), and any other requests should be cryptographically protected. Can this attack be successful with spoofed IPs??? * Java over DatagramSocket: Problematic, it cahces the paths... * C + Rust: How exactly do we map PathID to paths? How are paths updated?¶

To restore the behavior intended by QUIC, we must therefore extend the 4-tuple to a 6-tuple of ISD-AS + IP + port on both endpoints.¶

TODO This does not require any change in the QUIC layer but can probably be handled in the SCION ayer (the QUIC layer does not know about the ISD/AS code).¶

It seems with SCION we can insert arbitrary IP addresses into the stack before handing incoming packets up to the QUIC layer...?¶

15.3. More ?

See Section 13.4.¶

See other attacks in Section 7.2.4 of [SCION-CP]?¶

TODO¶

17.1. Normative References

[CC-ALGORITHMS]

Duke, M., Ed. and G. Fairhurst, Ed., "Specifying New Congestion Control Algorithms", BCP 133, RFC 9743, DOI 10.17487/RFC9743, March 2025, <https://www.rfc-editor.org/rfc/rfc9743>.

[CC-PRINCIPLES]

Floyd, S., "Congestion Control Principles", BCP 41, RFC 2914, DOI 10.17487/RFC2914, September 2000, <https://www.rfc-editor.org/rfc/rfc2914>.

[DCCP-UDPENCAP]

Phelan, T., Fairhurst, G., and C. Perkins, "DCCP-UDP: A Datagram Congestion Control Protocol UDP Encapsulation for NAT Traversal", RFC 6773, DOI 10.17487/RFC6773, November 2012, <https://www.rfc-editor.org/rfc/rfc6773>.

[MPTCP-ARCHITECTURE]

Ford, A., Raiciu, C., Handley, M., Barre, S., and J. Iyengar, "Architectural Guidelines for Multipath TCP Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, <https://www.rfc-editor.org/rfc/rfc6182>.

[MTU-DISCOVERY]

Fairhurst, G., Jones, T., Tüxen, M., Rüngeler, I., and T. Völker, "Packetization Layer Path MTU Discovery for Datagram Transports", RFC 8899, DOI 10.17487/RFC8899, September 2020, <https://www.rfc-editor.org/rfc/rfc8899>.

[QUIC-RECOVERY]

Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection and Congestion Control", RFC 9002, DOI 10.17487/RFC9002, May 2021, <https://www.rfc-editor.org/rfc/rfc9002>.

[QUIC-TLS]

Thomson, M., Ed. and S. Turner, Ed., "Using TLS to Secure QUIC", RFC 9001, DOI 10.17487/RFC9001, May 2021, <https://www.rfc-editor.org/rfc/rfc9001>.

[QUIC-TRANSPORT]

Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based Multiplexed and Secure Transport", RFC 9000, DOI 10.17487/RFC9000, May 2021, <https://www.rfc-editor.org/rfc/rfc9000>.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

[UDP-GUIDELINES]

Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, March 2017, <https://www.rfc-editor.org/rfc/rfc8085>.

17.2. Informative References

[CC-MULTIPATH-TCP]

Raiciu, C., Handley, M., and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", RFC 6356, DOI 10.17487/RFC6356, October 2011, <https://www.rfc-editor.org/rfc/rfc6356>.

[DMTP]

John, T. and T. Riechard, "Deadline Aware Streams in QUIC Multipath", Work in Progress, Internet-Draft, draft-tjohn-quic-multipath-dmtp-01, 5 June 2025, <https://datatracker.ietf.org/doc/html/draft-tjohn-quic-multipath-dmtp-01>.

[OLIA]

Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. Le Boudec, "MPTCP is not pareto-optimal: performance issues and a possible solution", Proceedings of the 8th international conference on Emerging networking experiments and technologies, ACM , 2012.

[PATH-VOCABULARY]

Enghardt, R. and C. Krähenbühl, "A Vocabulary of Path Properties", RFC 9473, DOI 10.17487/RFC9473, September 2023, <https://www.rfc-editor.org/rfc/rfc9473>.

[QUIC-ACKFREQUENCY]

Iyengar, J., Swett, I., and M. Kühlewind, "QUIC Acknowledgment Frequency", Work in Progress, Internet-Draft, draft-ietf-quic-ack-frequency-11, 28 February 2025, <https://datatracker.ietf.org/doc/html/draft-ietf-quic-ack-frequency-11>.

[QUIC-MP]

Liu, Y., Ma, Y., De Coninck, Q., Bonaventure, O., Huitema, C., and M. Kühlewind, "Multipath Extension for QUIC", Work in Progress, Internet-Draft, draft-ietf-quic-multipath-14, 23 April 2025, <https://datatracker.ietf.org/doc/html/draft-ietf-quic-multipath-14>.

[SCION-CP]

de Kater, C., Rustignoli, N., and S. Hitz, "SCION Control Plane", Work in Progress, Internet-Draft, draft-dekater-scion-controlplane-08, 3 March 2025, <https://datatracker.ietf.org/doc/html/draft-dekater-scion-controlplane-08>.

[SCION-DP]

de Kater, C., Rustignoli, N., Hugly, J., and S. Hitz, "SCION Data Plane", Work in Progress, Internet-Draft, draft-dekater-scion-dataplane-05, 3 March 2025, <https://datatracker.ietf.org/doc/html/draft-dekater-scion-dataplane-05>.

[UMCC]

Gartner, M. and D. Hausheer, "UMCC: Uncoupling Multipath Congestion Control through Shared Bottleneck Detection in Path-Aware Networks", Proceedings of the IEEE 49th Conference on Local Computer Networks (LCN) , 2024.