Independent Submission L. Song, Ed. Request for Comments: 8483 D. Liu Category: Informational Beijing Internet Institute ISSN: 2070-1721 P. Vixie TISF A. Kato Keio/WIDE S. Kerr October 2018 Yeti DNS Testbed
AbstractYeti DNS is an experimental, non-production root server testbed that provides an environment where technical and operational experiments can safely be performed without risk to production root server infrastructure. This document aims solely to document the technical and operational experience of deploying a system that is similar to but different from the Root Server system (on which the Internet's Domain Name System is designed and built). Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This is a contribution to the RFC Series, independently of any other RFC stream. The RFC Editor has chosen to publish this document at its discretion and makes no statement about its value for implementation or deployment. Documents approved for publication by the RFC Editor are not candidates for any level of Internet Standard; see Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8483.
Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Requirements Notation and Conventions . . . . . . . . . . . . 5 3. Areas of Study . . . . . . . . . . . . . . . . . . . . . . . 5 3.1. Implementation of a Testbed like the Root Server System . 5 3.2. Yeti-Root Zone Distribution . . . . . . . . . . . . . . . 5 3.3. Yeti-Root Server Names and Addressing . . . . . . . . . . 5 3.4. IPv6-Only Yeti-Root Servers . . . . . . . . . . . . . . . 6 3.5. DNSSEC in the Yeti-Root Zone . . . . . . . . . . . . . . 6 4. Yeti DNS Testbed Infrastructure . . . . . . . . . . . . . . . 7 4.1. Root Zone Retrieval . . . . . . . . . . . . . . . . . . . 8 4.2. Transformation of Root Zone to Yeti-Root Zone . . . . . . 9 4.2.1. ZSK and KSK Key Sets Shared between DMs . . . . . . . 10 4.2.2. Unique ZSK per DM; No Shared KSK . . . . . . . . . . 10 4.2.3. Preserving Root Zone NSEC Chain and ZSK RRSIGs . . . 11 4.3. Yeti-Root Zone Distribution . . . . . . . . . . . . . . . 12 4.4. Synchronization of Service Metadata . . . . . . . . . . . 12 4.5. Yeti-Root Server Naming Scheme . . . . . . . . . . . . . 13 4.6. Yeti-Root Servers . . . . . . . . . . . . . . . . . . . . 14 4.7. Experimental Traffic . . . . . . . . . . . . . . . . . . 16 4.8. Traffic Capture and Analysis . . . . . . . . . . . . . . 16 5. Operational Experience with the Yeti DNS Testbed . . . . . . 17 5.1. Viability of IPv6-Only Operation . . . . . . . . . . . . 17 5.1.1. IPv6 Fragmentation . . . . . . . . . . . . . . . . . 18 5.1.2. Serving IPv4-Only End-Users . . . . . . . . . . . . . 19 5.2. Zone Distribution . . . . . . . . . . . . . . . . . . . . 19 5.2.1. Zone Transfers . . . . . . . . . . . . . . . . . . . 19 5.2.2. Delays in Yeti-Root Zone Distribution . . . . . . . . 20 5.2.3. Mixed RRSIGs from Different DM ZSKs . . . . . . . . . 21 5.3. DNSSEC KSK Rollover . . . . . . . . . . . . . . . . . . . 22 5.3.1. Failure-Case KSK Rollover . . . . . . . . . . . . . . 22 5.3.2. KSK Rollover vs. BIND9 Views . . . . . . . . . . . . 22 5.3.3. Large Responses during KSK Rollover . . . . . . . . . 23 5.4. Capture of Large DNS Response . . . . . . . . . . . . . . 24 5.5. Automated Maintenance of the Hints File . . . . . . . . . 24
5.6. Root Label Compression in Knot DNS Server . . . . . . . . 25 6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 26 7. Security Considerations . . . . . . . . . . . . . . . . . . . 28 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 9.1. Normative References . . . . . . . . . . . . . . . . . . 29 9.2. Informative References . . . . . . . . . . . . . . . . . 29 Appendix A. Yeti-Root Hints File . . . . . . . . . . . . . . . . 33 Appendix B. Yeti-Root Server Priming Response . . . . . . . . . 34 Appendix C. Active IPv6 Prefixes in Yeti DNS Testbed . . . . . . 36 Appendix D. Tools Developed for Yeti DNS Testbed . . . . . . . . 36 Appendix E. Controversy . . . . . . . . . . . . . . . . . . . . 37 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 38 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 RFC1034] and [RFC1035], has proved to be an enduring and important platform upon which almost every end-user of the Internet relies. Despite its longevity, extensions to the protocol, new implementations, and refinements to DNS operations continue to emerge both inside and outside the IETF. The Root Server system in particular has seen technical innovation and development, for example, in the form of wide-scale anycast deployment, the mitigation of unwanted traffic on a global scale, the widespread deployment of Response Rate Limiting [RRL], the introduction of IPv6 transport, the deployment of DNSSEC, changes in DNSSEC key sizes, and preparations to roll the root zone's Key Signing Key (KSK) and corresponding trust anchor. These projects created tremendous qualitative operational change and required impressive caution and study prior to implementation. They took place in parallel with the quantitative expansion or delegations for new TLDs (see <https://newgtlds.icann.org/>). Aspects of the operational structure of the Root Server system have been described in such documents as [TNO2009], [ISC-TN-2003-1], [RSSAC001], and [RFC7720]. Such references, considered together, provide sufficient insight into the operations of the system as a whole that it is straightforward to imagine structural changes to the Root Server system's infrastructure and to wonder what the operational implications of such changes might be. The Yeti DNS Project was conceived in May 2015 with the aim of providing a non-production testbed that would be open for use by anyone from the technical community to propose or run experiments designed to answer these kinds of questions. Coordination for the
project was provided by BII, TISF, and the WIDE Project. Thus, Yeti DNS is an independently coordinated project and is not affiliated with the IETF, ICANN, IANA, or any Root Server Operator. The objectives of the Yeti Project were set by the participants in the project based on experiments that they considered would provide valuable information. Many volunteers collaborated to build a distributed testbed that at the time of writing includes 25 Yeti root servers with 16 operators and handles experimental traffic from individual volunteers, universities, DNS vendors, and distributed measurement networks. By design, the Yeti testbed system serves the root zone published by the IANA with only those structural modifications necessary to ensure that it is able to function usefully in the Yeti testbed system instead of the production Root Server system. In particular, no delegation for any top-level zone is changed, added, or removed from the IANA-published root zone to construct the root zone served by the Yeti testbed system, and changes in the root zone are reflected in the testbed in near real-time. In this document, for clarity, we refer to the zone derived from the IANA-published root zone as the Yeti-Root zone. The Yeti DNS testbed serves a similar function to the Root Server system in the sense that they both serve similar zones: the Yeti-Root zone and the IANA-published root zone. However, the Yeti DNS testbed only serves clients that are explicitly configured to participate in the experiment, whereas the Root Server system serves the whole Internet. Since the dependent end-users and systems of the Yeti DNS testbed are known and their operations well-coordinated with those of the Yeti project, it has been possible to deploy structural changes in the Yeti DNS testbed with effective measurement and analysis, something that is difficult or simply impractical in the production Root Server system. This document describes the motivation for the Yeti project, describes the Yeti testbed infrastructure, and provides the technical and operational experiences of some users of the Yeti testbed. This document neither addresses the relevant policies under which the Root Server system is operated nor makes any proposal for changing any aspect of its implementation or operation.
Section 1, the Yeti DNS is an independently coordinated project and is not affiliated with the IETF, ICANN, IANA, or any Root Server Operator. Thus, the topics and areas for study were selected by (and for) the proponents of the Yeti project to address their own concerns and in the hope that the information and tools provided would be of wider interest. Each example included below is illustrated with indicative questions.
o What automatic mechanisms might be useful to improve the rate at which clients of Yeti-Root servers are able to react to a Yeti- Root server renumbering event? RFC5011] ("Automated Updates of DNS Security (DNSSEC) Trust Anchors") supported by resolvers? o Does the KSK Rollover plan designed and in the process of being implemented by ICANN work as expected on the Yeti testbed? o What is the operational impact of using much larger RSA key sizes in the ZSKs used in a root? o What are the operational consequences of choosing DNSSEC algorithms other than RSA to sign a root?
Figure 1. ,----------. ,-----------. ,------------. | stub +------> | recursive +------> | Yeti-Root | | resolver | <------+ resolver | <------+ nameserver | `----------' `-----------' `------------' ^ ^ ^ | appropriate | Yeti-Root hints; | Yeti-Root zone `- resolver `- Yeti-Root trust `- with DNSKEY RRset configured anchor signed by Yeti-Root KSK Figure 1: High-Level Testbed Components To use the Yeti DNS testbed, a recursive resolver must be configured to use the Yeti-Root servers. That configuration consists of a list of names and addresses for the Yeti-Root servers (often referred to as a "hints file") that replaces the corresponding hints used for the production Root Server system (Appendix A). If resolvers are configured to validate DNSSEC, then they also need to be configured with a DNSSEC trust anchor that corresponds to a KSK used in the Yeti DNS Project, in place of the normal trust anchor set used for the Root Zone. Since the Yeti root(s) are signed with Yeti keys, rather than those used by the IANA Root, corresponding changes are needed in the resolver trust anchors. Corresponding changes are required in the Yeti-Root hints file Appendix A. Those changes would be properly rejected as bogus by any validator using the production Root Server system's root zone trust anchor set. Stub resolvers become part of the Yeti DNS testbed by their use of recursive resolvers that are configured as described above. The data flow from IANA to stub resolvers through the Yeti testbed is illustrated in Figure 2 and is described in more detail in the sections that follow.
,----------------. ,-- / IANA Root Zone / ---. | `----------------' | | | | | | | Root Zone ,--------------. ,---V---. ,---V---. ,---V---. | Yeti Traffic | | BII | | WIDE | | TISF | | Collection | | DM | | DM | | DM | `----+----+----' `---+---' `---+---' `---+---' | | ,-----' ,-------' `----. | | | | | Yeti-Root ^ ^ | | | Zone | | ,---V---. ,---V---. ,---V---. | `---+ Yeti | | Yeti | . . . . . . . | Yeti | | | Root | | Root | | Root | | `---+---' `---+---' `---+---' | | | | DNS | | | | Response | ,--V----------V-------------------------V--. `---------+ Yeti Resolvers | `--------------------+---------------------' | DNS | Response ,--------------------V---------------------. | Yeti Stub Resolvers | `------------------------------------------' The three coordinators of the Yeti DNS testbed: BII : Beijing Internet Institute WIDE: Widely Integrated Distributed Environment Project TISF: A collaborative engineering and security project by Paul Vixie Figure 2: Testbed Data Flow Note that the roots are not bound to Distribution Masters (DMs). DMs update their zone on a schedule described in Section 4.1. Each DM that updates the latest zone can notify all roots, so the zone transfer can happen between any DM and any root.
Since Yeti DNS DMs do not receive DNS NOTIFY [RFC1996] messages from the Root Server system, a polling approach is used to determine when new revisions of the root zone are available from the production Root Server system. Each Yeti DM requests the Root Zone SOA record from a Root server that permits unauthenticated zone transfers of the root zone, and performs a zone transfer from that server if the retrieved value of SOA.SERIAL is greater than that of the last retrieved zone. At the time of writing, unauthenticated zone transfers of the Root Zone are available directly from B-Root, C-Root, F-Root, G-Root, K-Root, and L-Root; two servers XFR.CJR.DNS.ICANN.ORG and XFR.LAX.DNS.ICANN.ORG; and via FTP from sites maintained by the Root Zone Maintainer and the IANA Functions Operator. The Yeti DNS testbed retrieves the Root Zone using zone transfers from F-Root. The schedule on which F-Root is polled by each Yeti DM is as follows: +-------------+-----------------------+ | DM Operator | Time | +-------------+-----------------------+ | BII | UTC hour + 00 minutes | | WIDE | UTC hour + 20 minutes | | TISF | UTC hour + 40 minutes | +-------------+-----------------------+ The Yeti DNS testbed uses multiple DMs, each of which acts autonomously and equivalently to its siblings. Any single DM can act to distribute new revisions of the Yeti-Root zone and is also responsible for signing the RRsets that are changed as part of the transformation of the Root Zone into the Yeti-Root zone described in Section 4.2. This multiple DM model intends to provide a basic structure to implement the idea of shared zone control as proposed in [ITI2014]. Sections 4.2.1 and 4.2.2. A third approach has also been proposed, but not yet implemented. The motivations and changes implied by that approach are described in Section 4.2.3.
Section 4.4). 1. SOA.MNAME is set to www.yeti-dns.org. 2. SOA.RNAME is set to <dm-operator>.yeti-dns.org, where <dm-operator> is currently one of "wide", "bii", or "tisf". 3. All DNSKEY, RRSIG, and NSEC records are removed. 4. The apex Name Server (NS) RRset is removed, with the corresponding root server glue (A and AAAA) RRsets. 5. A Yeti DNSKEY RRset is added to the apex, comprising the public parts of all Yeti KSK and ZSKs. 6. A Yeti NS RRset is added to the apex that includes all Yeti-Root servers. 7. Glue records (AAAA only, since Yeti-Root servers are v6-only) for all Yeti-Root servers are added. 8. The Yeti-Root zone is signed: the NSEC chain is regenerated; the Yeti KSK is used to sign the DNSKEY RRset; and the shared ZSK is used to sign every other RRset. Note that the SOA.SERIAL value published in the Yeti-Root zone is identical to that found in the root zone.
part of its own dedicated ZSK key pairs to be available locally, and no other secret key material is shared. The high-level approach is illustrated in Figure 3. ,----------. ,-----------. .--------> BII ZSK +---------> Yeti-Root | | signs `----------' signs `-----------' | ,-----------. | ,----------. ,-----------. | Yeti KSK +-+--------> TISF ZSK +---------> Yeti-Root | `-----------' | signs `----------' signs `-----------' | | ,----------. ,-----------. `--------> WIDE ZSK +---------> Yeti-Root | signs `----------' signs `-----------' Figure 3: Unique ZSK per DM The process of retrieving the Root Zone from the Root Server system and replacing and signing the apex DNSKEY RRset no longer takes place on the DMs; instead, it takes place on a central Hidden Master. The production of signed DNSKEY RRsets is analogous to the use of Signed Key Responses (SKRs) produced during ICANN KSK key ceremonies [ICANN2010]. Each DM now retrieves source data (with a premodified and Yeti-signed DNSKEY RRset, but otherwise unchanged) from the Yeti DNS Hidden Master instead of from the Root Server system. Each DM carries out a similar transformation to that described in Section 4.2.1, except that DMs no longer need to modify or sign the DNSKEY RRset, and the DM's unique local ZSK is used to sign every other RRset. Section 4.2.2 has been proposed as a Yeti experiment called PINZ [PINZ], which would preserve the NSEC chain from the Root Zone and all RRSIG RRs generated using the Root Zone's ZSKs. The DNSKEY RRset would continue to be modified to replace the Root Zone KSKs, but Root Zone ZSKs would be kept intact, and the Yeti KSK would be used to generate replacement signatures over the apex DNSKEY and NS RRsets. Source data would continue to flow from the Root Server system through the Hidden Master to the set of DMs, but no DNSSEC operations would be required on the DMs, and the source NSEC and most RRSIGs would remain intact.
This approach has been suggested in order to keep minimal changes from the IANA Root zone and provide cryptographically verifiable confidence that no owner name in the root zone had been changed in the process of producing the Yeti-Root zone from the Root Zone, thereby addressing one of the concerns described in Appendix E in a way that can be verified automatically. RFC1996] messages to. This also forms the basis for an address-based access-control list for zone transfers. Authentication by address could be replaced with more rigorous mechanisms (e.g., using Transaction Signatures (TSIGs) [RFC2845]). This has not been done at the time of writing since the use of address-based controls avoids the need for the distribution of shared secrets amongst the Yeti-Root server operators. Individual Yeti-Root servers are configured with a full set of Yeti DM addresses to which SOA and AXFR queries may be sent in the conventional manner.
The three Git repositories are synchronized by configuring them as remote servers. For example, at BII we push to all three DMs' repositories as follows: $ git remote -v origin email@example.com:dm (fetch) origin firstname.lastname@example.org:dm (push) origin email@example.com:dm (push) origin firstname.lastname@example.org:dm (push) For more detailed information on DM synchronization, please see this document in Yeti's GitHub repository: <https://github.com/BII-Lab/ Yeti-Project/blob/master/doc/Yeti-DM-Sync.md>. RSSAC023]. The principal benefit of this naming scheme was that DNS label compression could be used to produce a priming response that would fit within 512 bytes at the time it was introduced, where 512 bytes is the maximum DNS message size using UDP transport without EDNS(0) [RFC6891]. Yeti-Root servers do not use this optimization, but rather use free- form nameserver names chosen by their respective operators -- in other words, no attempt is made to minimize the size of the priming response through the use of label compression. This approach aims to challenge the need to minimize the priming response in a modern DNS ecosystem where EDNS(0) is prevalent. Priming responses from Yeti-Root servers (unlike those from Root Servers) do not always include server addresses in the additional section. In particular, Yeti-Root servers running BIND9 return an empty additional section if the configuration parameter "minimum- responses" is set, forcing resolvers to complete the priming process with a set of conventional recursive lookups in order to resolve addresses for each Yeti-Root server. The Yeti-Root servers running NSD were observed to return a fully populated additional section (depending, of course, on the EDNS buffer size in use). Various approaches to normalize the composition of the priming response were considered, including: o Require use of DNS implementations that exhibit a desired behavior in the priming response.
o Modify nameserver software or configuration as used by Yeti-Root servers. o Isolate the names of Yeti-Root servers in one or more zones that could be slaved on each Yeti-Root server, renaming servers as necessary, giving each a source of authoritative data with which the authority section of a priming response could be fully populated. This is the approach used in the Root Server system with the ROOT-SERVERS.NET zone. The potential mitigation of renaming all Yeti-Root servers using a scheme that would allow their names to exist directly in the root zone was not considered because that approach implies the invention of new top-level labels not present in the Root Zone. Given the relative infrequency of priming queries by individual resolvers and the additional complexity or other compromises implied by each of those mitigations, the decision was made to make no effort to ensure that the composition of priming responses was identical across servers. Even the empty additional sections generated by Yeti-Root servers running BIND9 seem to be sufficient for all resolver software tested; resolvers simply perform a new recursive lookup for each authoritative server name they need to resolve. RFC5890] (it is shown in the following list in punycode).
+-------------------------------------+---------------+-------------+ | Name | Operator | Location | +-------------------------------------+---------------+-------------+ | bii.dns-lab.net | BII | CHINA | | yeti-ns.tsif.net | TSIF | USA | | yeti-ns.wide.ad.jp | WIDE Project | Japan | | yeti-ns.as59715.net | as59715 | Italy | | dahu1.yeti.eu.org | Dahu Group | France | | ns-yeti.bondis.org | Bond Internet | Spain | | | Systems | | | yeti-ns.ix.ru | Russia | MSK-IX | | yeti.bofh.priv.at | CERT Austria | Austria | | yeti.ipv6.ernet.in | ERNET India | India | | yeti-dns01.dnsworkshop.org | dnsworkshop | Germany | | | /informnis | | | dahu2.yeti.eu.org | Dahu Group | France | | yeti.aquaray.com | Aqua Ray SAS | France | | yeti-ns.switch.ch | SWITCH | Switzerland | | yeti-ns.lab.nic.cl | NIC Chile | Chile | | yeti-ns1.dns-lab.net | BII | China | | yeti-ns2.dns-lab.net | BII | China | | yeti-ns3.dns-lab.net | BII | China | | ca...a23dc.yeti-dns.net | Yeti-ZA | South | | | | Africa | | 3f...374cd.yeti-dns.net | Yeti-AU | Australia | | yeti1.ipv6.ernet.in | ERNET India | India | | xn--r2bi1c.xn--h2bv6c0a.xn--h2brj9c | ERNET India | India | | yeti-dns02.dnsworkshop.org | dnsworkshop | USA | | | /informnis | | | yeti.mind-dns.nl | Monshouwer | Netherlands | | | Internet | | | | Diensten | | | yeti-ns.datev.net | DATEV | Germany | | yeti.jhcloos.net. | jhcloos | USA | +-------------------------------------+---------------+-------------+ The current list of Yeti-Root servers is made available to a participating resolver first using a substitute hints file Appendix A and subsequently by the usual resolver priming process [RFC8109]. All Yeti-Root servers are IPv6-only, because of the IPv6-only Internet of the foreseeable future, and hence the Yeti-Root hints file contains no IPv4 addresses and the Yeti-Root zone contains no IPv4 glue records. Note that the rationale of an IPv6-only testbed is to test whether an IPv6-only root can survive any problem or impact when IPv4 is turned off, much like the context of the IETF SUNSET4 WG [SUNSET4].
At the time of writing, all root servers within the Root Server system serve the ROOT-SERVERS.NET zone in addition to the root zone, and all but one also serve the ARPA zone. Yeti-Root servers serve the Yeti-Root zone only. Significant software diversity exists across the set of Yeti-Root servers, as reported by their volunteer operators at the time of writing: o Platform: 18 of 25 Yeti-Root servers are implemented on a Virtual Private Server (VPS) rather than bare metal. o Operating System: 15 Yeti-Root servers run on Linux (Ubuntu, Debian, CentOS, Red Hat, and ArchLinux); 4 run on FreeBSD; 1 on NetBSD; and 1 on Windows Server 2016. o DNS software: 16 of 25 Yeti-Root servers use BIND9 (versions varying between 9.9.7 and 9.10.3); 4 use NSD (4.10 and 4.15); 2 use Knot (2.0.1 and 2.1.0); 1 uses Bundy (1.2.0); 1 uses PowerDNS (4.1.3); and 1 uses MS DNS (10.0.14300.1000). Section 4, are a source of real-world, end- user traffic. Due to an efficient cache mechanism, the mean query rate is less than 100 qps in the Yeti testbed, but a variety of sources were observed as active during 2017, as summarized in Appendix C. Synthetic traffic has been introduced to the system from time to time in order to increase traffic loads. Approaches include the use of distributed measurement platforms such as RIPE ATLAS to send DNS queries to Yeti-Root servers and the capture of traffic (sent from non-Yeti resolvers to the Root Server system) that was subsequently modified and replayed towards Yeti-Root servers.
Traffic capture is performed on Yeti-Root servers using either o dnscap <https://www.dns-oarc.net/tools/dnscap> or o pcapdump, part of the pcaputils Debian package <https://packages.debian.org/sid/pcaputils>, with a patch to facilitate triggered file upload (see <https://bugs.debian.org/ cgi-bin/bugreport.cgi?bug=545985>). PCAP-format files containing packet captures are uploaded using rsync to central storage.