9. Augmented BNF Syntax for NNTP 9.1. Introduction Each of the following sections describes the syntax of a major element of NNTP. This syntax extends and refines the descriptions elsewhere in this specification and should be given precedence when resolving apparent conflicts. Note that ABNF [RFC4234] strings are case insensitive. Non-terminals used in several places are defined in a separate section at the end.
Between them, the non-terminals <command-line>, <command-datastream>, <command-continuation>, and <response> specify the text that flows between client and server. A consistent naming scheme is used in this document for the non-terminals relating to each command, and SHOULD be used by the specification of registered extensions. For each command, the sequence is as follows: o The client sends an instance of <command-line>; the syntax for the EXAMPLE command is <example-command>. o If the client is one that immediately streams data, it sends an instance of <command-datastream>; the syntax for the EXAMPLE command is <example-datastream>. o The server sends an instance of <response>. * The initial response line is independent of the command that generated it; if the 000 response has arguments, the syntax of the initial line is <response-000-content>. * If the response is multi-line, the initial line is followed by a <multi-line-data-block>. The syntax for the contents of this block after "dot-stuffing" has been removed is (for the 000 response to the EXAMPLE command) <example-000-ml-content> and is an instance of <multi-line-response-content>. o While the latest response is one that indicates more data is required (in general, a 3xx response): * the client sends an instance of <command-continuation>; the syntax for the EXAMPLE continuation following a 333 response is <example-333-continuation>; * the server sends another instance of <response>, as above. (There are no commands in this specification that immediately stream data, but this non-terminal is defined for the convenience of extensions.)
9.2. Commands This syntax defines the non-terminal <command-line>, which represents what is sent from the client to the server (see section 3.1 for limits on lengths). command-line = command EOL command = X-command X-command = keyword *(WS token) command =/ article-command / body-command / capabilities-command / date-command / group-command / hdr-command / head-command / help-command / ihave-command / last-command / list-command / listgroup-command / mode-reader-command / newgroups-command / newnews-command / next-command / over-command / post-command / quit-command / stat-command article-command = "ARTICLE" [WS article-ref] body-command = "BODY" [WS article-ref] capabilities-command = "CAPABILITIES" [WS keyword] date-command = "DATE" group-command = "GROUP" [WS newsgroup-name] hdr-command = "HDR" WS header-meta-name [WS range-ref] head-command = "HEAD" [WS article-ref] help-command = "HELP" ihave-command = "IHAVE" WS message-id last-command = "LAST" list-command = "LIST" [WS list-arguments] listgroup-command = "LISTGROUP" [WS newsgroup-name [WS range]] mode-reader-command = "MODE" WS "READER" newgroups-command = "NEWGROUPS" WS date-time newnews-command = "NEWNEWS" WS wildmat WS date-time next-command = "NEXT" over-command = "OVER" [WS range-ref]
post-command = "POST" quit-command = "QUIT" stat-command = "STAT" [WS article-ref] article-ref = article-number / message-id date = date2y / date4y date4y = 4DIGIT 2DIGIT 2DIGIT date2y = 2DIGIT 2DIGIT 2DIGIT date-time = date WS time [WS "GMT"] header-meta-name = header-name / metadata-name list-arguments = keyword [WS token] metadata-name = ":" 1*A-NOTCOLON range = article-number ["-" [article-number]] range-ref = range / message-id time = 2DIGIT 2DIGIT 2DIGIT 9.3. Command Continuation This syntax defines the further material sent by the client in the case of multi-stage commands and those that stream data. command-datastream = UNDEFINED ; not used, provided as a hook for extensions command-continuation = ihave-335-continuation / post-340-continuation ihave-335-continuation = encoded-article post-340-continuation = encoded-article encoded-article = multi-line-data-block ; after undoing the "dot-stuffing", this MUST match <article> 9.4. Responses 9.4.1. Generic Responses This syntax defines the non-terminal <response>, which represents the generic form of responses; that is, what is sent from the server to the client in response to a <command> or a <command-continuation>. response = simple-response / multi-line-response simple-response = initial-response-line multi-line-response = initial-response-line multi-line-data-block initial-response-line = initial-response-content [SP trailing-comment] CRLF initial-response-content = X-initial-response-content X-initial-response-content = 3DIGIT *(SP response-argument)
response-argument = 1*A-CHAR trailing-comment = *U-CHAR 9.4.2. Initial Response Line Contents This syntax defines the specific initial response lines for the various commands in this specification (see section 3.1 for limits on lengths). Only those response codes with arguments are listed. initial-response-content =/ response-111-content / response-211-content / response-220-content / response-221-content / response-222-content / response-223-content / response-401-content response-111-content = "111" SP date4y time response-211-content = "211" 3(SP article-number) SP newsgroup-name response-220-content = "220" SP article-number SP message-id response-221-content = "221" SP article-number SP message-id response-222-content = "222" SP article-number SP message-id response-223-content = "223" SP article-number SP message-id response-401-content = "401" SP capability-label 9.4.3. Multi-line Response Contents This syntax defines the content of the various multi-line responses; more precisely, it defines the part of the response in the multi-line data block after any "dot-stuffing" has been undone. The numeric portion of each non-terminal name indicates the response code that is followed by this data. multi-line-response-content = article-220-ml-content / body-222-ml-content / capabilities-101-ml-content / hdr-225-ml-content / head-221-ml-content / help-100-ml-content / list-215-ml-content / listgroup-211-ml-content / newgroups-231-ml-content / newnews-230-ml-content / over-224-ml-content article-220-ml-content = article body-222-ml-content = body capabilities-101-ml-content = version-line CRLF
*(capability-line CRLF) hdr-225-ml-content = *(article-number SP hdr-content CRLF) head-221-ml-content = 1*header help-100-ml-content = *(*U-CHAR CRLF) list-215-ml-content = list-content listgroup-211-ml-content = *(article-number CRLF) newgroups-231-ml-content = active-groups-list newnews-230-ml-content = *(message-id CRLF) over-224-ml-content = *(article-number over-content CRLF) active-groups-list = *(newsgroup-name SPA article-number SPA article-number SPA newsgroup-status CRLF) hdr-content = *S-NONTAB hdr-n-content = [(header-name ":" / metadata-name) SP hdr-content] list-content = body newsgroup-status = %x79 / %x6E / %x6D / private-status over-content = 1*6(TAB hdr-content) / 7(TAB hdr-content) *(TAB hdr-n-content) private-status = token ; except the values in newsgroup-status 9.5. Capability Lines This syntax defines the generic form of a capability line in the capabilities list (see Section 3.3.1). capability-line = capability-entry capability-entry = X-capability-entry X-capability-entry = capability-label *(WS capability-argument) capability-label = keyword capability-argument = token This syntax defines the specific capability entries for the capabilities in this specification. capability-entry =/ hdr-capability / ihave-capability / implementation-capability / list-capability / mode-reader-capability / newnews-capability / over-capability / post-capability / reader-capability hdr-capability = "HDR" ihave-capability = "IHAVE" implementation-capability = "IMPLEMENTATION" *(WS token)
list-capability = "LIST" 1*(WS keyword) mode-reader-capability = "MODE-READER" newnews-capability = "NEWNEWS" over-capability = "OVER" [WS "MSGID"] post-capability = "POST" reader-capability = "READER" version-line = "VERSION" 1*(WS version-number) version-number = nzDIGIT *5DIGIT 9.6. LIST Variants This section defines more specifically the keywords for the LIST command and the syntax of the corresponding response contents. ; active list-arguments =/ "ACTIVE" [WS wildmat] list-content =/ list-active-content list-active-content = active-groups-list ; active.times list-arguments =/ "ACTIVE.TIMES" [WS wildmat] list-content =/ list-active-times-content list-active-times-content = *(newsgroup-name SPA 1*DIGIT SPA newsgroup-creator CRLF) newsgroup-creator = U-TEXT ; distrib.pats list-arguments =/ "DISTRIB.PATS" list-content =/ list-distrib-pats-content list-distrib-pats-content = *(1*DIGIT ":" wildmat ":" distribution CRLF) distribution = token ; headers list-arguments =/ "HEADERS" [WS ("MSGID" / "RANGE")] list-content =/ list-headers-content list-headers-content = *(header-meta-name CRLF) / *((metadata-name / ":") CRLF) ; newsgroups list-arguments =/ "NEWSGROUPS" [WS wildmat] list-content =/ list-newsgroups-content list-newsgroups-content =
*(newsgroup-name WS newsgroup-description CRLF) newsgroup-description = S-TEXT ; overview.fmt list-arguments =/ "OVERVIEW.FMT" list-content =/ list-overview-fmt-content list-overview-fmt-content = "Subject:" CRLF "From:" CRLF "Date:" CRLF "Message-ID:" CRLF "References:" CRLF ( ":bytes" CRLF ":lines" / "Bytes:" CRLF "Lines:") CRLF *((header-name ":full" / metadata-name) CRLF) 9.7. Articles This syntax defines the non-terminal <article>, which represents the format of an article as described in Section 3.6. article = 1*header CRLF body header = header-name ":" [CRLF] SP header-content CRLF header-content = *(S-CHAR / [CRLF] WS) body = *(*B-CHAR CRLF) 9.8. General Non-terminals These non-terminals are used at various places in the syntax and are collected here for convenience. A few of these non-terminals are not used in this specification but are provided for the consistency and convenience of extension authors. multi-line-data-block = content-lines termination content-lines = *([content-text] CRLF) content-text = (".." / B-NONDOT) *B-CHAR termination = "." CRLF article-number = 1*16DIGIT header-name = 1*A-NOTCOLON keyword = ALPHA 2*(ALPHA / DIGIT / "." / "-") message-id = "<" 1*248A-NOTGT ">" newsgroup-name = 1*wildmat-exact token = 1*P-CHAR wildmat = wildmat-pattern *("," ["!"] wildmat-pattern) wildmat-pattern = 1*wildmat-item wildmat-item = wildmat-exact / wildmat-wild wildmat-exact = %x22-29 / %x2B / %x2D-3E / %x40-5A / %x5E-7E /
UTF8-non-ascii ; exclude ! * , ? [ \ ] wildmat-wild = "*" / "?" base64 = *(4base64-char) [base64-terminal] base64-char = UPPER / LOWER / DIGIT / "+" / "/" base64-terminal = 2base64-char "==" / 3base64-char "=" ; Assorted special character sets ; A- means based on US-ASCII, excluding controls and SP ; P- means based on UTF-8, excluding controls and SP ; U- means based on UTF-8, excluding NUL CR and LF ; B- means based on bytes, excluding NUL CR and LF A-CHAR = %x21-7E A-NOTCOLON = %x21-39 / %x3B-7E ; exclude ":" A-NOTGT = %x21-3D / %x3F-7E ; exclude ">" P-CHAR = A-CHAR / UTF8-non-ascii U-CHAR = CTRL / TAB / SP / A-CHAR / UTF8-non-ascii U-NONTAB = CTRL / SP / A-CHAR / UTF8-non-ascii U-TEXT = P-CHAR *U-CHAR B-CHAR = CTRL / TAB / SP / %x21-FF B-NONDOT = CTRL / TAB / SP / %x21-2D / %x2F-FF ; exclude "." ALPHA = UPPER / LOWER ; use only when case-insensitive CR = %x0D CRLF = CR LF CTRL = %x01-08 / %x0B-0C / %x0E-1F DIGIT = %x30-39 nzDIGIT = %x31-39 EOL = *(SP / TAB) CRLF LF = %x0A LOWER = %x61-7A SP = %x20 SPA = 1*SP TAB = %x09 UPPER = %x41-5A UTF8-non-ascii = UTF8-2 / UTF8-3 / UTF8-4 UTF8-2 = %xC2-DF UTF8-tail UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2UTF8-tail / %xED %x80-9F UTF8-tail / %xEE-EF 2UTF8-tail UTF8-4 = %xF0 %x90-BF 2UTF8-tail / %xF1-F3 3UTF8-tail / %xF4 %x80-8F 2UTF8-tail UTF8-tail = %x80-BF WS = 1*(SP / TAB) The following non-terminals require special consideration. They represent situations where material SHOULD be restricted to UTF-8, but implementations MUST be able to cope with other character encodings. Therefore, there are two sets of definitions for them.
Implementations MUST accept any content that meets this syntax: S-CHAR = %x21-FF S-NONTAB = CTRL / SP / S-CHAR S-TEXT = (CTRL / S-CHAR) *B-CHAR and MAY pass such content on unaltered. When generating new content or re-encoding existing content, implementations SHOULD conform to this syntax: S-CHAR = P-CHAR S-NONTAB = U-NONTAB S-TEXT = U-TEXT 9.9. Extensions and Validation The specification of a registered extension MUST include formal syntax that defines additional forms for the following non-terminals: command for each new command other than a variant of the LIST command - the syntax of each command MUST be compatible with the definition of <X-command>; command-datastream for each new command that immediately streams data; command-continuation for each new command that sends further material after the initial command line - the syntax of each continuation MUST be exactly what is sent to the server, including any escape mechanisms such as "dot-stuffing"; initial-response-content for each new response code that has arguments - the syntax of each response MUST be compatible with the definition of <X-initial- response-content>; multi-line-response-content for each new response code that has a multi-line response - the syntax MUST show the response after the lines containing the response code and the terminating octet have been removed and any "dot-stuffing" undone; capability-entry for each new capability label - the syntax of each entry MUST be compatible with the definition of <X-capability-entry>;
list-arguments for each new variant of the LIST command - the syntax of each entry MUST be compatible with the definition of <X-command>; list-content for each new variant of the LIST command - the syntax MUST show the response after the lines containing the 215 response code and the terminating octet have been removed and any "dot-stuffing" undone. The =/ notation of ABNF [RFC4234] and the naming conventions described in Section 9.1 SHOULD be used for this. When the syntax in this specification, or syntax based on it, is validated, it should be noted that: o the non-terminals <command-line>, <command-datastream>, <command-continuation>, <response>, and <multi-line-response-content> describe basic concepts of the protocol and are not referred to by any other rule; o the non-terminal <base64> is provided for the convenience of extension authors and is not referred to by any rule in this specification; o for the reasons given above, the non-terminals <S-CHAR>, <S-NONTAB>, and <S-TEXT> each have two definitions; and o the non-terminal <UNDEFINED> is deliberately not defined. 10. Internationalisation Considerations 10.1. Introduction and Historical Situation RFC 977 [RFC977] was written at a time when internationalisation was not seen as a significant issue. As such, it was written on the assumption that all communication would be in ASCII and use only a 7-bit transport layer, although in practice just about all known implementations are 8-bit clean. Since then, Usenet and NNTP have spread throughout the world. In the absence of standards for handling the issues of language and character sets, countries, newsgroup hierarchies, and individuals have found a variety of solutions that work for them but that are not necessarily appropriate elsewhere. For example, some have adopted a default 8-bit character set appropriate to their needs (such as ISO/IEC 8859-1 in Western Europe or KOI-8 in Russia), others have used ASCII (either US-ASCII or national variants) in headers but
local 16-bit character sets in article bodies, and still others have gone for a combination of MIME [RFC2045] and UTF-8. With the increased use of MIME in email, it is becoming more common to find NNTP articles containing MIME headers that identify the character set of the body, but this is far from universal. The resulting confusion does not help interoperability. One point that has been generally accepted is that articles can contain octets with the top bit set, and NNTP is only expected to operate on 8-bit clean transport paths. 10.2. This Specification Part of the role of this present specification is to eliminate this confusion and promote interoperability as far as possible. At the same time, it is necessary to accept the existence of the present situation and not break existing implementations and arrangements gratuitously, even if they are less than optimal. Therefore, the current practice described above has been taken into consideration in producing this specification. This specification extends NNTP from US-ASCII [ANSI1986] to UTF-8 [RFC3629]. Except in the two areas discussed below, UTF-8 (which is a superset of US-ASCII) is mandatory, and implementations MUST NOT use any other encoding. Firstly, the use of MIME for article headers and bodies is strongly recommended. However, given widely divergent existing practices, an attempt to require a particular encoding and tagging standard would be premature at this time. Accordingly, this specification allows the use of arbitrary 8-bit data in articles subject to the following requirements and recommendations. o The names of headers (e.g., "From" or "Subject") MUST be in US-ASCII. o Header values SHOULD use US-ASCII or an encoding based on it, such as RFC 2047 [RFC2047], until such time as another approach has been standardised. At present, 8-bit encodings (including UTF-8) SHOULD NOT be used because they are likely to cause interoperability problems. o The character set of article bodies SHOULD be indicated in the article headers, and this SHOULD be done in accordance with MIME. o Where an article is obtained from an external source, an implementation MAY pass it on and derive data from it (such as the
response to the HDR command), even though the article or the data does not meet the above requirements. Implementations MUST transfer such articles and data correctly and unchanged; they MUST NOT attempt to convert or re-encode the article or derived data. (Nevertheless, a client or server MAY elect not to post or forward the article if, after further examination of the article, it deems it inappropriate to do so.) This requirement affects the ARTICLE (Section 6.2.1), BODY (Section 6.2.3), HDR (Section 8.5), HEAD (Section 6.2.2), IHAVE (Section 6.3.2), OVER (Section 8.3), and POST (Section 6.3.1) commands. Secondly, the following requirements are placed on the newsgroups list returned by the LIST NEWSGROUPS command (Section 7.6.6): o Although this specification allows UTF-8 for newsgroup names, they SHOULD be restricted to US-ASCII until a successor to RFC 1036 [RFC1036] standardises another approach. 8-bit encodings SHOULD NOT be used because they are likely to cause interoperability problems. o The newsgroup description SHOULD be in US-ASCII or UTF-8 unless and until a successor to RFC 1036 standardises other encoding arrangements. 8-bit encodings other than UTF-8 SHOULD NOT be used because they are likely to cause interoperability problems. o Implementations that obtain this data from an external source MUST handle it correctly even if it does not meet the above requirements. Implementations (in particular, clients) MUST handle such data correctly. 10.3. Outstanding Issues While the primary use of NNTP is for transmitting articles that conform to RFC 1036 (Netnews articles), it is also used for other formats (see Appendix A). It is therefore most appropriate that internationalisation issues related to article formats be addressed in the relevant specifications. For Netnews articles, this is any successor to RFC 1036. For email messages, it is RFC 2822 [RFC2822]. Of course, any article transmitted via NNTP needs to conform to this specification as well. Restricting newsgroup names to UTF-8 is not a complete solution. In particular, when new newsgroup names are created or a user is asked to enter a newsgroup name, some scheme of canonicalisation will need to take place. This specification does not attempt to define that
canonicalization; further work is needed in this area, in conjunction with the article format specifications. Until such specifications are published, implementations SHOULD match newsgroup names octet by octet. It is anticipated that any approved scheme will be applied "at the edges", and therefore octet-by-octet comparison will continue to apply to most, if not all, uses of newsgroup names in NNTP. In the meantime, any implementation experimenting with UTF-8 newsgroup names is strongly cautioned that a future specification may require that those names be canonicalized when used with NNTP in a way that is not compatible with their experiments. Since the primary use of NNTP is with Netnews, and since newsgroup descriptions are normally distributed through specially formatted articles, it is recommended that the internationalisation issues related to them be addressed in any successor to RFC 1036. 11. IANA Considerations This specification requires IANA to keep a registry of capability labels. The initial contents of this registry are specified in Section 3.3.4. As described in Section 3.3.3, labels beginning with X are reserved for private use, while all other names are expected to be associated with a specification in an RFC on the standards track or defining an IESG-approved experimental protocol. Different entries in the registry MUST use different capability labels. Different entries in the registry MUST NOT use the same command name. For this purpose, variants distinguished by a second or subsequent keyword (e.g., "LIST HEADERS" and "LIST OVERVIEW.FMT") count as different commands. If there is a need for two extensions to use the same command, a single harmonised specification MUST be registered. 12. Security Considerations This section is meant to inform application developers, information providers, and users of the security limitations in NNTP as described by this document. The discussion does not include definitive solutions to the problems revealed, though it does make some suggestions for reducing security risks.
12.1. Personal and Proprietary Information NNTP, because it was created to distribute network news articles, will forward whatever information is stored in those articles. Specification of that information is outside this scope of this document, but it is likely that some personal and/or proprietary information is available in some of those articles. It is very important that designers and implementers provide informative warnings to users so that personal and/or proprietary information in material that is added automatically to articles (e.g., in headers) is not disclosed inadvertently. Additionally, effective and easily understood mechanisms to manage the distribution of news articles SHOULD be provided to NNTP Server administrators, so that they are able to report with confidence the likely spread of any particular set of news articles. 12.2. Abuse of Server Log Information A server is in the position to save session data about a user's requests that might identify their reading patterns or subjects of interest. This information is clearly confidential in nature, and its handling can be constrained by law in certain countries. People using this protocol to provide data are responsible for ensuring that such material is not distributed without the permission of any individuals that are identifiable by the published results. 12.3. Weak Authentication and Access Control There is no user-based or token-based authentication in the basic NNTP specification. Access is normally controlled by server configuration files. Those files specify access by using domain names or IP addresses. However, this specification does permit the creation of extensions to NNTP for such purposes; one such extension is [NNTP-AUTH]. While including such mechanisms is optional, doing so is strongly encouraged. Other mechanisms are also available. For example, a proxy server could be put in place that requires authentication before connecting via the proxy to the NNTP server. 12.4. DNS Spoofing Many existing NNTP implementations authorize incoming connections by checking the IP address of that connection against the IP addresses obtained via DNS lookups of lists of domain names given in local configuration files. Servers that use this type of authentication and clients that find a server by doing a DNS lookup of the server name rely very heavily on the Domain Name Service, and are thus
generally prone to security attacks based on the deliberate misassociation of IP addresses and DNS names. Clients and servers need to be cautious in assuming the continuing validity of an IP number/DNS name association. In particular, NNTP clients and servers SHOULD rely on their name resolver for confirmation of an IP number/DNS name association, rather than cache the result of previous host name lookups. Many platforms already can cache host name lookups locally when appropriate, and they SHOULD be configured to do so. It is proper for these lookups to be cached, however, only when the TTL (Time To Live) information reported by the name server makes it likely that the cached information will remain useful. If NNTP clients or servers cache the results of host name lookups in order to achieve a performance improvement, they MUST observe the TTL information reported by DNS. If NNTP clients or servers do not observe this rule, they could be spoofed when a previously accessed server's IP address changes. As network renumbering is expected to become increasingly common, the possibility of this form of attack will increase. Observing this requirement thus reduces this potential security vulnerability. This requirement also improves the load-balancing behaviour of clients for replicated servers using the same DNS name and reduces the likelihood of a user's experiencing failure in accessing sites that use that strategy. 12.5. UTF-8 Issues UTF-8 [RFC3629] permits only certain sequences of octets and designates others as either malformed or "illegal". The Unicode standard identifies a number of security issues related to illegal sequences and forbids their generation by conforming implementations. Implementations of this specification MUST NOT generate malformed or illegal sequences and SHOULD detect them and take some appropriate action. This could include the following: o Generating a 501 response code. o Replacing such sequences by the sequence %xEF.BF.BD, which encodes the "replacement character" U+FFFD. o Closing the connection. o Replacing such sequences by a "guessed" valid sequence (based on properties of the UTF-8 encoding).
In the last case, the implementation MUST ensure that any replacement cannot be used to bypass validity or security checks. For example, the illegal sequence %xC0.A0 is an over-long encoding for space (%x20). If it is replaced by the correct encoding in a command line, this needs to happen before the command line is parsed into individual arguments. If the replacement came after parsing, it would be possible to generate an argument with an embedded space, which is forbidden. Use of the "replacement character" does not have this problem, since it is permitted wherever non-US-ASCII characters are. Implementations SHOULD use one of the first two solutions where the general structure of the NNTP stream remains intact and SHOULD close the connection if it is no longer possible to parse it sensibly. 12.6. Caching of Capability Lists The CAPABILITIES command provides a capability list, which is information about the current capabilities of the server. Whenever there is a relevant change to the server state, the results of this command are required to change accordingly. In most situations, the capabilities list in a given server state will not change from session to session; for example, a given extension will be installed permanently on a server. Some clients may therefore wish to remember which extensions a server supports to avoid the delay of an additional command and response, particularly if they open multiple connections in the same session. However, information about extensions related to security and privacy MUST NOT be cached, since this could allow a variety of attacks. For example, consider a server that permits the use of cleartext passwords on links that are encrypted but not otherwise: [Initial connection set-up completed.] [S] 200 NNTP Service Ready, posting permitted [C] CAPABILITIES [S] 101 Capability list: [S] VERSION 2 [S] READER [S] NEWNEWS [S] POST [S] XENCRYPT [S] LIST ACTIVE NEWSGROUPS [S] . [C] XENCRYPT [Client and server negotiate encryption on the link] [S] 283 Encrypted link established
[C] CAPABILITIES [S] 101 Capability list: [S] VERSION 2 [S] READER [S] NEWNEWS [S] POST [S] XSECRET [S] LIST ACTIVE NEWSGROUPS [S] . [C] XSECRET fred flintstone [S] 290 Password for fred accepted If the client caches the last capabilities list, then on the next session it will attempt to use XSECRET on an unencrypted link: [Initial connection set-up completed.] [S] 200 NNTP Service Ready, posting permitted [C] XSECRET fred flintstone [S] 483 Only permitted on secure links This exposes the password to any eavesdropper. While the primary cause of this is passing a secret without first checking the security of the link, caching of capability lists can increase the risk. Any security extension should include requirements to check the security state of the link in a manner appropriate to that extension. Caching should normally only be considered for anonymous clients that do not use any security or privacy extensions and for which the time required for an additional command and response is a noticeable issue. 13. Acknowledgements This document is the result of much effort by the present and past members of the NNTP Working Group, chaired by Russ Allbery and Ned Freed. It could not have been produced without them. The author acknowledges the original authors of NNTP as documented in RFC 977 [RFC977]: Brian Kantor and Phil Lapsey. The author gratefully acknowledges the following: o The work of the NNTP committee chaired by Eliot Lear. The organization of this document was influenced by the last available version from this working group. A special thanks to Eliot for generously providing the original machine-readable sources for that document.
o The work of the DRUMS working group, specifically RFC 1869 [RFC1869], that drove the original thinking that led to the CAPABILITIES command and the extensions mechanism detailed in this document. o The authors of RFC 2616 [RFC2616] for providing specific and relevant examples of security issues that should be considered for HTTP. Since many of the same considerations exist for NNTP, those examples that are relevant have been included here with some minor rewrites. o The comments and additional information provided by the following individuals in preparing one or more of the progenitors of this document: Russ Allbery <email@example.com> Wayne Davison <firstname.lastname@example.org> Chris Lewis <email@example.com> Tom Limoncelli <firstname.lastname@example.org> Eric Schnoebelen <email@example.com> Rich Salz <firstname.lastname@example.org> This work was motivated by the work of various news reader authors and news server authors, including those listed below: Rick Adams Original author of the NNTP extensions to the RN news reader and last maintainer of Bnews. Stan Barber Original author of the NNTP extensions to the news readers that are part of Bnews. Geoff Collyer Original author of the OVERVIEW database proposal and one of the original authors of CNEWS. Dan Curry Original author of the xvnews news reader. Wayne Davison Author of the first threading extensions to the RN news reader (commonly called TRN). Geoff Huston Original author of ANU NEWS.
Phil Lapsey Original author of the UNIX reference implementation for NNTP. Iain Lea Original maintainer of the TIN news reader. Chris Lewis First known implementer of the AUTHINFO GENERIC extension. Rich Salz Original author of INN. Henry Spencer One of the original authors of CNEWS. Kim Storm Original author of the NN news reader. Other people who contributed to this document include: Matthias Andree Greg Andruk Daniel Barclay Maurizio Codogno Mark Crispin Andrew Gierth Juergen Helbing Scott Hollenbeck Urs Janssen Charles Lindsey Ade Lovett David Magda Ken Murchison Francois Petillon Peter Robinson Rob Siemborski Howard Swinehart Ruud van Tol Jeffrey Vinocur Erik Warmelink The author thanks them all and apologises to anyone omitted. Finally, the present author gratefully acknowledges the vast amount of work put into previous versions by the previous author: Stan Barber <email@example.com>
14. References 14.1. Normative References [ANSI1986] American National Standards Institute, "Coded Character Set - 7-bit American Standard Code for Information Interchange", ANSI X3.4, 1986. [RFC977] Kantor, B. and P. Lapsley, "Network News Transfer Protocol", RFC 977, February 1986. [RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996. [RFC2047] Moore, K., "MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text", RFC 2047, November 1996. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003. [RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005. [RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 4648, October 2006. [TF.686-1] International Telecommunications Union - Radio, "Glossary, ITU-R Recommendation TF.686-1", ITU-R Recommendation TF.686-1, October 1997. 14.2. Informative References [NNTP-AUTH] Vinocur, J., Murchison, K., and C. Newman, "Network News Transfer Protocol (NNTP) Extension for Authentication", RFC 4643, October 2006. [NNTP-STREAM] Vinocur, J. and K. Murchison, "Network News Transfer Protocol (NNTP) Extension for Streaming Feeds", RFC 4644, October 2006.
[NNTP-TLS] Murchison, K., Vinocur, J., and C. Newman, "Using Transport Layer Security (TLS) with Network News Transfer Protocol (NNTP)", RFC 4642, October 2006. [RFC1036] Horton, M. and R. Adams, "Standard for interchange of USENET messages", RFC 1036, December 1987. [RFC1305] Mills, D., "Network Time Protocol (Version 3) Specification, Implementation and Analysis", RFC 1305, March 1992. [RFC1869] Klensin, J., Freed, N., Rose, M., Stefferud, E., and D. Crocker, "SMTP Service Extensions", STD 10, RFC 1869, November 1995. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC2629] Rose, M., "Writing I-Ds and RFCs using XML", RFC 2629, June 1999. [RFC2822] Resnick, P., "Internet Message Format", RFC 2822, April 2001. [RFC2980] Barber, S., "Common NNTP Extensions", RFC 2980, October 2000. [ROBE1995] Robertson, R., "FAQ: Overview database / NOV General Information", January 1995. There is no definitive copy of this document known to the author. It was previously posted as the Usenet article <news:nov-faq-1-930909720@agate.Berkeley.EDU> [SALZ1992] Salz, R., "Manual Page for wildmat(3) from the INN 1.4 distribution, Revision 1.10", April 1992. There is no definitive copy of this document known to the author.
Appendix A. Interaction with Other Specifications NNTP is most often used for transferring articles that conform to RFC 1036 [RFC1036] (such articles are called "Netnews articles" here). It is also sometimes used for transferring email messages that conform to RFC 2822 [RFC2822] (such articles are called "email articles" here). In this situation, articles must conform both to this specification and to that other one; this appendix describes some relevant issues. A.1. Header Folding NNTP allows a header line to be folded (by inserting a CRLF pair) before any space or TAB character. Both email and Netnews articles are required to have at least one octet other than space or TAB on each header line. Thus, folding can only happen at one point in each sequence of consecutive spaces or TABs. Netnews articles are further required to have the header name, colon, and following space all on the first line; folding may only happen beyond that space. Finally, some non-conforming software will remove trailing spaces and TABs from a line. Therefore, it might be inadvisable to fold a header after a space or TAB. For maximum safety, header lines SHOULD conform to the following syntax rather than to that in Section 9.7. header = header-name ":" SP [header-content] CRLF header-content = [WS] token *( [CRLF] WS token ) A.2. Message-IDs Every article handled by an NNTP server MUST have a unique message-id. For the purposes of this specification, a message-id is an arbitrary opaque string that merely needs to meet certain syntactic requirements and is just a way to refer to the article. Because there is a significant risk that old articles will be reinjected into the global Usenet system, RFC 1036 [RFC1036] requires that message-ids are globally unique for all time. This specification states that message-ids are the same if and only if they consist of the same sequence of octets. Other specifications may define two different sequences as being equal because they are putting an interpretation on particular characters. RFC 2822 [RFC2822] has a concept of "quoted" and "escaped" characters. It therefore considers the three message-ids:
<firstname.lastname@example.org> <"ab.cd"@example.com> <"ab.\cd"@example.com> as being identical. Therefore, an NNTP implementation handing email articles must ensure that only one of these three appears in the protocol and that the other two are converted to it as and when necessary, such as when a client checks the results of a NEWNEWS command against an internal database of message-ids. Note that RFC 1036 [RFC1036] never treats two different strings as being identical. Its successor (as of the time of writing) restricts the syntax of message-ids so that, whenever RFC 2822 would treat two strings as equivalent, only one of them is valid (in the above example, only the first string is valid). This specification does not describe how the message-id of an article is determined; it may be deduced from the contents of the article or derived from some external source. If the server is also conforming to another specification that contains a definition of message-id compatible with this one, the server SHOULD use those message-ids. A common approach, and one that SHOULD be used for email and Netnews articles, is to extract the message-id from the contents of a header with name "Message-ID". This may not be as simple as copying the entire header contents; it may be necessary to strip off comments and undo quoting, or to reduce "equivalent" message-ids to a canonical form. If an article is obtained through the IHAVE command, there will be a message-id provided with the command. The server MAY either use it or determine one from the article contents. However, whichever it does, it SHOULD ensure that, if the IHAVE command is repeated with the same argument and article, it will be recognized as a duplicate. If an article does not contain a message-id that the server can identify, it MUST synthesize one. This could, for example, be a simple sequence number or be based on the date and time when the article arrived. When email or Netnews articles are handled, a Message-ID header SHOULD be added to ensure global consistency and uniqueness. Note that, because the message-id might not have been derived from the Message-ID header in the article, the following example is legitimate (though unusual):
[C] HEAD <email@example.com> [S] 221 0 <firstname.lastname@example.org> [S] Path: pathost!demo!whitehouse!not-for-mail [S] Message-ID: <email@example.com> [S] From: "Demo User" <firstname.lastname@example.org> [S] Newsgroups: misc.test [S] Subject: I am just a test article [S] Date: 6 Oct 1998 04:38:40 -0500 [S] Organization: An Example Net, Uncertain, Texas [S] . A.3. Article Posting As far as NNTP is concerned, the POST and IHAVE commands provide the same basic facilities in a slightly different way. However, they have rather different intentions. The IHAVE command is intended for transmitting conforming articles between a system of NNTP servers, with all articles perhaps also conforming to another specification (e.g., all articles are Netnews articles). It is expected that the client will already have done any necessary validation (or that it has in turn obtained the article from a third party that has done so); therefore, the contents SHOULD be left unchanged. In contrast, the POST command is intended for use when an end-user is injecting a newly created article into a such a system. The article being transferred might not be a conforming email or Netnews article, and the server is expected to validate it and, if necessary, to convert it to the right form for onward distribution. This is often done by a separate piece of software on the server installation; if so, the NNTP server SHOULD pass the incoming article to that software unaltered, making no attempt to filter characters, to fold or limit lines, or to process the incoming text otherwise. The POST command can fail in various ways, and clients should be prepared to re-send an article. When doing so, however, it is often important to ensure (as far as possible) that the same message-id is allocated to both attempts so that the server, or other servers, can recognize the two articles as duplicates. In the case of email or Netnews articles, therefore, the posted article SHOULD contain a header with the name "Message-ID", and the contents of this header SHOULD be identical on each attempt. The server SHOULD ensure that two POSTed articles with the same contents for this header are recognized as identical and that the same message-id is allocated, whether or not those contents are suitable for use as the message-id.
Appendix B. Summary of Commands This section contains a list of every command defined in this document, ordered by command name and by indicating capability. Ordered by command name: +-------------------+-----------------------+---------------+ | Command | Indicating capability | Definition | +-------------------+-----------------------+---------------+ | ARTICLE | READER | Section 6.2.1 | | BODY | READER | Section 6.2.3 | | CAPABILITIES | mandatory | Section 5.2 | | DATE | READER | Section 7.1 | | GROUP | READER | Section 6.1.1 | | HDR | HDR | Section 8.5 | | HEAD | mandatory | Section 6.2.2 | | HELP | mandatory | Section 7.2 | | IHAVE | IHAVE | Section 6.3.2 | | LAST | READER | Section 6.1.3 | | LIST | LIST | Section 7.6.1 | | LIST ACTIVE.TIMES | LIST | Section 7.6.4 | | LIST ACTIVE | LIST | Section 7.6.3 | | LIST DISTRIB.PATS | LIST | Section 7.6.5 | | LIST HEADERS | HDR | Section 8.6 | | LIST NEWSGROUPS | LIST | Section 7.6.6 | | LIST OVERVIEW.FMT | OVER | Section 8.4 | | LISTGROUP | READER | Section 6.1.2 | | MODE READER | MODE-READER | Section 5.3 | | NEWGROUPS | READER | Section 7.3 | | NEWNEWS | NEWNEWS | Section 7.4 | | NEXT | READER | Section 6.1.4 | | OVER | OVER | Section 8.3 | | POST | POST | Section 6.3.1 | | QUIT | mandatory | Section 5.4 | | STAT | mandatory | Section 6.2.4 | +-------------------+-----------------------+---------------+
Ordered by indicating capability: +-------------------+-----------------------+---------------+ | Command | Indicating capability | Definition | +-------------------+-----------------------+---------------+ | CAPABILITIES | mandatory | Section 5.2 | | HEAD | mandatory | Section 6.2.2 | | HELP | mandatory | Section 7.2 | | QUIT | mandatory | Section 5.4 | | STAT | mandatory | Section 6.2.4 | | HDR | HDR | Section 8.5 | | LIST HEADERS | HDR | Section 8.6 | | IHAVE | IHAVE | Section 6.3.2 | | LIST | LIST | Section 7.6.1 | | LIST ACTIVE | LIST | Section 7.6.3 | | LIST ACTIVE.TIMES | LIST | Section 7.6.4 | | LIST DISTRIB.PATS | LIST | Section 7.6.5 | | LIST NEWSGROUPS | LIST | Section 7.6.6 | | MODE READER | MODE-READER | Section 5.3 | | NEWNEWS | NEWNEWS | Section 7.4 | | OVER | OVER | Section 8.3 | | LIST OVERVIEW.FMT | OVER | Section 8.4 | | POST | POST | Section 6.3.1 | | ARTICLE | READER | Section 6.2.1 | | BODY | READER | Section 6.2.3 | | DATE | READER | Section 7.1 | | GROUP | READER | Section 6.1.1 | | LAST | READER | Section 6.1.3 | | LISTGROUP | READER | Section 6.1.2 | | NEWGROUPS | READER | Section 7.3 | | NEXT | READER | Section 6.1.4 | +-------------------+-----------------------+---------------+
Appendix C. Summary of Response Codes This section contains a list of every response code defined in this document and indicates whether it is multi-line, which commands can generate it, what arguments it has, and what its meaning is. Response code 100 (multi-line) Generated by: HELP Meaning: help text follows. Response code 101 (multi-line) Generated by: CAPABILITIES Meaning: capabilities list follows. Response code 111 Generated by: DATE 1 argument: yyyymmddhhmmss Meaning: server date and time. Response code 200 Generated by: initial connection, MODE READER Meaning: service available, posting allowed. Response code 201 Generated by: initial connection, MODE READER Meaning: service available, posting prohibited. Response code 205 Generated by: QUIT Meaning: connection closing (the server immediately closes the connection). Response code 211 The 211 response code has two completely different forms, depending on which command generated it: (not multi-line) Generated by: GROUP 4 arguments: number low high group Meaning: group selected. (multi-line) Generated by: LISTGROUP 4 arguments: number low high group Meaning: article numbers follow.
Response code 215 (multi-line) Generated by: LIST Meaning: information follows. Response code 220 (multi-line) Generated by: ARTICLE 2 arguments: n message-id Meaning: article follows. Response code 221 (multi-line) Generated by: HEAD 2 arguments: n message-id Meaning: article headers follow. Response code 222 (multi-line) Generated by: BODY 2 arguments: n message-id Meaning: article body follows. Response code 223 Generated by: LAST, NEXT, STAT 2 arguments: n message-id Meaning: article exists and selected. Response code 224 (multi-line) Generated by: OVER Meaning: overview information follows. Response code 225 (multi-line) Generated by: HDR Meaning: headers follow. Response code 230 (multi-line) Generated by: NEWNEWS Meaning: list of new articles follows. Response code 231 (multi-line) Generated by: NEWGROUPS Meaning: list of new newsgroups follows. Response code 235 Generated by: IHAVE (second stage) Meaning: article transferred OK. Response code 240 Generated by: POST (second stage) Meaning: article received OK.
Response code 335 Generated by: IHAVE (first stage) Meaning: send article to be transferred. Response code 340 Generated by: POST (first stage) Meaning: send article to be posted. Response code 400 Generic response and generated by initial connection Meaning: service not available or no longer available (the server immediately closes the connection). Response code 401 Generic response 1 argument: capability-label Meaning: the server is in the wrong mode; the indicated capability should be used to change the mode. Response code 403 Generic response Meaning: internal fault or problem preventing action being taken. Response code 411 Generated by: GROUP, LISTGROUP Meaning: no such newsgroup. Response code 412 Generated by: ARTICLE, BODY, GROUP, HDR, HEAD, LAST, LISTGROUP, NEXT, OVER, STAT Meaning: no newsgroup selected. Response code 420 Generated by: ARTICLE, BODY, HDR, HEAD, LAST, NEXT, OVER, STAT Meaning: current article number is invalid. Response code 421 Generated by: NEXT Meaning: no next article in this group. Response code 422 Generated by: LAST Meaning: no previous article in this group. Response code 423 Generated by: ARTICLE, BODY, HDR, HEAD, OVER, STAT Meaning: no article with that number or in that range.
Response code 430 Generated by: ARTICLE, BODY, HDR, HEAD, OVER, STAT Meaning: no article with that message-id. Response code 435 Generated by: IHAVE (first stage) Meaning: article not wanted. Response code 436 Generated by: IHAVE (either stage) Meaning: transfer not possible (first stage) or failed (second stage); try again later. Response code 437 Generated by: IHAVE (second stage) Meaning: transfer rejected; do not retry. Response code 440 Generated by: POST (first stage) Meaning: posting not permitted. Response code 441 Generated by: POST (second stage) Meaning: posting failed. Response code 480 Generic response Meaning: command unavailable until the client has authenticated itself. Response code 483 Generic response Meaning: command unavailable until suitable privacy has been arranged. Response code 500 Generic response Meaning: unknown command. Response code 501 Generic response Meaning: syntax error in command.
Response code 502 Generic response and generated by initial connection Meaning for the initial connection and the MODE READER command: service permanently unavailable (the server immediately closes the connection). Meaning for all other commands: command not permitted (and there is no way for the client to change this). Response code 503 Generic response Meaning: feature not supported. Response code 504 Generic response Meaning: error in base64-encoding [RFC4648] of an argument. Appendix D. Changes from RFC 977 In general every attempt has been made to ensure that the protocol specification in this document is compatible with the version specified in RFC 977 [RFC977] and the various facilities adopted from RFC 2980 [RFC2980]. However, there have been a number of changes, some compatible and some not. This appendix lists these changes. It is not guaranteed to be exhaustive or correct and MUST NOT be relied on. o A formal syntax specification (Section 9) has been added. o The default character set is changed from US-ASCII [ANSI1986] to UTF-8 [RFC3629] (note that US-ASCII is a subset of UTF-8). This matter is discussed further in Section 10. o All articles are required to have a message-id, eliminating the "<0>" placeholder used in RFC 977 in some responses. o The newsgroup name matching capabilities already documented in RFC 977 ("wildmats", Section 4) are clarified and extended. The new facilities (e.g., the use of commas and exclamation marks) are allowed wherever wildmats appear in the protocol. o Support for pipelining of commands (Section 3.5) is made mandatory.
o The principles behind response codes (Section 3.2) have been tidied up. In particular: * the x8x response code family, formerly used for private extensions, is now reserved for authentication and privacy extensions; * the x9x response code family, formerly intended for debugging facilities, are now reserved for private extensions; * the 502 and 503 generic response codes (Section 3.2.1) have been redefined; * new 401, 403, 480, 483, and 504 generic response codes have been added. o The rules for article numbering (Section 6) have been clarified (also see Section 188.8.131.52). o The SLAVE command (which was ill-defined) is removed from the protocol. o Four-digit years are permitted in the NEWNEWS (Section 7.4) and NEWGROUPS (Section 7.3) commands (two-digit years are still permitted). The optional distribution parameter to these commands has been removed. o The LIST command (Section 7.6.1) is greatly extended; the original is available as LIST ACTIVE, while new variants include ACTIVE.TIMES, DISTRIB.PATS, and NEWSGROUPS. A new "m" status flag is added to the LIST ACTIVE response. o A new CAPABILITIES command (Section 5.2) allows clients to determine what facilities are supported by a server. o The DATE command (Section 7.1) is adopted from RFC 2980 effectively unchanged. o The LISTGROUP command (Section 6.1.2) is adopted from RFC 2980. An optional range argument has been added, and the 211 initial response line now has the same format as the 211 response from the GROUP command. o The MODE READER command (Section 5.3) is adopted from RFC 2980 and its meaning and effects clarified. o The XHDR command in RFC 2980 has been formalised as the new HDR (Section 8.5) and LIST HEADERS (Section 8.6) commands.
o The XOVER command in RFC 2980 has been formalised as the new OVER (Section 8.3) and LIST OVERVIEW.FMT (Section 8.4) commands. The former can be applied to a message-id as well as to a range. o The concept of article metadata (Section 8.1) has been formalised, allowing the Bytes and Lines pseudo-headers to be deprecated. Client authors should note in particular that lack of support for the CAPABILITIES command is a good indication that the server does not support this specification.
Full Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- email@example.com. Acknowledgement Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA).