Internet Engineering Task Force (IETF) M. Barnes Request for Comments: 6385 Polycom Category: Informational A. Doria ISSN: 2070-1721 Research Consultant H. Alvestrand Google B. Carpenter University of Auckland October 2011 General Area Review Team (Gen-ART) Experiences
AbstractThe General Area Review Team (Gen-ART) has been doing reviews of Internet-Drafts (I-Ds) since 2004. This document discusses the experience and the lessons learned over the past 7 years of this process. The review team initially reviewed the I-Ds before each of the IESG telechats. Since late 2005, review team members have been assigned to review I-Ds during IETF Last Call, unless no IETF Last Call is necessary for the I-D. The same reviewer then reviews any updates when the I-D is placed on an IESG telechat agenda. Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 5741. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc6385.
Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 1. Introduction ....................................................2 2. Who Are the Gen-ART Members? ....................................3 3. Goals of Gen-ART ................................................3 4. Gen-ART Reviews .................................................4 4.1. IETF Last Call Review Process ..............................4 4.2. IESG Telechat Review Process ...............................5 4.3. Form of Review .............................................5 4.4. Gen-ART Process Overview ...................................8 5. Secretarial Process ............................................10 5.1. Maintaining Review Spreadsheet ............................10 5.2. Last Call Assignment Procedure ............................12 5.3. Telechat Assignment Procedure .............................13 5.4. Capturing Reviews .........................................14 6. Results ........................................................14 7. Impressions ....................................................15 7.1. Reviewers' Impressions ....................................15 7.2. General Area Directors' Impressions .......................17 7.3. Gen-ART Secretaries' Impressions ..........................18 8. Needed Improvements ............................................18 9. Applicability ..................................................20 10. Security Considerations .......................................20 11. Acknowledgments ...............................................20 12. Informative References ........................................21
This process is likely to continue to change over time. The review team has been retained by subsequent General Area Directors. It has no official role in the IETF standards process, except as a set of individuals entitled, like everyone, to comment on Internet-Drafts (I-Ds). Its volunteers, including a secretary and the team of reviewers, serve at the invitation of the General AD. Both the reviews and the reviewer names are public. Section 11 provides a list of currently active reviewers, along with those who have served on the review team in the past.
summarized in Section 4.3. The overall objective is to ensure that the I-Ds are well structured; can be easily understood, at least at a high level; and provide a reasonable basis for implementation (for I-Ds intended for the Standards Track). While other area (and WG) directorates/review teams existed prior to Gen-ART and more have been established since Gen-ART, the roles of each are fairly distinct. Thus, there is little overlap between the goals and review criteria for the various review teams. It is also very valuable for these other review teams to operate independently. For example, when both Gen-ART reviews and Security Directorate (SecDir) reviews raise the same sorts of concerns, it's a clear red flag that the I-D needs more work before progressing. In addition, due to the typical thoroughness (and objectiveness) of the various review teams' reviews, the sponsoring AD and document shepherd are often able to work with the editors/WG (and vice versa, depending upon area and WG structure) to improve the overall quality of the final I-D. It should also be noted that some ADs take the Gen-ART reviews into consideration when preparing their own evaluations. Statistics from the Gen-ART reviews over the past 6+ years show a trend of increased quality and readiness for progression of I-Ds by the time they are placed on the telechat agenda. Additional statistics are discussed in Section 6.
The Last Call assignments are done on a fairly strict round-robin basis to ensure a fair workload amongst all the reviewers. Reviewers that are unavailable (vacations, etc.) during the review period timeframe obviously are excluded from that round of assignments, but remain in the same queue position for the next round. The order is occasionally modified to avoid assigning an editor/author or WG chair their own I-Ds. A reviewer may also NACK an assignment if they feel they may have some bias (although corporate affiliations are not considered to be sources of bias) or they don't feel they can review the I-D in a timely manner. The assignment process is completely manual, although a spreadsheet tremendously facilitates the process. The details are described in Section 5. Ideally, this process could be automated. However, manual intervention would still be required to maintain the appropriate available reviewer list (unless reviewers took on the task of maintaining their data in some sort of database). Further details on the tools necessary to automate the entire process are provided in Section 8. Section 4.4 provides the step-by-step telechat review assignment process, with specific details on the maintenance of the review assignment data, which is in turn maintained in review spreadsheets (Section 5). SIRS], making adaptations for the special "late, quick review" case and the nature of the General Area's concerns.
Each review must start with a summary statement chosen from or adapted from the following list: o This draft is ready for publication as a [type] RFC, where [type] is Informational, Experimental, etc. (In some cases, the review might recommend publication as a different [type] than requested by the author.) o This draft is basically ready for publication, but has nits that should be fixed before publication. o This draft is on the right track but has open issues, described in the review. o This draft has serious issues, described in the review, and needs to be rethought. o This draft has very fundamental issues, described in the review, and further work is not recommended. o Unfortunately, I don't have the expertise to review this draft. The length of a review can vary greatly according to circumstances, and it is considered acceptable for purely editorial comments to be sent privately if it's obvious that the I-D needs substantial revision. All substantive comments, however, must be included in the public review. Wherever possible, comments should be written as suggestions for improvement rather than as simple criticism. Explicit references to prior work and prior IETF discussion should be given whenever possible. Reviewers are asked to review for all kinds of problems, such as basic architectural or security issues, Internet-wide impact, technical nits, problems of form and format (such as IANA Considerations or incorrect references), and editorial issues. Since these reviews are on I-Ds that are supposed to be finished, the review should consider "no issue too small" -- but should cover the whole range, from the general architectural level to the editorial level. All reviews should apply generally agreed-upon IETF criteria, such as: o [RFC1958]: "Architectural Principles of the Internet" o [RFC3426]: "General Architectural and Policy Considerations" o [RFC3439]: "Some Internet Architectural Guidelines and Philosophy"
o ID-Checklist: The "ID checklist" document maintained by the IESG o [RFC2223bis]: "Instructions to Request for Comments (RFC) Authors" as updated by [RFC-STYLE]: "RFC Document Style" o [RFC5226]: BCP 26 - "Guidelines for Writing an IANA Considerations Section in RFCs" o [RFC3552]: BCP 72 - "Guidelines for Writing RFC Text on Security Considerations" o Any other applicable architectural or procedural documents. It is considered important that reviews give precise references to such criteria when relevant to a comment. Of special interest to the General area, because they do not fall under any other area, are: o A clear description of why the I-D or protocol is useful to the Internet. o Adherence to IETF formalities, such as capitalized "must", "should", etc. in normative statements, per the ID-Checklist. o Useful and reasonable IANA considerations. Ensure that all necessary registries are defined/referenced, and ensure definition and compliance with IANA assignment criteria. o Correct dependencies for normative references. o That the I-D is written in reasonably clear English. o Checking the updates/obsoletes information. o Running idnits and checking the output. o Checking that things imported by reference, especially from other RFCs, make sense (notably definitions of terms, security considerations, and lists of criteria) and ensuring they are used as intended by the referenced document. o That examples (e.g., Fully Qualified Domain Names (FQDNs), telephone numbers, IP addresses) are taken from the right spaces.
Section 5. o The availability of reviewers and the order of assignments for the next round of Last Call document assignments are updated weekly and are available on the directory where all the assignments and reviews are cached. o At telechat assignment time, all previously reviewed I-Ds are assigned to the reviewer who reviewed them previously, assuming that reviewer is available. Otherwise, these I-Ds are assigned to a new person in the process described below. o The secretary attempts to avoid assigning I-Ds that might conflict with other IETF roles such as WG chairs, other directorates, etc. However, in the cases where the secretary doesn't note the conflict, the reviewer should notify the secretary and Gen-ART mailing list so another reviewer may be assigned. o It should be emphasized that assignment is never made according to a reviewer's technical specialty. Even though it happens, when, for example, routing I-Ds fall on routing experts or MIB documents fall on MIB doctors, it is coincidental. To the reviewer, the choice looks random. o There is an attempt to evenly distribute I-Ds amongst reviewers at LC time by using a round-robin process, starting from where the previous week's assignments stopped. o Typically, there is no attempt made to actually equalize the load, as the length and complexity of the I-Ds are not taken into account in this process. (Thus, a reviewer could end up with a couple of hundred-page I-Ds, but this is statistically rare.) However, in the case of a reviewer that might receive more than one new LC I-D at one time, the secretary does try to ensure that both are not large I-Ds. o Once the assignments are made, the web pages that list the reviews and the assignments are posted. Since the telechat agenda is not published until the end of the day on the Thursdays prior to the telechats (i.e., one week prior), the secretary needs to complete the assignments on that Thursday evening. This often requires working later in the evening and also requires an Internet connection even when traveling.
o If the reviewers notice any problems or conflict of interest, a bargaining process, shifting I-Ds from one reviewer to another, takes place. The secretary updates the assignment files with any new assignments. o Once the review has been completed, the reviewer sends the review to the Gen-ART list, ideally using the template provided in the review assignment emails. Typically, reviews are also sent to authors, the responsible AD, and the WG chairs/document shepherd. The only case where this might not be done is when there are no issues found for a re-review and none had been found on an initial review. Sending the review to the authors, ADs, and/or WG chairs/ Proto Shepherds was originally voluntary but is now considered standard practice. Reviewers may also send the reviews to the IETF discussion list, but that is entirely at the discretion of the reviewer, in which case the author must be copied on the review to ensure they see any follow-up discussion. Reviewers may also send the comments to the WG; however, this typically causes the review to end up in the moderation queue, as most reviewers do not want to subscribe to the WG lists for the I-Ds they review. Thus, it is expected that any of the original recipients (i.e., authors, WG chairs/document shepherd, or responsible AD) may forward the review to the WG mailing list if they believe it is necessary. In the past, sending these reviews resulted in confusion among the authors, who may not have been expecting a Gen-ART review and may not be familiar with Gen-ART. Thus, reviewers are reminded to prepend to the email the description of Gen-ART and the purpose of the review. This information is part of the standard template provided in the review assignment emails. o The secretary gathers the reviews, sometimes edits them for format, and records the review in the spreadsheet on the web pages, including the synopsis. This is typically done on Thursday. This is one aspect of the process that can be easily delegated such that one volunteer uploads all the reviews and then the secretary need only update the fields in the spreadsheet. If the reviewer has not provided a synopsis ("Summary" field in the template), the secretary makes a best guess based on the review details. Note that in most cases the reviewers do include a synopsis. o Ideally, the reviews should be posted to the Gen-ART mailing list by close of business on the East coast on Tuesday. This is necessary to allow the General AD time to consider the reviews prior to the telechat. If the reviews are received after Tuesday, the review may not be read by the AD before the IESG telechat. Due to time constraints, the spreadsheets containing review summaries/assignments are only updated on Thursday evenings when
the new LC assignments and upcoming telechat assignments are done. Ideally, the reviews would get uploaded on the Tuesdays prior to the telechat along with the updated spreadsheets. o If the AD concludes that the concerns raised by the reviewer warrant placing a DISCUSS comment on the I-D, the AD will do so, and the DISCUSS must be resolved before the I-D advances. Usually, the reviewer will be involved in the resolution process, but the responsibility for the DISCUSS rests with the AD. Section 8 are carried out. Section 5.2. For telechat assignments, I-Ds are obviously only added in the cases where there is no previous LC assignment. For the other I-Ds, the appropriate fields are updated as described in Section 5.3. All the reviews can be accessed from the spreadsheet via hyperlinks from specific fields, as summarized below. The following information is maintained in the spreadsheet (in the order listed): 1. "Chat/LC Date": Indicates either the date on which the LC review is due or the date of the telechat. 2. "Document": Filename for the I-D, which includes a hyperlink to the IETF I-D tracker. 3. "Assigned": Name of the reviewer assigned to that I-D. 4. "Category": This field contains one of the following self- explanatory values: "Doc - WG", "Doc - Ind/AD", or "IETF LC". Note that Gen-ART does not review I-Ds submitted directly to the RFC Editor. The "IETF LC" value is of course entered for all I-Ds at LC time. It is changed to one of the other appropriate values, based on the information in the telechat agenda.
5. "Previous Review": This includes a link to any previous reviews. For example, when a doc appears on a telechat agenda, if an IETF LC review was done, this field is updated with the review summary for the LC review (i.e., the information from the "Current Review Summary" as described below is copied to this column). The field is set to "New" when an I-D is first assigned/added to the spreadsheet. In the case of returns, this field has a value of "Return" or "Return/IETF-LC" for I-Ds for which there is an LC review. It should be noted that since Gen-ART started doing reviews at LC time, there seem to be far fewer returns on the agenda. 6. "Current Review Summary": This field includes the review type and version number of the document that is to be reviewed or has been reviewed (e.g., LC: -02). When the field also contains a review summary after the review type/version number (e.g., Telechat: -06 Ready), the active hyperlink points to the review. Occasionally, a reviewer will re-review an I-D prior to its telechat assignment, in which case it is added to the spreadsheet, but the date does not change to maintain consistency in the date field, since the reviews themselves contain the review date. The following summarizes the steps to add a new I-D to the spreadsheet: 1. In order to optimize steps, blank rows are first inserted for the number of new I-Ds to be added. 2. To minimize data entry, a row with default fields (including the hyperlinks) is kept at the end of the file. There is a separate default row for IETF LC versus telechat assignments. This row is copied into each of the new blank rows. The dates are then entered (this allows double-checking that all I-Ds from the review assignments are accounted for, especially LC). 3. The I-D name is then copied to the name field as well as being appended to the hyperlink for the "Review Summary" field. The hyperlink is included as part of the default row. This minimizes the steps to enter the reviews in the spreadsheet. 4. The data is also sorted by "Chat/LC Date", "Assigned", and "Document". The file is then saved and closed. 5. The file is then reopened and saved as HTML. 6. The file is opened a second time and sorted by "Assigned", "Chat/LC Date", and "Document" to provide the I-D reviewers an easy way to find any outstanding assignments.
Section 5.1. 8. The assignment files and updated spreadsheets are then cached on the Gen-ART server. 9. An email providing a link to the assignment file along with the updated spreadsheets is sent to the Gen-ART mailing list. This email has a standard form, such that the reviewers can simply cut and paste the template to include the Gen-ART context statement and link to the FAQ.
Section 4.4, the spreadsheet is typically updated with the review summaries on Thursday evenings just prior to entering the data for that week's LC and any telechat assignments. The following summarizes the steps to capture the reviews: 1. Currently, an additional volunteer is assisting the secretary in caching the email reviews as they arrive. 2. In the cases where the review is included inline in the body of the email, the review is cut and pasted into a text file and saved with the reviewer's last name appended to the filename -- e.g., draft-ietf-xyz-00-smith.txt. 3. In the case where the review is included as an attachment to the email, the file can be directly saved and uploaded. 4. The volunteer uploads the reviews by around 5PM CST on Thursdays; thus, they are available to the secretary at the time that week's assignments are done. This sequence is necessary to ensure the information for I-Ds on the upcoming telechat is up to date. 5. The review summary is entered into the "Current Summary" field. Note that the hyperlink to the review (added at assignment time) will automatically work when the file is uploaded. 6. Once all the reviews have been entered and the spreadsheets formatted, the review spreadsheet is saved and files uploaded per the last three steps in Section 5.1.
(based on a sample of 50 reviews). For the I-Ds that were not reviewed at LC time, only about 1/4 of those were deemed "Ready" when they were reviewed for the telechat. So, doing the Gen-ART reviews at Last Call time does seem to improve the quality of the I-Ds for the telechat.
o Reading this stuff is interesting. I like having a reason to read a wide range of materials. o I am more than convinced that this can be, and is, a valuable process. It is, in my opinion, a pity that Senior IETF Reviewers (SIRS) and so on did not take off, because this late-stage reviewing is a poor substitute for doing the same thing at a much earlier stage. Very few of the drafts that have come past my screen are truly fully ready for IESG review. It is actually a joy to find the occasional nugget that is both well written and is a proper technical job, such that the reviewer really can say "This is ready". o I have certainly found the process intellectually stimulating! It encourages me to take a wider interest in what is going on in the IETF, but consumes a fair bit of time to do a proper job, and requires a very wide knowledge to be able to properly catch the cross-area implications: I hope (believe!) that this is something that one gets better at with experience and doing a few of these reviews. o There is probably a very limited pool of people who have both the time and the inclination to keep on doing these reviews. It does require a fair bit of dedication. o It is difficult to avoid correcting the English, even if that is not really the point: Often, really bad English (whether as a result of non-mother-tongue authors with limited grasp or mother- tongue authors using informal language) obscures/corrupts what is being said or just makes it impossible to read. o Mostly authors welcome the comments: I think most of them understand the concept of "ego-free reviewing", and we have generally been constructive rather than destructive. o Part of the job of Gen-ART is to think the unthinkable from another point of view, to challenge (apparently undocumented) assumptions, and apply experience from other fields.
o There are some mechanical issues. The process followed is far too manual. Everything needs to be robotic except for the judgment calls about which reviewer gets which draft. Similarly, the reviewer should be able to just paste the review into a web form, click, and it's sent off to everyone appropriate and posted to the review site.
secretary's process; thus, the only blocking point is finding the right folks that are interested in this type of volunteer role. As noted in Section 7.2, 30 would be a good size for the review team. This would cut the workload for an individual reviewer in half (given the current model of 9 reviewers on the "one doc per month" assignment cycle). Obviously, automation of the process would be a good thing. However, Gen-ART secretaries are not necessarily highly motivated to transition to a more automated approach until a significant part of the process is automated. In more recent consideration of this situation, it likely would be best to first automate the process of entering the reviews, as that benefits the review team as a whole. This automation should allow the reviewers to enter the reviews via a web interface that would automatically generate the appropriate emails -- quite similar to how the draft "Upload" tool currently works. Also, given consistent naming conventions for the review forms, this step would automate some of the process for the secretary, as the reviews would automatically appear via the spreadsheet hyperlinks, although there would still be a need to manually enter the summary. But this would eliminate the need to edit/normalize and upload files and, hopefully, eliminate the problem encountered with unflowed text in emails and getting the review properly formatted using some text editors. Section 5 was written to facilitate the process of determining tools requirements, by providing the very detailed steps currently applied to the process. As noted above, automating the upload of the reviews could be a good first step. This is somewhat starting at the end of the process. However, it seems that by automating in this direction, we may have optimal results; since one of the earliest steps in the process is the task of assigning reviewers, it likely needs the most manual intervention, even with tools available. The current SecDir secretary does use some tools for assignments and generating assignment emails. These tools could be considered for use by the Gen-ART secretary. Since the SecDir reviews are not cached and the information maintained for those reviews is less detailed, there would be no reusability of that aspect. However, if the Gen-ART spreadsheet can be automatically populated (with assignments and completed reviews), the SecDir may be able to make use of that same tool.
John Loughney Lucy Lynch Enrico Marocco Michael Patton Stefan Santesson Robert Sparks Tom Taylor Sean Turner Christian Vogt Suzanne Woolf We also thank the current team of reviewers/secretary: Mary Barnes (past secretary, 2005-2010) Richard Barnes David Black Ben Campbell Brian Carpenter (past General AD) Elwyn Davies Francis Dupont Roni Even Miguel-Angel Garcia Vijay Gurbani (assisting secretary to upload reviews) Wassim Haddad Joel Halpern Suresh Krishnan Peter McCann Jean Mahoney (secretary as of Jan. 2011) Alexey Melnikov Kathleen Moriarty [RFC1958] Carpenter, B., Ed., "Architectural Principles of the Internet", RFC 1958, June 1996. [RFC2223bis] Reynolds, J., Ed., and R. Braden, Ed., "Instructions to Request for Comments (RFC) Authors", Work in Progress, August 2004. [RFC-STYLE] Braden, R., Ginoza, S., and A. Hagens, "RFC Document Style", September 2009, <http://www.rfc-editor.org/rfc-style-guide/rfc-style>. [RFC3426] Floyd, S., "General Architectural and Policy Considerations", RFC 3426, November 2002.
[RFC3439] Bush, R. and D. Meyer, "Some Internet Architectural Guidelines and Philosophy", RFC 3439, December 2002. [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC Text on Security Considerations", BCP 72, RFC 3552, July 2003. [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008. [SIRS] Carpenter, B. and D. Crocker, "Careful Additional Review of Documents (CARD) by Senior IETF Reviewers (SIRS)", Work in Progress, June 2003.