EMNLP 2026 AI Reviewing Experiment

EMNLP 2026 is running an AI Reviewing Experiment to collect feedback from authors about the quality of AI reviews of their submissions. This experiment is taking place on an opt-in basis, in which authors need to declare whether they agree to take part in the experiment. In this blog, we describe the systems that will be used in the experiment, outline the design of the experiment, and explain how you can opt-in / withdraw your consent after ARR submission.

AI Review Systems

We are collaborating with two research groups to provide AI-generated reviews for opted-in submissions.

The ReviewerToo system is developed by researchers from Mila - Quebec AI Institute, ServiceNow Research, HEC Montréal, Polytechnique Montréal, Université de Montréal, and Unified Sciences. The system uses the GPT-OSS-120B language model hosted in inference mode on a dedicated server, and generates reviews through a multi-stage workflow where the agents are grounded in prior work via literature search. To ensure data integrity and confidentiality, no experimental data shall be utilized for training purposes or disclosed to any third party. Deployment and operating costs will be covered by Unified Sciences.

The AAAI AI Review System was developed for the AAAI-26 AI Review Pilot by a multi-institutional research team consisting of researchers from the University of Texas at Austin, the University of Alberta, the University of Michigan, and Oregon State University. The system uses OpenAI’s GPT-5.4 model under a Zero Data Retention agreement, and generates reviews through a custom multi-stage workflow. For EMNLP, deployment and operation will be handled by the University of Texas at Austin. OpenAI API usage will be covered through API credits provided for this purpose.

Experimental Design

The AI review experiment is designed to collect feedback from the EMNLP community about the strengths and weaknesses of the tested AI Review systems, and the expected benefits or drawbacks of using AI reviewing tools, in general.

The AI reviews will initially only be visible to the authors of opted-in submissions and the AI Review Chairs. Each submission will be randomly allocated to one of the AI review systems. The reviewers, Area Chairs, and Senior Area Chairs will not be able to see the AI review of any submission during the reviewing and decision making process. This is in contrast to other recent experiments in the community, e.g. ICLR 2025 and AAAI 2026, in which AI tools have played a more active role in the review process.

The AI reviews will be posted directly to the OpenReview page of the opted-in submissions after the end of the Author Response period of the ARR May 2026 cycle. It will then be possible for opted-in authors to view the AI review for their submission and complete a survey to share their thoughts about the accuracy and relevance of their AI review. The timing has been chosen to maximise the potential for survey response, while removing the opportunity for authors to use information in the AI reviews when drafting their author response.

The content and metadata of the opted-in submissions will not be stored after the end of the experiment.

AI Review Content

The AI Reviews will be written to mirror the standard format of ARR reviews. The AI review will not provide a direct accept/reject recommendation or numerical scores. The AI review will be clearly marked as generated by AI. You can expect to read the following content:

  • Overall summary of the paper
  • Strengths
  • Weaknesses
  • Discussion of related work
  • Clarification questions

Participant Survey

We will use a participant survey as the primary means of collecting information from authors. This survey will either be available directly in OpenReview, or hosted via a secure third-party site. The survey will contain a combination of Likert-scale, forced-choice, and free-form response questions. A limited number of free-form responses may be used in public presentations about the survey.

Opt-in and Withdrawal Process

Authors can opt-in to the AI Reviewing Experiment for each submission during ARR submission. If you opt-in at the time of submission, and later wish to withdraw from the experiment at any time after submission, you can send an email to emnlp2026-ai-review-chairs@googlegroups.com. Please write [Submission XXX Withdraw Consent] in the title of your email.

Institutional Review Board Determination

This experiment is being conducted under IRB determination from the University of Texas at Austin. IRB Protocol #: STUDY00007931.

Contact Information

General Chair:

  • André Martins, Instituto Superior Técnico, Universidade de Lisboa

Program Chairs:

  • Sunipa Dev, Google
  • Desmond Elliott, University of Copenhagen
  • Hung-yi Lee, National Taiwan University
  • Jessy Li, The University of Texas at Austin

AI Reviewing Chairs:

  • Joydeep Biswas, University of Texas at Austin
  • Gaurav Sahu, Mila - Quebec AI Insitute

For questions about the AI reviewing experiment, email: emnlp2026-ai-review-chairs@googlegroups.com

For questions about EMNLP 2026 commitment and other review related topics, email: emnlp2026-programchairs@googlegroups.com

Updated: