Information for authors of papers that we would like to reproduce

This page contains information for authors of papers that we wish to include in our multi-lab study. We have sent emails to lead authors of all such papers. We need to select 15-20 papers to be reproduced and each selected paper will have one or more of its human evaluation experiments reproduced by two of the 22 partner labs that will be running reproduction experiments as part of the ReproHum multi-lab study.

To identify candidate papers for inclusion in our study, we created a shortlist of papers from both ACL and TACL proceedings between 2018 and 2022 that include human evaluations of NLP system outputs.

We will be reproducing only the human evaluations; we will not be recreating systems. Therefore, we require the following information from the original authors before we can include the paper in our study:

  1. The system outputs that were shown to participants (i.e., a spreadsheet or other files containing the outputs that were shown).
  2. The interface, form, or document that participants completed; the exact document or form that was used would be ideal, although we can recreate the interface if provided with details.
  3. Details on the number and type of participants (students, researchers, Mechanical Turk, etc.) that took part in the study.
  4. The total cost of the original human evaluation(s), or a rough estimate if you do not have the exact figure.

Some of this information might already be in your paper or supplementary materials, although when reading the papers we found that one or more pieces of information was not available in most cases. Therefore, we would be most grateful if you could provide or confirm the above. We do not require full details immediately in order to continue considering a paper for our study, but we do need to know as soon as possible if the information can be made available at all, and if so, how long that might take.

We greatly appreciate your time reading this, and it would be very helpful if you could let us know whether you have the above information. Alternatively, it would also be helpful if you can forward the email that we sent you about the paper to another author who might have the information.

If you have any questions, please contact us.

Thank you.