AD/AE Process & Badges

Guidelines for artifact providers

Please carefully read the guidelines when compiling the AE Appendix.

Important Disclaimer

The Reproducibility Initiative works in a best effort manner. Please note that personnel and computational resources are limited. Sometimes it is simply not possible to run a given experiment at a certain time, due to unavailability of specialized hardware such as GPUs during the artifact evaluation period, unpredictable maintenance times of compute centers, or individual time constraints of reviewers (time zones, etc.). Nonetheless, the Reproducibility Initiative attempts to fairly judge the computational artifacts of SC papers that demand reproducibility badges.

AE Time Budget

Each AE Appendix is allocated an 8-hour review budget. This means artifact reviewers are expected to dedicate this amount of time to result reproduction. When preparing their computational artifacts, artifact providers should be mindful of this time constraint. It is crucial to note that the 8-hour budget pertains to the net time required for reproducing results. For instance, if artifact reviewers have to wait several hours for access to a shared hardware infrastructure, like Chameleon Cloud, these waiting periods are not counted within the allocated budget.

Computational Resources

In order to evaluate the computational artifacts provided by the authors, reviewers rely on the availability of computational resources, such as large-scale compute clusters or dedicated GPUs. 

Chameleon Cloud supports the SC24 Reproducibility Initiative by providing computational resources for authors and reviewers. 

In principal, there are three distinct scenarios for the AE:

1. (Preferred) The authors (artifact providers) create the computational artifacts in a downscaled manner on Chameleon Cloud, e.g., use less compute nodes or simplified datasets/instances to reduce reproduction times. Now, authors and reviewers will have access to the same computational resources, and thus, an artifact evaluation has the highest chance of being successful. It is important to note that while a downscaled experiment, i.e., reducing the number of resources and computational time required, allows reviewers to assess the functionality of an artifact, a RESULTS REPLICATED badge cannot be granted if the principal results in the paper were achieved from experiments requiring an order of magnitude greater in resources and time.
2. Computational artifacts cannot or should not be executed on Chameleon Cloud, as specialized hardware or software is required. Now, there are two sub-cases:
   (a) The authors can provide access to the specific computational resources for the reviewers. In this case, reviewers should be actively assisted to get accounts on the respective machines in a timely manner.
   (b) The authors do not or cannot provide access to computational resources needed for the artifact evaluation. Under these circumstances, the artifact reviewers will attempt to meet the requirements. However, the likelihood of obtaining the RESULTS REPLICATED badge will be diminished.
3. (Optional) This year’s initiative proposes the optional use of Guix, a software tool designed to support reproducible software deployment. Guix allows deployment of the exact same software environment on different machines and at different points in time, while still retaining complete provenance info. By eliminating almost entirely variability induced by the software environment, Guix gives authors and reviewers more confidence into the results of computational experiments. Interested authors will find more information at (AE) Alternative with Guix.

Please note that deployments on commercial cloud infrastructures or any other platform requiring reviewers to incur monetary costs for reproducing experiments are not allowed, as such practices can create barriers for potential reviewers and for future third-party researchers interested in the artifacts. If you submit artifacts that violate this requirement, the only badge you may apply for is ARTIFACTS AVAILABLE.

Granularity of AE Description

Since public, open software and data repositories such as Zenodo, DataPort, Dryad, FigShare, or Harvard Dataverse are explicitly encouraged to be used for archiving artifacts, we would like to clarify where to put the information on how to build and execute experiments, i.e., should the information be put into the AE Appendix or into READMEs on data repositories. 

We recommend that authors put all instructions required to reproduce experiments in the AE Appendix, i.e., simply referring to READMEs on external repositories such as Zenodo should be avoided.

For the sake of readability, the detailed nature of build and run instructions might be very excessive for an AE Appendix. Please think about the right level of granularity. For example, it may suffice to state that a script like “compile.sh” needs to be called on machine X, while “compile.sh” itself may contain hundreds of individual steps.

If authors would like to provide a README in the data repositories (e.g., Zenodo), we recommend to provide only one single, comprehensive top-level README file, which should encompass all necessary information that needs to be conveyed.

artifacts available

The following are necessary to receive this badge:

• Assigned DOI to your research object by the Artifact Freeze deadline. DOIs can be acquired via Zenodo, FigShare, Dryad, Software Heritage. Zenodo provides an integration with Github to automatically generate DOIs from Git tags.
• Links to code and data repositories on a hosting platform that supports versioning: GitHub, or GitLab. Please do NOT provide Dropbox links or gzipped files hosted through personal webpages.

Note that, for physical objects relevant to the research, the metadata about the object should be made available.

What do we mean by accessible? Artifacts used in the research (including data and code) are permanently archived in a public repository that assigns a global identifier and guarantees persistence, and are made available via standard open licenses that maximize artifact availability.

Artifacts Evaluated-Functional

The criteria for the Artifacts Evaluated-Functional badge require an AD/AE committee member to agree whether the artifact provides enough details to exercise the artifact of components in the paper. For example, is it possible to compile the artifact, use a Makefile, or perform a small run? If the artifact runs on a large cluster—can it be compiled on a single machine? Can analysis be run on a small scale? Does the artifact describe the components to nurture future use of this artifact?

The reviewer will assess the details of the research artifact based on the following criteria:

• Documentation: Are the artifacts sufficiently documented to enable them to be exercised by readers of the paper?
• Completeness: Do the submitted artifacts include all of the key components described in the paper?
• Exercisability: Do the submitted artifacts include the scripts and data needed to run the experiments described in the paper, and can the software be successfully executed?

We encourage authors to describe their (i) workflow underlying the paper, (ii) describing some of the black boxes, or a white box (e.g., source, configuration files, build environment), (iii) input data: either the process to generate the input data should be made available, or when the data is not generated, the actual data itself or a link to the data should be provided, (iv) environment (system configuration and initialization, scripts, workload, measurement protocol) used to produce the raw experimental data, and (v) the scripts needed to transform the raw data into the graphs included in the paper.

Results Replicated

The evaluators successfully reproduced the key computational results using the author-created research objects, methods, code, and conditions of analysis. Note we do not aim to recreate the exact or identical results, especially hardware-based results. However, we do aim to:

• Reproduce Behavior: This is of specific importance where results are hardware-dependent. Bitwise reproducibility is not our goal. If we get access to the same hardware as used by experiments, we will aim to reproduce the results on that hardware. If not, we aim to work with authors to determine the equivalent or approximate behavior on available hardware. For example, if results are about response time, our objective will be to check if a given algorithm is significantly faster than another one, or that a given parameter affects negatively or positively the behavior of a system.
• Reproduce the Central Results and Claims of the Paper: We do not aim to reproduce all results and claims shown in the paper. The AD/AE committee will evaluate all artifacts that the authors have declared as supporting the central results of the accepted paper.