AD/AE Appendices • SC24

A LaTeX template for the AD/AE Appendix and further information can be found on GitHub:

SC24 Repro on Github

AD Appendix

The AD Appendix should contain the following two parts:

Part 1: Overview of Contributions & Artifacts

A. Paper’s Main Contributions

Provide a list of paper’s contributions for which computational artifacts are created.

B. Computational Artifacts

Provide a table with the relevant computational artifacts, highlight their relation to the contributions (from above) and point to the elements in the paper that will be reproduced (figures, tables).

Part 2: Artifact Identification

For each computational artifact, provide the following information:

A. Relation to Contributions

Briefly explain the relationship between the artifact and contributions.

B. Expected Results

Clearly state what the expected result should be after reproducing the artifact.
Provide an explanation of how the results substantiate the main contributions.

C. Expected Reproduction Time (in minutes)

Estimate the time required to reproduce the artifact, providing separate estimates for the individual steps: Artifact Setup, Artifact Execution, and Artifact Analysis.

D. Artifact Setup

Hardware

Specify the hardware requirements and dependencies (e.g., a specific interconnect or GPU type is required).

Software

Introduce all required software packages, including the computational artifact. For each software package, specify the version and provide the URL.

Datasets/Input

Describe the datasets required by the artifact. Indicate whether the datasets can be generated, including instructions, or if they are available for download, providing the corresponding URL.

Installation & Deployment

Detail the requirements for compiling, deploying, and executing the experiments, including necessary compilers and their versions.

E. Artifact Evaluation

Provide an abstract description of the experiment workflow of the artifact. It is important to identify the main tasks (processes) and how they depend on each other.
A workflow may consist of three tasks: T1, T2, and T3. The task T1 may generate a specific dataset. This dataset is then used as input by a computational task T2, and the output of T2 is processed by another task T3, which produces the final results (e.g., plots, tables, etc.). State the individual tasks Ti and provide their dependencies, e.g., T1 -> T2 -> T3.
Provide details on the experimental parameters. How and why were parameters set to a specific value (if relevant for the reproduction of an artifact), e.g., size of dataset, number of data points, input sizes, etc. Additionally, include details on statistical parameters, like the number of repetitions.

F. Artifact Analysis

Outline the general data analysis process, where raw experimental results are often processed further to yield the final experimental outcomes, such as plots. Briefly detail the data processing steps involved at this stage.

AE Appendix

The AE Appendix’s objective is to enable other researchers and reviewers to replicate the findings presented by the computational artifacts. The AD Appendix comprehensively details the required software and hardware for result reproduction. The AE Appendix’s distinct role is to elucidate how to compile, operate, and analyze the components specified in the AD Appendix. In other words, while the AD Appendix introduces and details the fundamental components of the computational artifacts, the AE Appendix offers detailed guidelines for constructing, executing, and analyzing these artifacts.

The authors should provide the following information for each computational artifact defined in the AD Appendix:

A. Artifact Setup

Provide instructions for installing and compiling libraries and code.
Offer guidelines on deploying the code to resources.

B. Artifact Execution

Describe the experiment workflow.
If encapsulated within a workflow description or equivalent (such as a makefile or script), clearly outline the primary tasks and their interdependencies. Detail the main steps in the workflow. Merely instructing to “Run script.sh” is inadequate.

C. Artifact Analysis

Provide a description of the expected results and a methodology for evaluating these results.
Explain how the expected results from the experiment workflow correlate with the contributions stated in the article.
For example, if the article presents results in a figure, the artifact evaluation should also produce a similar figure, depicting the same generalizable outcome. Authors must focus on these aspects to reduce the time required for others to understand and verify an artifact.

AD/AE PRocess & Badges

Our aim is to aid SC authors in submitting the required documentation describing their artifact, facilitating our evaluation and subsequent badge assignment for the artifact.