Goals

In this tutorial, we will get basic understand of BugSwarm CLI and BugSwarm Artifacts.

There are two parts in this experiment:

Part 1
    - select an artifact with the Dataset Browser
    - install the BugSwarm client
    - download the artifact with the client
    - Reproducing the `failed/passed` job

Part 2 (Optional)
    - run a code coverage tool inside the artifact

Part 3 (Optional)
    - run a bug finding tool inside the artifact

Requirements

This tutorial requires no prior knowledge of BugSwarm. You will need Docker installed on your machine.

Part 1 (Video Demo)

Selecting an artifact

Navigate to the Dataset Browser to view metadata for every existing artifact. Click on any column header to sort the metadata. Type in the "Filter column" text fields to change the displayed metadata based on project name, language, build system, test framework, number of tests ran or failed, and several other attributes.

In this tutorial, we'll use an artifact from the square/okio project.

Type the image tag square-okio-140452393 into the Image Tag column filter to display metadata for the specific artifact identified by that image tag.

An image tag uniquely identifies a BugSwarm artifact.

The Dataset Browser is useful for selecting a small number of artifacts. To facilitate artifact selection at scale, we expose a (soon to be public) REST API for querying the artifact database based on any of the metadata attributes.

Installing the BugSwarm client

Now that we've selected an image tag and viewed its metadata, we can use the client to download a local copy of the corresponding artifact Docker image. There are two ways to install the client.

From PyPI:

$ pip3 install bugswarm-client

From source:

$ git clone https://github.com/BugSwarm/bugswarm.git
$ cd bugswarm
$ cp setup.client.py setup.py
$ pip3 install --user .

Downloading an artifact

To download the artifact we chose earlier, you'll need the image tag (square-okio-140452393). Make sure the Docker daemon is running, and then run the following command. Depending on how Docker is configured on your machine, you may need to enter an administrator password.

$ bugswarm run --image-tag square-okio-140452393 --use-sandbox

See the BugSwarm CLI documentation for a longer description of the client API.

The above command will download all the layers of the Docker image, initialize an artifact Docker container from the layers, and then provide an interactive shell in the artifact container.

It may take a few minutes to download your first artifact, depending on your network connection. Thanks to the properties of Docker's layered file system, all subsequent artifact downloads will be much faster.

Interacting with an artifact

Now that we're "inside" an artifact container, we can view the source code, modify files, install tools, etc.

Also, since we specified the --use-sandbox flag when invoking bugswarm run, we have a shared "sandbox" directory that is accessible from within the artifact container and from the host machine. The sandbox directory is a convenient way to move files into and out of an artifact container.

Reproducing failed and passed Travis jobs

At this point, we could reproduce the failed or passed Travis jobs by invoking a script included with the artifact.

Reproducing the failed job

$ run_failed.sh

It might take a few minutes to setting up the project and finish the CI testing.

At the end you will see exited with 1, which indicate job failed.

Restart the container

$ exit

Then start with a fresh new container

$ bugswarm run --image-tag square-okio-140452393 --use-sandbox

We should not consecutively run the failed and passed script because running failed script can pollute the container since failed and passed script may have different ways of setting up projects or different dependencies.

Reproducing the passed job

$ run_passed.sh

At the end you will see exited with 0, which indicate the job passed.

Notes:

• Make sure your are running the failed/passed "inside" the container.
• See Anatomy of an Artifact to learn more about the contents of an artifact.

Video Demo

<iframe width="560" height="315" src="https://www.youtube.com/embed/i6fWb1CotS0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Part 2 (Optional)

Installing a code coverage tool

Let's install a code coverage tool to be invoked when the project is built and tested.

Namely, let's install Cobertura.

First, cd to the version of the project that resulted in the failed Travis job.

$ cd ~/build/failed/square/okio/

Next, modify the POM file that holds configuration instructions for the Maven build system.

$ vim pom.xml

Jump to the end of the build section by typing ?/build and then Enter. Insert the following snippet right before the nearest closing </plugins> tag.

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>cobertura-maven-plugin</artifactId>
    <version>2.7</version>
    <configuration>
        <formats>
            <format>html</format>
            <format>xml</format>
        </formats>
    </configuration>
</plugin>

Hit Esc. Then jump to the end of the build section by typing ?/build and then Enter. Insert the following snippet right after the closing </build> tag.

<reporting>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <version>2.7</version>
            <configuration>
                <check></check>
                <formats>
                    <format>html</format>
                    <format>xml</format>
                </formats>
            </configuration>
        </plugin>
    </plugins>
</reporting>

Hit Esc. Then save and close the file by typing :wq and then Enter.

Now invoke Maven to begin the build and test process, which now includes running Cobertura and producing a machine-readable code coverage report.

$ /usr/local/maven-3.2.5/bin/mvn cobertura:cobertura -Dcobertura.report.format=xml -Dhttps.protocols=TLSv1.2

Copy the report to the sandbox so it can be viewed later after exiting the artifact container.

$ cd ~/build/failed/square/okio/okio/target/site/cobertura
$ cp coverage.xml /bugswarm-sandbox

Exit the artifact container.

$ exit

The code coverage report is available on your machine at ~/bugswarm-sandbox/coverage.xml. You could perform further analysis on the report, but for now we'll just view it.

$ vim ~/bugswarm-sandbox/coverage.xml

Part 3 (Optional)

Installing Static Bug Detectors

If we wish to install a static bug detector that is used during the testing of your selected artifact, we will follow similar steps to what was done in Part 2. We will include steps to install the following tools:

• SpotBugs
• Infer
• NullAway
• Eradicate
• Checker Framework

SpotBugs, NullAway, Checker Framework

For these three tools, we simply just need to modify the POM file for the relevant artifacts. First, cd to the version of the project that resulted in the failed Travis job. Note: If you were running the artifact in a container in previous steps, you must perform the following steps within the container as well.

$ cd ~/build/failed/square/okio/

Next, modify the POM file that holds configuration instructions for the Maven build system. Modify the POM file according to which tool that you wish to use. Note: All further commands are vi/vim specific. You may choose the text editor of your choice if you wish.

$ vim pom.xml

SpotBugs

Jump to the end of the build section by typing ?/build and then Enter. Insert the following snippet right before the nearest closing </plugins> tag. If no </plugins> tag exists, you may need to create one. We have currently only tested this process with version 3.1.6, you may attempt to use other versions is you wish. If you attempt to use other versions, please check the website to make sure that there are no dependency or environment issues. Note: You may select either low or high for your threshold, the example below is using a low threhsold.

<plugin>
    <groupId>com.github.spotbugs</groupId>
    <artifactId>spotbugs-maven-plugin</artifactId>
    <version>3.1.6</version>
    <configuration>
        <effort>Max</effort>
        <threshold>low</threshold>
    </configuration>
</plugin>

Hit Esc. Then jump to the end of the build section by typing ?/build and then Enter. Insert the following snippet right after the closing </build> tag. Note: You may select either low or high for your threshold, the example below is using a low threhsold.

<reporting>
    <plugins>
        <plugin>
            <groupId>com.github.spotbugs</groupId>
            <artifactId>spotbugs-maven-plugin</artifactId>
            <version>3.1.6</version>
            <configuration>
                <effort>Max</effort>
                <threshold>low</threshold>
            </configuration>
        </plugin>
    </plugins>
</reporting>

Hit Esc. Then save and close the file by typing :wq and then Enter.

Now invoke Maven to begin the build and test process, which now includes running SpotBugs to generate a bug report. Spotbugs requires class files for analysis, which is why the compile command is included. If no class files exist for the project when you attempt to run Spotbugs, no report will be generated.

$ /usr/local/maven-3.2.5/bin/mvn compile com.github.spotbugs:spotbugs-maven-plugin:3.1.6:spotbugs -Dhttps.protocols=TLSv1.2

Note: SpotBugs requires Java 8 or newer, meaning that you should ensure that the Java version is correct for your artifact when attempting to run SpotBugs. If your Java version is not set to 8, you can typically just assign JAVA_HOME to the directory where it is located. Example for artifact square-okio-140452393:

$ JAVA_HOME="/usr/lib/jvm/java-8-oracle/" && /usr/local/maven-3.2.5/bin/mvn compile com.github.spotbugs:spotbugs-maven-plugin:3.1.6:spotbugs -Dhttps.protocols=TLSv1.2

Copy the report to the sandbox so it can be viewed later after exiting the artifact container.

$ cd ~/build/failed/square/okio/okio/target/
$ cp spotbugsXml.xml /bugswarm-sandbox

Exit the artifact container.

$ exit

Automating tool installation

In our experience, running an experiment commonly contains three main steps:

1. copy necessary scripts or executables into the artifact container
2. invoke the scripts or executables inside the artifact container
3. copy any produced files out of the artifact container via the sandbox directory.

Although we created a (soon to be public) framework to automate this workflow, which is available upon request, we performed the installation manually during this walkthrough for the sake of clarity.