Spear: Client-Guided Test Generation for C Libraries

This document describes how to use the pre-configured Docker images provided for each library to reproduce the evaluation results from the paper. Overview For each library evaluated in the paper (except LMDB due to size of the image and difficulty uploading), we provide Docker images containing: - The library and all client applications used in the evaluation - Pre-built bitcode files for each client - All necessary scripts to reproduce results - Complete experiment results and coverage reports All libraries and clients are located in `/tmp/` within each Docker container. We also provide all prompts we used in our pipeline in prompts.tar A comprehensive README.md is included to provide instructions on how to setup a new library-client pair. Note: LMDB Artifact Format For LMDB, we do not provide a pre-built Docker container. Instead, we provide a `lmdb_results.tar` archive containing: - Pre-built client source code directories - Pre-extracted and inlined bitcode files - The complete Spear framework - All the results discussed in the paper under Spear/experiments/lmdb/. - A comprehensive `README.md` with instructions To reproduce LMDB results: 1. Extract the `lmdb_results.tar` archive 2. Follow the instructions in the included `README.md` to create your own Docker image 3. The README provides complete Dockerfile examples and step-by-step instructions The `lmdb_results.tar` archive includes everything needed to build a Docker container and reproduce the Spear evaluation results for LMDB. Loading Docker Images Step 1: Download and Load the Image Download the Docker image for your library and load it: gunzip -c .tar.gz | docker load Step 2: Access the Container Get terminal access to the Docker container: docker run -it --rm /bin/bash Step 3: Set Up Environment Variables Once inside the container, set the required environment variables: Set Claude API key (required for test case generation) export ANTHROPIC_API_KEY= These are typically already set in the image, but verify: export SPEAR_DIR=/tmp/Spear/src export CLANG_DIR=/usr/lib/llvm-16 Exploring Results Experiment Results Navigate to the experiment results directory: cd /tmp/Spear/experiments/ / This directory contains results from the paper evaluation. Each client has a folder named using the pattern: - `output_ _ ` - Full execution results from Stage 1 onwards - Example: `output_curl_4Jan` means client `curl` run on January 4th Understanding Output Folders - `output_ _ `: Complete results from Stage 1 resulting in the sliced test cases - `output_ _ post`: Results from Stage 2 onwards (Stage 1 was already completed) - Under the `output_ _ post` directory the generated tests can be seen in `success_tests`. These are all tests pre-validation. Inside that folder there is another folder called `verified` which includes all the tests that have been verified to execute manually. - `validations/`: Test cases instrumented with print statements to validate that the target API is executed. - Used to collect metrics `tAPIs` reported in the paper - Contains test cases that have been verified to exercise the target APIs Coverage Reports All coverage reports are located in: cd /tmp/Spear/experiments/ /coverage_reports These reports show code coverage metrics for the generated test cases. - all.info/final.info these are coverage reports for Claude Code and Spear combined. - all_spear.info this is the LCOV coverage report after executing all the tests in generated by Spear in addition to the test suite of the library. - baseline_with_claude.info this is the LCOV coverage report after executing all the tests generated by Claude Code in addition to the test suite of the library. Claude Code Generated Tests You can find all the claude code generated tests under Spear/experiments/ /claude_tests Re-running Experiments Running Full Experiments To re-run an experiment for a specific client: 1. Navigate to the library data directory: cd /tmp/Spear/experiments/ /lib_data 2. Find the execution script for your client: - Scripts are named `execute_ .sh` - Example: `execute_curl.sh` for the curl client 3. Run the script: ./execute_curl.sh Running Stage 2 Only If Stage 1 has already completed and you want to re-run only Stage 2 onwards: 1. Use the post-cleanup scripts: - Scripts are named `execute_post_cleanup_ .sh` - Example: `execute_post_cleanup_curl.sh` 2. Edit and run: # Edit the --out-dir parameter if needed ./execute_post_cleanup_curl.sh These scripts skip Stage 1 and start from Stage 2 (API Sequence Initialisation). Available Data in Docker Images Each Docker image contains the following: Source Code - Libraries: Cloned/downloaded in `/tmp/` - Clients: All client applications used in evaluation, located in `/tmp/` Bitcode Files For each client, you'll find: - `/tmp/ _bc_files/`: Extracted bitcode files - `/tmp/ _bc_files_inlined/`: Inlined bitcode files used for analysis Example: - `/tmp/curl_bc_files/` - Extracted bitcode from curl - `/tmp/curl_bc_files_inlined/` - Inlined bitcode for curl Scripts - Execution scripts: Located in `/tmp/Spear/experiments/ /lib_data/` - `execute_ .sh` - Full execution scripts - `execute_post_cleanup_ .sh` - Stage 2+ only scripts - Helper scripts: Located in `/tmp/Spear/scripts/` - See `benchmark_setup.md` for instructions on setting up new library-client pairs Setting Up New Library-Client Pairs To set up a new library-client pair within the Docker environment follow the instructions provided in the README.md file LMDB Artifact (Special Format) For LMDB, we do not provide a pre-built Docker container. Instead, we provide a `lmdb_results.tar` archive. Using the LMDB Artifact 1. Extract the archive: tar -xf lmdb_results.tar cd lmdb_results A- To build a docker image: Read the included README The archive contains a comprehensive `README.md` with detailed instructions on: - Creating a Docker image from the artifact - Copying all directories to `/tmp` in the container - Installing dependencies from `requirements.txt` - Building and running the container - Executing experiments B- To explore the results discussed in the paper please see the results under Spear/experiments/lmdb. The folder structure follows the same structure explained above under Exploring Results. Claude Prompts Below are the prompts used with Claude Code in the different modes of execution. Claude Read the file apis_to_test.txt which contains APIs from {Library}. Generate tests for those APIs and make sure they compile and run. Create those tests in a separate folder called claude_tests so they could be run independently of the test suite Claude with Documentation Read the file apis_to_test.txt which contains APIs from {Library}.Generate tests for those APIs and make sure they compile and run. Create those tests in a separate folder called claude_tests so they could be run independently of the test suite. In the folder experiments/documentation you will have a JSON file which shows the documentation for all APIs of {Library}. Use the documentation to help you construct tests that invoke the APIs correctly. Claude with Apisequences Read the file apis_to_test.txt which contains APIs from {Library}.Generate tests for those APIs and make sure they compile and run. Create those tests in a separate folder called claude_tests so they could be run independently of the test suite. In the folder experiments/apisequences you will multiple subdirectories labeled by clients that use {Library}. Each subdirectory contains a JSON file that shows API sequences extracted from each client. Use those API sequences to construct the tests for the APIs in apis_to_test.txt so the invocation is correct. Claude with Sliced Tests Read the file apis_to_test.txt which contains APIs from {Library}.Generate tests for those APIs and make sure they compile and run. Create those tests in a separate folder called claude_tests so they could be run independently of the test suite. In the folder experiments/sliced_tests you will multiple subdirectories labeled by clients that use {Library}. Each subdirectory contains sliced tests from the clients that show how the APIs were used by the clients. Use those sliced tests to construct the tests for the APIs in apis_to_test.txt so the invocation is correct. Claude with all Read the file apis_to_test.txt which contains APIs from {Library}.Generate tests for those APIs and make sure they compile and run. Create those tests in a separate folder called claude_tests so they could be run independently of the test suite. In the folder experiments/apisequences you will multiple subdirectories labeled by clients that use {Library}. Each subdirectory contains a JSON file that shows API sequences extracted from each client. Use those API sequences to construct the tests for the APIs in apis_to_test.txt so the invocation is correct. In the folder experiments/documentation you will have a JSON file which shows the documentation for all APIs of {Library}. Use the documentation to help you construct tests that invoke the APIs correctly. In the folder experiments/sliced_tests you will multiple subdirectories labeled by clients that use {Library}. Each subdirectory contains sliced tests from the clients that show how the APIs were used by the clients. Use those sliced tests to construct the tests for the APIs in apis_to_test.txt so the invocation is correct.