A Survey of Web Application Security Tutorials

A Survey of Web Application Security Tutorials: Replication Package Bhagya Chembakottu and Martin P. Robillard School of Computer Science, McGill University, Montréal, Canada Abstract This replication package provides the complete infrastructure to reproduce the study "A Survey of Web Application Security Tutorials" submitted in IEEE Software. The package includes automated scripts for data collection, ranking algorithms, text extraction, and statistical analysis. The replication can be executed on Windows, macOS, and Linux systems with minimal setup requirements. 1. Introduction This replication package enables researchers and practitioners to reproduce our study on web application security tutorials. The package implements the complete methodology described in the paper, including query-based data collection, rank summation algorithms, manual coding guidelines, and statistical analysis procedures. More details of setting up sample replication study can be found [here](./documentation/methodology.md) The replication is designed to be executed with minimal technical expertise, providing both graphical and command-line interfaces for different user preferences. 2. Package Overview 2.1 Directory Structure ``` replication-package/ ├── README.md # This file ├── requirements.txt # Python dependencies ├── run-replication.sh # Unix/Linux/macOS script ├── run-replication.bat # Windows batch script ├── config.yaml # Configuration parameters ├── data/ # Data storage │ ├── query-set-sample/ # Sample query results │ └── final-dataset/ # Final coded dataset ├── scripts/ # Analysis scripts │ ├── 01-extract-queries.py │ ├── 02-rank-summation.py │ ├── 03-text-extraction.py │ ├── 04-statistical-analysis.py │ └── utils/ ├── documentation/ │ ├── coding-guide.md │ └── methodology.md └── results/ # Output files ``` 2.2 System Requirements - Python 3.7 or higher - 2GB available disk space - Internet connection (for initial data collection) - Windows 10/11, macOS 10.14+, or Linux (Ubuntu 18.04+) 3 Installation 3.1 Automated Installation (Recommended) Windows Double-click installation setup-windows.bat 3.2 Manual Installation Install Python dependencies: pip install -r requirements.txt 4 Execution 4.1 Windows Execution Option 1: Batch File (Recommended for beginners) Double-click in File Explorer run-replication.bat Or command prompt run-replication.bat > Note : If any configuration issues try to run `fix-windows-issues.bat` Option 3: Manual Command Line Step-by-step execution - python scripts\01-extract-queries-sample.py --output data\query-set-sample - python scripts\02-rank-summation.py --input-dir data\query-set-sample --output results\top-25-urls.txt - python scripts\03-text-extraction.py --input data\final-dataset\security-tutorials-sample.csv --output results\text-lengths-sample.csv --simulate - python scripts\04-statistical-analysis.py --input results\text-lengths-sample.csv --output results\statistics-report-sample.txt 4.2 Unix/Linux/macOS Execution Make executable and run chmod +x run-replication.sh ./run-replication.sh Option 2: Manual Execution Step-by-step execution - python scripts/01-extract-queries-sample.py --output data/query-set-sample - python scripts/02-rank-summation.py --input-dir data/query-set-sample --output results/top-25-urls.txt - python scripts/03-text-extraction.py --input data/final-dataset/security-tutorials-sample.csv --output results/text-lengths-sample.csv --simulate - python scripts/04-statistical-analysis.py --input results/text-lengths-sample.csv --output results/statistics-report-sample.txt 5 Configuration Edit `config.yaml` to customize: ```yaml # Analysis parameters sample_settings: tutorial_count: 25 statistical_significance: 0.05 bonferroni_corrections: 4 # File paths (auto-adjusted for OS) paths: windows_separator: "\" unix_separator: "/" query_pattern: "query_*.txt" # Execution settings execution: delay_between_requests: 2 # seconds timeout: 15 # seconds max_retries: 3 ``` 6 Expected Outputs 6.1 Primary Results - **Top 25 URLs**: `results/top-25-urls.txt` - **Text lengths**: `results/text-lengths-sample.csv` - **Statistical report**: `results/statistics-report-sample.txt` 6.2 Sample Statistical Output ``` Statistical Analysis Results - Sample Dataset ============================================================ Sample size: 25 tutorials Bonferroni corrected alpha: 0.0125 Tutorial Types: Vendor Learning Resource: 18 (72.0%) Community / Education Project: 7 (28.0%) Code Example Presence: none: 20 (80.0%) snippet: 3 (12.0%) full working code: 2 (8.0%) Mann-Whitney U (Code vs No Code): U-statistic: 45.5 p-value: 0.042 Significant (α=0.0125): No ``` 7 Troubleshooting 7.1 Common Issues | Issue | Windows Solution | Unix/Linux/macOS Solution | |-------|------------------|---------------------------| | Python not found | Install Python and check PATH | `which python` or install via package manager | | Permission denied | Run as Administrator | `chmod +x script.sh` | | Package installation fails | `pip install --user package` | `pip install --user package` or use `sudo` | | Network timeouts | Increase timeout in config.yaml | Same as Windows | | Character encoding errors | Set `chcp 65001` in CMD | Set `export PYTHONIOENCODING=utf-8` | 7.2 Platform-Specific Debugging Windows Debugging ```batch # Check Python installation where python python --version # Check package installation pip list | findstr pandas # Run with verbose output python -v scripts\01-extract-queries-sample.py --output data\query-set-sample ``` Unix/Linux/macOS Debugging ```bash # Check Python installation which python python --version # Check package installation pip list | grep pandas # Run with verbose output python -v scripts/01-extract-queries-sample.py --output data/query-set-sample ``` 9 Validation 9.2 Expected Validation Results ```bash # Validate installation python -c "import pandas, numpy, scipy; print('Core packages OK')" # Validate data integrity python -c "import pandas as pd; df = pd.read_csv('data/final-dataset/security-tutorials-sample.csv'); print(f'Sample size: {len(df)}')" # Validate statistical output python -c "with open('results/statistics-report-sample.txt') as f: content = f.read(); print('Statistical report generated' if 'Mann-Whitney U' in content else 'Missing statistics')" ``` 10. Full Dataset Replication To replicate with the full 132-tutorial dataset: 1. Modify `config.yaml` to use full query set 2. Run with original query parameters 3. Apply manual coding to top 200 URLs 4. Execute full statistical analysis