Replication Package of Beyond Lines of Code: Probing the Latent Representation of Magnitude-Based Code Smells in Large Language Models

Read full text Summarise with VIRAT
Authors: anonymous, anonymous
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Replication Package⚠️⚠️⚠️ Description truncated due to length constraint. Please refer to README.md for detail.⚠️⚠️⚠️ Description truncated due to length constraint. Please refer to README.md for detail.⚠️⚠️⚠️ Description truncated due to length constraint. Please refer to README.md for detail. Beyond Lines of Code: Probing the Latent Representation of Magnitude-Based Code Smells in Large Language Models TL; DRRun full pipeline: RQ1 logit extraction → RQ2 probing → RQ3 explanation # Extract Java source archive first7z x java_code_samples.7z -o data/export PYTHONPATH="${PYTHONPATH}:$(pwd)/scripts/shared"python scripts/rq1/rq1_magnitude_alignment.py # ~8 GPU-hours per model × 8 variantspython scripts/rq2/rq2_representation_probing.py # ~8 GPU-hours per model × 4 modelspython scripts/rq3/rq3_robustness_check.py # ~4 GPU-hours 1. Overview This replication package provides the complete experimental pipeline, raw data, pre-computed results, and analysis scripts required to reproduce every table, figure, and statistical claim in the manuscript. The study investigates the internal representations and behavioral patterns of four open-weight large language model (LLM) architectures when detecting two canonical magnitude-based code smells, i.e., Long Method and God Class (a.k.a. Blob), using a suite of white-box interpretability techniques. 1.1 Research Questions RQ Question Key Technique 1 How do different evaluation modalities (generative decoding vs. logit extraction) and post-training alignment affect LLM-based code smell detection? White-box logit extraction, persona-driven prompting, Platt/Isotonic calibration 2 At which representational depth do LLMs encode code smell signals, and how do attention patterns and causal interventions reveal the underlying mechanism? Layer-wise linear/non-linear probing, perimeter attention analysis, activation patching 3 How robust are the LLM's code smell judgments to adversarial physical perturbations and structure-preserving refactoring? AST padding attack, Extract Class refactoring, variable-renaming robustness check 2. Requirements 2.1 Hardware Component Minimum Recommended GPU NVIDIA GPU with ≥16 GB VRAM NVIDIA A6000 48 GB RAM 32 GB 64 GB Storage 50 GB free 100 GB free OS Linux / Windows with CUDA Ubuntu 22.04 LTS Full pipeline execution (all 4 models × 3 RQs × supplementary experiments) requires approximately 48 GPU-hours on an A100. Individual experiments range from minutes (CPU-only calibration analyses) to ~8 hours (full-scale probing across 33 layers × 4 models). 2.2 Software Dependencies pip install -r requirements.txt Core dependencies: torch>=2.1.0 with CUDA support transformers>=4.36.0 pandas>=2.0, numpy>=1.24, scipy>=1.10 scikit-learn>=1.3 matplotlib>=3.7, tqdm accelerate>=0.25 (for device_map="auto") 2.3 Model Weights Models are automatically downloaded from HuggingFace Hub on first use. To pre-download: python -c "from transformers import AutoModelForCausalLM, AutoTokenizerfor model_id in [ 'Qwen/Qwen2.5-Coder-7B-Instruct', 'Qwen/Qwen2.5-Coder-7B', 'meta-llama/Meta-Llama-3-8B-Instruct', 'meta-llama/Meta-Llama-3-8B', 'deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct', 'deepseek-ai/DeepSeek-Coder-V2-Lite-Base', 'google/gemma-2-9b-it', 'google/gemma-2-9b',]: print(f'Downloading {model_id}...') AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype='bfloat16')" All four model families (Qwen, Llama, DeepSeek, Gemma) are available via HuggingFace Hub. Access to gated models requires a HuggingFace account. 2.4 Dataset Setup The raw Java source files are distributed as a separate archive due to their size. Before running any experiment scripts, extract the archive into the data/ directory: # Using 7-Zip (Windows / Linux)7z x java_code_samples.7z -o data/ # Using p7zip (Linux/macOS)p7zip -d java_code_samples.7z && mv java_code_samples data/ After extraction, verify the structure: data/└── java_code_samples/ ├── alibaba_druid_768469fe/ ├── apache_accumulo_f896c98c/ └── ... (522 repo directories) Scripts that analyze raw source code (e.g., LOC counting, probing) require the Java sources. Table reproduction and calibration analysis scripts operate on the pre-computed CSV results included in results/. 3. Prompt Templates and Verbalizer Configuration The logit extraction pipeline uses a persona-conditioned prompting strategy with three distinct system personas paired with binary verbalizer token pairs. This section documents the exact templates used. 3.1 System Personas Three system prompts are used to probe behavioral alignment under different instruction-following contexts: Persona 1: Helpful Assistant You are a helpful and polite AI assistant. This represents the default aligned behavior. The model receives a cooperative, non-adversarial framing that mirrors typical deployment scenarios. Persona 2: Harsh Critic You are a harsh, cynical Senior Staff Engineer. You aggressively find flaws and detest poorly written code. You do not hold back criticism. This adversarial persona is designed to override the model's tendency toward polite, non-committal responses. By framing the interaction as adversarial code review, we test whether the underlying structural judgment (encoded in the logit distribution) is modulated by the instruction-following layer's persona conditioning. Persona 3: Deterministic System You are a deterministic structural analysis engine evaluating syntax metrics. This mechanistic persona attempts to strip away conversational artifacts entirely, framing the model as a pure computation engine. This tests the hypothesis that the alignment tax manifests differently when the model is conditioned to behave as a deterministic tool vs. a conversational agent. 4. Directory Structure replication_package/├── README.md # This file├── requirements.txt # Python dependencies│├── config/│ └── personas_verbalizers.json # Full prompt/verbalizer configuration (see §3)│├── data/│ ├── mlcq-with-metrics.csv # MLCQ dataset with 63 AST metrics│ ├── all_metrics_class.csv # Class-level metric features│ ├── all_metrics_method.csv # Method-level metric features│ ├── MLCQCodeSmellSamples.xlsx # Original MLCQ annotations│ └── java_code_samples/ # Extract from java_code_samples.7z (see setup below)│ ├── alibaba_druid_768469fe/│ ├── apache_accumulo_f896c98c/│ ├── google_guava_1c26b85c/│ └── ... (522 repos total, ~1.2 GB compressed)│├── scripts/│ ├── rq1/ # RQ1 experiment scripts│ │ ├── rq1_magnitude_alignment.py│ │ ├── rq1_full_group_kfold.py│ │ ├── rq1_contextual_calibration.py│ │ ├── rq_baselines_calibration.py│ │ ├── rq_platt_calibration.py│ │ └── rq_verbalizer_sensitivity.py│ ││ ├── rq2/ # RQ2 experiment scripts│ │ ├── rq2_representation_probing.py│ │ ├── rq2_attention_rigor.py│ │ ├── rq2_attention_sensitivity.py│ │ ├── rq2_causal_tracing.py│ │ ├── rq2_activation_patching_multimodel.py│ │ ├── causal_attention_masking.py│ │ ├── eval_partial_correlation.py│ │ └── rq_pooling_comparison.py│ ││ ├── rq3/ # RQ3 experiment scripts│ │ ├── rq3_robustness_check.py│ │ ├── rq_extract_class_control.py│ │ ├── rq_token_type_ablation.py│ │ └── rq3_extract_class_multimodel.py│ ││ ├── shared/ # Shared utilities│ │ ├── data_provider.py # Dataset loading and code sample retrieval│ │ ├── run_multimodel_full_pipeline.py│ │ ├── rf_baseline.py # Random Forest baseline (63-metric features)│ │ ├── static_baseline.py # LOC Heuristic baseline│ │ └── compute_delong.py # DeLong test for AUC comparison│ ││ ├── figures/ # Figure generation│ │ ├── plot_fig2_persona_alignment.py # Figure 2: Persona alignment curves│ │ └── plot_rq3_comparison.py # Figure 3: Robustness cosine distance│ ││ ├── tables/ # Table computation│ │ ├── calc_4model_metrics.py│ │ ├── calc_auc.py│ │ ├── calc_baseline_f1.py│ │ ├── calc_metrics.py│ │ └── reproduce_tables.py # Reproduces Tables 1-3 from pre-computed CSVs│ ││ └── supplementary/ # Supplementary experiments│ ├── rq2_hidden_state_probing_bf16.py│ ├── rq1_verbalizer_sweep.py│ ├── rq2_perimeter_attention_control.py│ ├── rq1_temperature_scaling.py│ ├── rq1_stratified_loc_analysis.py│ ├── rq1_isolated_long_method_eval.py│ ├── rq2_partial_correlation_cc.py│ ├── rq1_verbalizer_sensitivity_contextual.py│ ├── rq3_size_neutral_refactoring.py│ ├── rq1_isotonic_audit.py│ ├── rq1_contrastive_verbalizer.py│ ├── rq3_extract_class_kwdensity.py│ └── rq1_rf_llm_hybrid.py│├── results/│ ├── rq1/ # Per-model probability extraction results│ │ ├── rq1_results_robust_Qwen.csv│ │ ├── rq1_results_robust_Qwen-Base.csv│ │ ├── rq1_results_robust_Llama.csv│ │ ├── rq1_results_robust_Llama-Base.csv│ │ ├── rq1_results_robust_DeepSeek.csv│ │ ├── rq1_results_robust_DeepSeek-Base.csv│ │ ├── rq1_results_robust_Gemma.csv│ │ ├── rq1_results_robust_Gemma-Base.csv│ │ ├── rq1_results_robust.csv│ │ ├── contrastive_verbalizer_results.csv│ │ ├── verbalizer_sensitivity_fast_results.csv│ │ ├── full_dataset_group_kfold_results.csv│ │ ├── temperature_scaling_results.csv│ │ └── isotonic_audit_results.csv│ ││ ├── rq2/│ │ ├── rq2_probing_results_Qwen_bf16.csv│ │ ├── rq2_probing_results_Llama_bf16.csv│ │ ├── rq2_probing_results_DeepSeek_bf16.csv│ │ ├── rq2_probing_results_Gemma_bf16.csv│ │ ├── rq2_probing_results_*-Base.csv (Base model probing)│ │ ├── causal_tracing_results.csv│ │ └── causal_masking_results_red.csv│ ││ ├── rq3/│ │ ├── rq3_robustness_results_{Qwen,Llama,DeepSeek,Gemma}.csv│ │ ├── extract_class_kwdensity_results.csv│ │ ├

Fields of Study & Subjects

Software Engineering Research Explainable Artificial Intelligence (XAI) Topic Modeling

Publication details

Journal / source
Figshare
Publication date
30 March 2026
Language
EN
Document type
Dataset