Can we estimate privacy vulnerability of individual records? Towards Mitigating Attribute Inference Attacks on ML Models

The accompanying artifacts support the reproducibility of the experiments presented in Can We Estimate Privacy Vulnerability of Individual Records? Towards Mitigating Attribute Inference Attacks on ML Models. The artifacts include the source code, datasets, and Jupyter notebooks used to implement and evaluate the proposed record-level vulnerability estimation tools (NeighVE and AttriVET) and the defense mechanism VESL. The datasets consist of publicly available tabular datasets—Adult, Census-19, and Texas-100X—processed according to the experimental setup described in the paper. The codebase provides modular components for data preprocessing, neighborhood construction, vulnerability estimation, model training, attack execution, and defense evaluation. The provided notebooks execute the experimental pipelines used in the paper, reproducing key results such as neighborhood angular difference analysis, record-level vulnerability prediction accuracy, the effectiveness of VESL against attribute inference attacks, and the resulting impacts on model utility and fairness. All artifacts are released in accordance with the conference’s open science policy and are intended to enable independent verification of the reported findings. 📊 Datasets census19.csv – Census-19 dataset. Adult_35222.csv and Adult_10000.csv – Adult dataset partitions for training and testing, respectively. texas_100_cleaned.csv – Texas-100X dataset. 🖥️ Codebase Source Files data_utils.py Loads and preprocesses datasets Implements data sampling matching target correlation (Section 6.1, “Sampling Technique”) model_utils.py Defines model architecture and training hyperparameters experiment_utils.py Sets up experiments for specific scenarios attack_utils.py Implements existing attacks: CSMIA, LOMIA, and Imputation Attack disparity_inference_utils.py Implements Angular Difference computation Novel Algorithms adv_vuln_utils.py – Implementation of neighVE and synthetic data generation subspace_learning_utils.py – Implementation of VESL vulnerability_score_utils.py – Implementation of AttriVET 📓 Notebooks attriVET_evaluation.ipynb – Runs experiments from Section 6.2 vesl_evaluation.ipynb – Runs experiments from Section 6.3 neighve_evaluation.ipynb – Runs experiments from Sections 4.1 and 6.4 ⚙️ Environment Setup Recommended Python version: 3.9.16 Create a virtual environment (using Conda or venv) and activate it: # With Conda conda create -n vesl_env python=3.9.16 conda activate vesl_env Install required packages: pip install -r requirements.txt The specific versions of the listed packages that were used in implementation are included in requirements_w_version.txt for reproducibility. To install those exact versions, run: pip install -r requirements_w_version.txt Note: The package torch may require specific installation commands based on your system configuration (e.g., CUDA version). Please refer to the official PyTorch installation guide if you encounter issues. (Optional) Install Jupyter Notebook: pip install notebook ✅ Reproducing Main Results AttriVET (Section 6.2) — attriVET_evaluation.ipynb This notebook reproduces the AttriVET evaluation in the paper by: training target (victim) models under different datasets, computing AttriVET record-level vulnerability labels, and measuring how well those labels align with attack prediction validity (i.e., whether an attack correctly inferred the sensitive attribute). Open attriVET_evaluation.ipynb and Run All (cells must be executed top-to-bottom). What each cell does (cell-by-cell) Cell 1-3 — Setup and imports Notebook setup (autoreload, warning suppression). Imports project modules, attack implementations, evaluation metrics. Cell 4 — Define experiment configurations Creates an experiments dictionary and a job_list specifying which components to run (e.g., training a target model and computing AttriVET scores). Instantiates multiple MIAExperiment(...) objects to cover different settings. During initialization, each experiment object internally: Loads Training, Test, and Auxiliary datasets Loads or trains target models (e.g., MLP) Loads or computes per-record vulnerability scores using AttriVET. Cell 5 — Run attacks and cache per-record correctness masks For each configured experiment: Runs CSMIA and LOMIA against the trained target model(s). Stores per-record predicted sensitive attribute values for each attack. Computes boolean masks such as: correct_indices_CSMIA_dict[model] / incorrect_indices_CSMIA_dict[model] correct_indices_LOMIA_dict[model] / incorrect_indices_LOMIA_dict[model] These masks represent attack prediction validity and are later used as the “ground truth” vulnerability indicator. Cell 6 — Implement the evaluation helper Implement a helper that evaluates how well AttriVET’s vulnerability labels match attack prediction validity: Inputs: experiments (all scenarios) model_name (e.g., 'MLP') return_confusion toggles confusion-matrix reporting Core comparison: preds = experiment.all_vuln_scores_rounded (AttriVET 0/1 labels) trues = experiment.correct_indices_{ATTACK}_dict[model_name].astype(int) (attack correctness as ground truth) Outputs: Overall accuracy / precision / recall / F1 per (attack, dataset setting) Or a confusion matrix breakdown when return_confusion=True. Cell 7 — Compute metrics Calls evaluate_attrivet(experiments, model_name='MLP'). The output is a nested dictionary with the main metrics for CSMIA and LOMIA across the configured experiment settings. Cell 8 — Confusion-matrix view Calls evaluate_attrivet(experiments, model_name='MLP', return_confusion=True). Prints TP/TN/FP/FN counts for each attack and setting. VESL (Section 6.3) — vesl_evaluation.ipynb This notebook reproduces the VESL defense evaluation (Section 6.3), comparing No Defense, SMIR, VESL-MV, and VESL-RS under attribute inference attacks. Open vesl_evaluation.ipynb and Run All (cells must be executed top-to-bottom). What each cell does (cell-by-cell) Cell 1–3 — Setup and imports Notebook setup (autoreload, warning suppression). Imports project modules, attack implementations, evaluation metrics, and fairness utilities. Cell 4 — Define experiment configurations Creates an experiments dictionary and a job_list specifying which components to run. Instantiates multiple MIAExperiment objects covering different datasets and experiment settings. During initialization, each experiment object internally: Loads Training, Test, and Auxiliary datasets Loads or trains target models Loads or computes per-record vulnerability scores using AttriVET. Loads or trains defense models (VESL-MV, VESL-RS). Cell 5 — Train or load SMIR baseline Builds train loaders augmented with AttriVET vulnerability labels. Loads a cached SMIR model if available; otherwise trains it and saves the checkpoint. Uses mutual-information regularization to suppress sensitive attribute leakage. Cell 6 — Run attribute inference attacksFor each experiment and defense model: Defines evaluated models: No Defense, SMIR, VESL-MV, VESL-RS. Runs CSMIA and LOMIA attacks and stores predicted sensitive attributes. For VESL-RS, runs multiple randomized inference passes and averages predictions. Cell 7 — Define evaluation helpers Defines metric helpers for accuracy, precision, recall, and F1 (macro-averaged). Implements get_perf(...), which: Compares attack predictions against true sensitive attributes Adds Random Guessing and Imputation baselines Cell 8 — Attack performance Computes and prints attack performance for CSMIA and LOMIA. Reports metrics for all defenses and baselines. Cell 10 — Model utility evaluation Computes task accuracy on the test set for: No Defense SMIR VESL-MV VESL-RS Cell 11 — Fairness evaluation Computes group fairness metrics on the test set: Equalized Odds Difference (EOD) Demographic Parity Difference (DPD) Reports fairness for No Defense and VESL variants. NeighVE (Sections 4.1 and 6.4) — neighve_evaluation.ipynb This notebook reproduces the NeighVE vulnerability estimation experiments, including: the validation of using n.a.d. (Section 4.1), and the effectiveness of NeighVE under attribute inference attacks (Section 6.4). Open neighve_evaluation.ipynb and Run All (cells must be executed top-to-bottom). NeighVE (neighve_evaluation.ipynb) Cells 1–3: Setup and imports Suppress warnings for cleaner output. Import required utilities (datasets, experiments, attacks, plotting) and set global plotting style. Cells 4: Define experiments Creates an experiments dictionary and a job_list specifying which components to run. Instantiates multiple MIAExperiment objects covering different datasets and experiment settings. During initialization, each experiment object internally: Loads Training, Test, and Auxiliary datasets Loads or trains target models Loads or computes per-record vulnerability scores using AttriVET. Loads or trains defense models (VESL-MV, VESL-RS). Loads or computes per-record NeighVE scores. Cell 5: Run attacks and compute correctness labels For each experiment: Computes per-record confidence scores. Runs CSMIA, storing predicted sensitive attributes and attack-case indices. Runs LOMIA, storing predicted sensitive attributes. Computes boolean correctness arrays indicating whether each attack correctly inferred the sensitive attribute. Cell 6: NeighVE histograms (CSMIA) For each experiment: Plots histograms of neighborhood angular differences. Distributions are split by whether CSMIA inferred the sensitive attribute correctly or incorrectly. Cell 7: Neighborhood similarity comparison (CSMIA) For each experiment: Computes the 75th percentile of neighborhood angular differences. Separately for each sensitive group: Computes mean and stand