Towards Edge AI Pipeline Replicability in Presence of Unavailable Data

Read full text Summarise with VIRAT
Authors: Neelesh Karthikeyan, Manikya Swathi Vallabhajosyula, Beth Plale
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Overview This artifact accompanies the paper Towards Edge AI Pipeline Replicability in Presence of Unavailable Data submitted to the 4th ACM Conference on Reproducibility and Replicability (ACM REP 2026). The study introduces an ML inference replicability tuple (Dataset × Model × ExperimentPlan × ServingConfiguration × EdgeDevice) that formalizes the core components of an AI experiment within the Patra Model Cards framework. It further proposes a schema-based dataset substitution approach using LLM-driven schema extraction to identify structurally compatible public datasets when original data is unavailable. The artifact supports two claims: Claim 1 (Inference Replicability): ML inference behavior remains bounded under a fixed experimental configuration but varies systematically when serving configuration or hardware changes. Serving configuration changes (e.g., PyTorch → ONNX FP16) cause larger accuracy deviations than hardware changes alone. Claim 2 (Schema-Based Dataset Substitution): When original datasets are unavailable, a hybrid schema search can identify structurally compatible public alternatives by matching dataset schemas encoded as JSON-LD / RO-Crate, outperforming a pure cosine baseline. Contents ML Inference Pipeline (Claim 1) A fully containerized inference pipeline that runs YOLO object detection models across cloud and edge hardware, compares outputs, and generates publication-ready tables. ComponentDescriptionentrypoint.shPipeline orchestrator (export → inference → metrics → compare → tables)export_models.pyConverts .pt weights to ONNX FP32, ONNX FP16, TorchScriptrun_inference.pyBatch inference loop (6 models × 4 formats per VM)compute_metrics.pymAP, Precision, Recall against ground-truth labelscompare_predictions.pyPairwise divergence: serving config (Scenario 1) and hardware (Scenario 2)generate_tables.pyAggregates results into CSV and Markdown tablesDockerfile.cloud / Dockerfile.edgeContainerized environments for cloud (GPU) and edge (CPU)docker-compose.ymlOrchestration for both profiles Data Data format The artifact produces and includes tabular data in CSV format. Each CSV file contains one header row followed by data rows, with columns separated by commas. The input dataset uses YOLO annotation format: images stored as JPEG/PNG files in an images/ directory, with corresponding text label files in a labels/ directory (one .txt per image, each line containing class_id x_center y_center width height in normalized coordinates). Data content Ohio Small Animal Dataset A 100-image subset used as the inference target for Claim 1, in YOLO annotation format. Source: Google Drive Format: YOLO (images + bounding box label files) Size: 100 images Telemetry: cloud.csv Cloud-side serving configuration comparison. Each row represents one model evaluated under one serving configuration on the Cloud VM, with divergence metrics computed against the .pt baseline. ColumnDescriptionModelModel variant (e.g., yolov8n, yolo11m)Serving Config.pt (baseline), ONNX FP32, ONNX FP16, TorchScriptmAP@0.5Mean Average Precision at IoU threshold 0.5mAP@0.5:0.95Mean Average Precision averaged over IoU thresholds 0.5 to 0.95PrecisionDetection precisionRecallDetection recallLabel Agree. vs .ptPercentage of images where predicted label set matches .pt baselineConf Delta vs .ptMean absolute confidence score difference vs .pt baselineMean IoU vs .ptMean Intersection-over-Union of bounding boxes vs .pt baseline Telemetry: serving_config.csv Full serving configuration comparison across both Cloud and Edge VMs. Same columns as cloud.csv with an additional VM column (cloud or edge). Telemetry: hardware.csv Hardware divergence analysis. Each row compares the same model and serving configuration across Cloud and Edge hardware. ColumnDescriptionModelModel variantServing ConfigServing configurationHardwarecloud or edgemAP@0.5Mean Average Precision at IoU 0.5mAP@0.5:0.95Mean Average Precision at IoU 0.5:0.95PrecisionDetection precisionRecallDetection recallLabel Agree. vs CloudPercentage of images where edge labels match cloud labelsConf Delta vs CloudMean absolute confidence difference vs cloudMean IoU vs CloudMean bounding box IoU vs cloud predictions Telemetry: summary_nondeterminism.csv Summary of maximum non-determinism observed per model across scenarios. ColumnDescriptionModelModel variantScenarioserving_config or hardwareMax Label DisagreementHighest label disagreement observedMax Conf DeltaLargest confidence delta observedMin Mean IoULowest Mean IoU (worst-case spatial divergence) Gen-Parallel-Workloads (Claim 2) Schema substitution corpus containing 31 dataset-backed schemas extracted from the Gen-Parallel-Workloads repository, including original training traces and synthetic traces from HPC systems (Theta, Helios, Philly, Blue Waters, SDSC). Source: Link to be provided Models Six YOLO object detection model variants were evaluated: ModelWeights (persistent URL)YOLOv8nultralytics/v8.2.0/yolov8n.ptYOLOv8sultralytics/v8.2.0/yolov8s.ptYOLOv8multralytics/v8.2.0/yolov8m.ptYOLO11nultralytics/v8.3.0/yolo11n.ptYOLO11sultralytics/v8.3.0/yolo11s.ptYOLO11multralytics/v8.3.0/yolo11m.pt Each model is evaluated in four serving configurations: PyTorch (.pt), ONNX FP32, ONNX FP16, and TorchScript. Experimental Environment Experiments were conducted on Jetstream2, an NSF-funded cloud computing resource. PropertyCloud VM (g3.large)Edge VM (m3.tiny)Jetstream2 Flavorg3.largem3.tinyvCPUs161System RAM120 GB3 GBGPUPartial NVIDIA A100 (20 GB vGPU)None (CPU-only) Software Stack ComponentVersionPython3.11PyTorch2.2.2Ultralytics8.3.3ONNX Runtime1.18.0 (onnxruntime-gpu on Cloud, onnxruntime on Edge)Docker Engine24.x+ The Cloud image builds from nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04; the Edge image from python:3.11-slim-bookworm. Patra Framework & LLMs Patra ComponentRepositoryPatra Knowledge Graphgithub.com/Plale-Lab/patra-kgPatra Frontendgithub.com/Plale-Lab/patra-frontendCKNgithub.com/Plale-Lab/cyberinfrastructure-knowledge-network Patra operationalizes the replicability tuple through a knowledge graph using JSON-LD and RO-Crate standards. LLMs Used for Schema Extraction LLMVersionProviderLlama 3.18BMeta (open-weight)GPT-4o-mini—OpenAIGPT-4.1-mini—OpenAI Repository Source code: https://github.com/Plale-Lab/replicability_study Reproducing Results Quick Validation (≤ 15 minutes, no GPU required) git clone https://github.com/Plale-Lab/replicability_study.git cd replicability_study mkdir -p dataset models results docker compose build edge docker compose run --rm -e VM=edge -e TEST_MODE=true -e RUN_ANALYSIS=true edge ls results/tables/ Full Experiment (requires two Jetstream2 VMs) Cloud VM (g3.large): Build and run the Cloud container for GPU-accelerated inference across all 6 models × 4 formats. Transfer: Copy models/ and results/cloud/ to the Edge VM. Edge VM (m3.tiny): Build and run the Edge container for CPU-only inference. Analysis: Run with RUN_ANALYSIS=true to generate comparison tables. See DEPLOYMENT.md in the repository for detailed step-by-step instructions. Verifying Claims Claim 1: Compare results/tables/cloud.csv against the paper’s Table 2 (Cloud columns). Confirm that ONNX FP16 shows the largest label agreement drop (78–88%) and Mean IoU degradation. Check hardware.csv for cross-hardware drift. Claim 2: Instructions to be added once schema search code is deposited. License BSD 3-Clause License. See LICENSE file.

Fields of Study & Subjects

Scientific Computing and Data Management IoT and Edge/Fog Computing Advanced Neural Network Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
22 March 2026
Language
EN