UTH Reinforcement Learning Policy Model for Dynamic FPGA DPU Configuration Selection

Overview This repository provides a trained reinforcement-learning policy model, exported to ONNX, that selects an FPGA DPU configuration—defined by DPU size and number of DPU compute units (instances)—given an observation vector describing the current system and job context. The model is intended for offline analysis, benchmarking, and decision support in FPGA-based ML inference pipelines, where selecting an appropriate DPU configuration is critical for performance and efficiency. Problem Statement Modern FPGA platforms support multiple DPU bitstream configurations, trading off parallelism, resource usage, and performance. Selecting an optimal configuration at runtime is non-trivial due to: varying model characteristics, changing workload intensity, contention between FPGA and ARM CPU resources, and complex performance–power trade-offs. This repository addresses the problem of DPU configuration selection as a discrete decision-making task, learned via reinforcement learning from prior experimentation. Goal:Given an observation vector describing the system/job state, predict the most suitable DPU configuration from a fixed action space. Dataset The policy model was trained using telemetry and experiment data collected from repeated ML inference runs on a Xilinx ZCU102 FPGA platform. Dataset repository (Zenodo):https://zenodo.org/records/18494461 The dataset provides: experiment configurations, time-series telemetry, performance, power, and system metrics, and serves as the basis for training and evaluating this policy. Model Summary Task: Discrete action selection (RL policy inference) Framework: Reinforcement Learning (trained offline) Export format: ONNX Inference backend: ONNX Runtime Output: Action index corresponding to a DPU configuration The model outputs logits over a discrete action space.At inference time, the selected action is obtained via argmax. Action Space (DPU Configurations) Each model output corresponds to one action in a fixed action space.An action maps to: DPU Size (bitstream size), and CU count (number of DPU compute units / instances). The canonical mapping is defined in: action_interpreter.py → DEFAULT_ACTION_CONFIGS This mapping is part of the model contract and must remain consistent with training and export. Model Input/Output Inputs (IMPORTANT: Normalized) Training used normalized observations.All numeric features must be normalized to [0, 1] using fixed caps/bounds (see below).If you pass raw telemetry units directly (e.g., MB/s, W, CPU%), the model output will not be meaningful. Observation layout (22 features total) The model input is a single flat vector with this exact order: System telemetry (16) cpu_0, cpu_1, cpu_2, cpu_3 S0_read, S1_read, S2_read, S3_read, S4_read S0_write, S1_write, S2_write, S3_write, S4_write fpga_power, arm_power Job static characteristics (5) gmac, ldfm, ldwb, stfm, parameters Constraints (1) target_fps Normalization caps / bounds (training contract) System caps: max_bw_mb_s = 7000 max_power_w = 20.0 Job static bounds: canonical_depth: (18.0, 152.0) (not used in this release’s observation vector) gmac: (0.3, 12.303) ldfm: (0.792, 91.787) ldwb: (3.326, 65.66) stfm: (0.192, 61.125) parameters: (3.5, 60.2) Dynamic bounds (used for internal training instrumentation; not part of the released observation vector): total_fps: (10.0, 100.0) ppw: (1.0, 100.0) Normalization formula: x_norm = clip((x - low) / (high - low), 0, 1) Outputs Logits tensor of shape: (1, num_actions) or (batch_size, num_actions) The inference examples select: action_idx = argmax(logits) Usage Prerequisites Install required dependencies: onnxruntime numpy pandas Action Interpretation action_interpreter.py provides the ActionInterpreter utility, which: maps action_idx → (dpu_size, cu_count), produces human-readable descriptions, enables consistent decoding of model outputs, supports action-space inspection and statistics. Note:Changing the action ordering will silently invalidate predictions. Programmatic Inference (Recommended) Use inference_wrapper.py to integrate the model into other scripts or services. Load the ONNX model Pass one or more normalized observation vectors Receive decoded DPU configuration and confidence The following example shows how to load the model and run inference on a single observation vector. import numpy as np from src.inference_wrapper import FPGARLInference, normalize_observation_row # 1) Load the ONNX policy model = FPGARLInference("fpga_rl_policy.onnx", verbose=True) # 2) Option A (recommended): provide RAW features in dataset column names, # then normalize using the same caps/bounds as training. raw_features = { "cpu_0": 12.5, "cpu_1": 10.2, "cpu_2": 8.8, "cpu_3": 9.7, "S0_read": 1200.0, "S1_read": 1180.0, "S2_read": 150.0, "S3_read": 120.0, "S4_read": 90.0, "S0_write": 220.0, "S1_write": 210.0, "S2_write": 35.0, "S3_write": 28.0, "S4_write": 20.0, "fpga_power": 3.10, "arm_power": 1.85, "gmac": 1.82, "ldfm": 25.40, "ldwb": 14.20, "stfm": 9.10, "parameters": 11.70, "target_fps": 30.0, } # Normalize to the 22-feature vector expected by the model obs_norm = normalize_observation_row(raw_features) # shape: (22,) result = model.predict(obs_norm) # runs ONNX inference print(model.format_result(result)) print("Action index:", result["action_idx"]) # 3) Option B: pass an already-normalized observation vector (float32, 22 dims). # (Only do this if you *know* your normalization matches training.) obs_norm_direct = obs_norm.astype(np.float32) result2 = model.predict(obs_norm_direct, return_logits=True) print("Top logit:", float(result2["logits"][result2["action_idx"]])) # 4) Batch inference example (N x 22) batch_raw = [raw_features, {**raw_features, "cpu_0": 35.0, "target_fps": 60.0}] batch_obs = np.stack([normalize_observation_row(r) for r in batch_raw], axis=0) # (N, 22) batch_results = model.predict_batch(batch_obs) for i, r in enumerate(batch_results): print(f"Sample {i}: {r['action_str']} (confidence={r['confidence']:.2%})") CSV-Based Inference Demo onnx_inference_example.py provides an end-to-end example that: loads the ONNX model, reads observations from a CSV file, runs inference row-by-row, prints selected actions and summary statistics. This is useful for offline replay, inspection, and experimentation. python3 src/onnx_inference_example.py \ --onnx model/fpga_rl_policy.onnx \ --csv sample_data.csv \ --out inference_results.csv Model Constraints The action space is fixed and cannot be extended without retraining. The model produces one discrete decision only (no multi-objective output). Predictions assume consistent feature preprocessing and observation semantics. Inference is stateless; temporal dependencies are not modeled. Citation If you use this model in academic work, please cite the corresponding Zenodo release. Acknowledgements & Funding This work is part of the MLSysOps project and is funded by the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101092912. More information: https://mlsysops.eu/