Pairwise Comparison Aggregation with Tool-Augmented LLMs for Procurement Value Lever Prioritization

Traditional prioritization of procurement Value Levers (VLs) relies on manual expert evaluation across multiple strategic dimensions, making the process time-intensive, resource-demanding, and vulnerable to subjective inconsistency. This study investigates the use of Large Language Models (LLMs) to semi-automate VL ranking through a pairwise comparison aggregation framework enriched with established strategic tools. The proposed methodology integrates LLM-based contextual reasoning with the Kraljic Matrix, Procurement with Purpose, and SWOT analysis to produce consistent, context-aware rankings across a portfolio of 15 VLs. A systematic experimental design evaluates two model architectures (Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct), alternative prompt representations (numeric vs. categorical), varying contextual depth, and multiple aggregation mechanisms. Results show moderate alignment with proprietary baseline rankings (weighted Kendall’s τ = 0.38–0.45), with performance improving through prompt calibration and strategic tool integration. Categorical representations consistently outperform numeric formats, indicating that qualitative descriptors better match LLM internal reasoning in procurement contexts. Strategic tool integration reduces ranking position variance by 84% and increases stability sixfold, while maintaining acceptable agreement with domain experts (68% exact agreement; Cohen’s κ ₍quadratic₎ = 0.62). Position-specific variance is observed for a small subset of VLs (13–20%), which may reflect either model-induced structural effects or the capture of widely accepted procurement best practices. Overall, the findings demonstrate that LLM-based VL ranking is operationally viable for initial screening and sensitivity analysis, substantially reducing manual effort while preserving decision quality.